1. The problem I’m having:
I am trying my best to describe my problem here - at first: I am one of the poor people behind a CGNAT and so I am trying to get my home lab, or at least some services, exposed to the www for ease of use.
I bought a VPS server with a static external IP and installed Wireguard there - I set up a small Wireguard client on my home lab as a Proxmox LXC which accepts all incoming things from my VPS.
This connection is working, I can ping everything in my home lab over the tunnel and I can ping out over the tunnel - at least from the Wireguard Client machine.
I went through the configuration of several reverse proxy services/applications and found Caddy the easiest to use.
I installed Caddy to the VPS server and to the Wireguard Client machine and hoped that those two will work together to achieve everything I want, but little did I know. And it brought me down to an issue with Caddy.
Now I want to achieve, that the external communication (everything when I am not at home) is going through the VPS → WG → WG-Client → SERVICE (this is working).
And I also want to achieve, that the internal communication (everything when I am at home) is just going Smartphone/Computer → WG-Client → SERVICE (this is not working).
I got told, that this must be a DNS issue, but every time I set my DNS rewrites in my AdGuard to rewrite my *.mydomain.com to the WG-Client, I get a ERR_CONNECTION_REFUSED.
I hope the following picture will help understanding everything.
2. Error messages and/or full log output:
This is the output of curl -vL to nc.mydomain.com on the WG client
curl -vL nc.mydomain.com
* Trying 192.168.178.253:80...
* Connected to nc.mydomain.com (192.168.178.253) port 80 (#0)
> GET / HTTP/1.1
> Host: nc.mydomain.com
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Cache-Control: no-store, no-cache, must-revalidate
< Content-Length: 0
< Content-Security-Policy: default-src 'self'; script-src 'self' 'nonce-OUkwVVNXd1MwS0djejZYdGI4c0xMU3VJeWFuYjBoZGFrOEk5N0xQcmQrUT06amVzc0JTVjVwK3ZNaS9PNlBabENWMExmNXR1UW5FRUsxN2QvdHRLWlByND0='; style-src 'self' 'unsafe-inline'; frame-src *; img-src * data: blob:; font-src 'self' data:; media-src *; connect-src *; object-src 'none'; base-uri 'self';
< Content-Type: text/html; charset=UTF-8
< Date: Sat, 13 Apr 2024 10:42:49 GMT
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Location: https://nc.mydomain.com/login
< Pragma: no-cache
< Referrer-Policy: no-referrer
< Server: Caddy
< Server: Apache
< Set-Cookie: oc_sessionPassphrase=a5rpOec90HzxsM7i6Zhmbtg%2BIf2%2BKlq9b3zVRW4h3dnWEk2i24KPxYA2hua%2Ba%2B%2Bejr4QhWbyokYhly50x%2F2%2F3HONCeT0bBYCgNjkm7EGKcV7Iyr4KIimNqtSQtSRXOtv; path=/; secure; HttpOnly; SameSite=Lax
< Set-Cookie: __Host-nc_sameSiteCookielax=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=lax
< Set-Cookie: __Host-nc_sameSiteCookiestrict=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=strict
< Set-Cookie: ocoa4aex63uo=34jpunags6ud54edft52gadnl7; path=/; secure; HttpOnly; SameSite=Lax
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Permitted-Cross-Domain-Policies: none
< X-Robots-Tag: noindex, nofollow
< X-Xss-Protection: 1; mode=block
<
* Connection #0 to host nc.mydomain.com left intact
* Clear auth, redirects to port from 80 to 443Issue another request to this URL: 'https://nc.mydomain.com/login'
* Trying 192.168.178.253:443...
* connect to 192.168.178.253 port 443 failed: Verbindungsaufbau abgelehnt
* Failed to connect to nc.mydomain.com port 443: Verbindungsaufbau abgelehnt
* Closing connection 1
curl: (7) Failed to connect to nc.mydomain.com port 443: Verbindungsaufbau abgelehnt
This is the cur -vL output from nc.mydomain.com on the VPS (external way):
(I stripped the HTML-Part)
* Connected to nc.mydomain.com (1.2.3.4) port 80 (#0)
> GET / HTTP/1.1
> Host: nc.mydomain.com
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://nc.mydomain.com/
< Server: Caddy
< Date: Sat, 13 Apr 2024 10:58:48 GMT
< Content-Length: 0
<
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
* Closing connection 0
* Clear auth, redirects to port from 80 to 443
* Issue another request to this URL: 'https://nc.mydomain.com/'
* Trying 1.2.3.4:443...
* Connected to nc.mydomain.com (1.2.3.4) port 443 (#1)
* ALPN: offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [15 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2382 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [78 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
* subject: CN=nc.mydomain.com
* start date: Apr 12 11:15:48 2024 GMT
* expire date: Jul 11 11:15:47 2024 GMT
* subjectAltName: host "nc.mydomain.com" matched cert's "nc.mydomain.com"
* issuer: C=US; O=Let's Encrypt; CN=R3
* SSL certificate verify ok.
} [5 bytes data]
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: nc.mydomain.com]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x556e2c15a790)
} [5 bytes data]
> GET / HTTP/2
> Host: nc.mydomain.com
> user-agent: curl/7.88.1
> accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [122 bytes data]
< HTTP/2 302
< alt-svc: h3=":443"; ma=2592000
< cache-control: no-store, no-cache, must-revalidate
< content-security-policy: default-src 'self'; script-src 'self' 'nonce-dUsyeW9qbklkV2lLQWZGQmRFeERCRFlNZnR2amRtWWlXeVBZSnIyWkExbz06aTUzY3pHK0ZKeUxUY0lja1BYVVRTMHh1RitQTUl3bHVibHVOVXU2eVN5az0='; style-src 'self' 'unsafe-inline'; frame-src *; img-src * data: blob:; font-src 'self' data:; media-src *; connect-src *; object-src 'none'; base-uri 'self';
< content-type: text/html; charset=UTF-8
< date: Sat, 13 Apr 2024 10:58:48 GMT
< expires: Thu, 19 Nov 1981 08:52:00 GMT
< location: https://nc.mydomain.com/login
< pragma: no-cache
< referrer-policy: no-referrer
< server: Caddy
< server: Caddy
< server: Apache
< set-cookie: oc_sessionPassphrase=ttW0l%2FVHeyf3BIm%2BSt3aVXZl5i1wdnSiuY3fu24%2BHlPfagY9%2BJdGMnyBhSKMqxmNpwb1sFK%2BS0IkuA2%2BrYBWPYdUjabkybaANYWNeL5arcxio51PTwju3zIK4UXt%2B5kj; path=/; secure; HttpOnly; SameSite=Lax
< set-cookie: __Host-nc_sameSiteCookielax=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=lax
< set-cookie: __Host-nc_sameSiteCookiestrict=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=strict
< set-cookie: ocoa4aex63uo=a34c133374rlbplais8d3ltkg0; path=/; secure; HttpOnly; SameSite=Lax
< strict-transport-security: max-age=63072000
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< x-permitted-cross-domain-policies: none
< x-robots-tag: noindex, nofollow
< x-xss-protection: 1; mode=block
< content-length: 0
<
{ [0 bytes data]
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
* Connection #1 to host nc.mydomain.com left intact
* Issue another request to this URL: 'https://nc.mydomain.com/login'
* Found bundle for host: 0x556e2c152190 [can multiplex]
* Re-using existing connection #1 with host nc.mydomain.com
* h2h3 [:method: GET]
* h2h3 [:path: /login]
* h2h3 [:scheme: https]
* h2h3 [:authority: nc.mydomain.com]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 3 (easy handle 0x556e2c15a790)
} [5 bytes data]
> GET /login HTTP/2
> Host: nc.mydomain.com
> user-agent: curl/7.88.1
> accept: */*
>
{ [5 bytes data]
< HTTP/2 200
< alt-svc: h3=":443"; ma=2592000
< cache-control: no-cache, no-store, must-revalidate
< content-security-policy: default-src 'none';base-uri 'none';manifest-src 'self';script-src 'self';style-src 'self' 'unsafe-inline';img-src 'self' data: blob: https://*.tile.openstreetmap.org;font-src 'self' data:;connect-src 'self';media-src 'self';frame-src 'self';frame-ancestors 'self';form-action 'self'
< content-type: text/html; charset=UTF-8
< date: Sat, 13 Apr 2024 10:58:48 GMT
< expires: Thu, 19 Nov 1981 08:52:00 GMT
< feature-policy: autoplay 'self';camera 'none';fullscreen 'self';geolocation 'none';microphone 'none';payment 'none'
< pragma: no-cache
< referrer-policy: no-referrer
< server: Caddy
< server: Caddy
< server: Apache
< set-cookie: oc_sessionPassphrase=TWd2ELZq7XpoacpP99eyJiOYk6NxQ6igFIzvOBvvuumFO3Q16ZHCBptsSbU7xASauPL1UIhVUr%2Frr9Q%2BrLdnmlj34hTjtGYzJ05ver5uxKye0tbMNL2llJ4rWpHXupkg; path=/; secure; HttpOnly; SameSite=Lax
< set-cookie: __Host-nc_sameSiteCookielax=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=lax
< set-cookie: __Host-nc_sameSiteCookiestrict=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=strict
< set-cookie: ocoa4aex63uo=kccrehp5la8rn57bj2913kqkhe; path=/; secure; HttpOnly; SameSite=Lax
< strict-transport-security: max-age=63072000
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< x-permitted-cross-domain-policies: none
< x-request-id: yyLgXzH0OWmAA7Prw2My
< x-robots-tag: noindex, nofollow
< x-xss-protection: 1; mode=block
<
{ [5 bytes data]
100 12185 0 12185 0 0 59047 0 --:--:-- --:--:-- --:--:-- 59047
* Connection #1 to host nc.mydomain.com left intact
3. Caddy version:
v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=
4. How I installed and ran Caddy:
a. System environment:
System on VPS:
Operating System: Debian GNU/Linux 12 (bookworm)
Kernel: Linux 6.1.0-18-cloud-amd64
Architecture: x86-64
System on WG client:
Operating System: Debian GNU/Linux 12 (bookworm)
Kernel: Linux 6.2.16-19-pve
Architecture: x86-64
b. Command:
Installed Caddy with the help of the standard docs:
sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy
d. My complete Caddy config:
WG client
Caddyfile:
# (headers) {
# header {
# Permissions-Policy interest-cohort=()
# Strict-Transport-Security "max-age=31536000; includeSubdomains"
# X-XSS-Protection "1; mode=block"
# X-Content-Type-Options "nosniff"
# X-Robots-Tag noindex, nofollow
# Referrer-Policy "same-origin"
# Content-Security-Policy "frame-ancestors mydomain.com *.mydomain.com"
# -Server
# Permissions-Policy "geolocation=(self mydomain.com *.mydomain.com), microphone=()"
# }
# }
# The Caddyfile is an easy way to configure your Caddy web server
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace ":80" below with your
# domain name.
# :80 {
# Set this path to your site's directory.
# root * /usr/share/caddy
# Enable the static file server.
# file_server
# Another common task is to set up a reverse proxy:
# reverse_proxy localhost:8080
# Or serve a PHP site through php-fpm:
# php_fastcgi localhost:9000
# }
# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile
# {
# disable_auto_cert_gen
# }
http://nc.mydomain.com {
reverse_proxy http://192.168.178.228
}
WG config:
[Interface]
Address = 10.0.0.2/24
DNS = 192.168.178.250
#SaveConfig = true
PostUp = iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE; iptables -A FORWARD --in-interface wg0 -j ACCEPT
PostDown = iptables -D FORWARD --in-interface wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
PrivateKey = <keyhere>
[Peer]
PublicKey = <keyhere>
AllowedIPs = 0.0.0.0/0
Endpoint = 1.2.3.4:55107
PersistentKeepalive = 25
VPS:
Caddyfile:
(hsts) {
header Strict-Transport-Security max-age=63072000
}
# (headers) {
# header {
# Permissions-Policy interest-cohort=()
# Strict-Transport-Security "max-age=31536000; includeSubdomains"
# X-XSS-Protection "1; mode=block"
# X-Content-Type-Options "nosniff"
# X-Robots-Tag noindex, nofollow
# Referrer-Policy "same-origin"
# Content-Security-Policy "frame-ancestors mydomain.com *.mydomain.com"
# -Server
# Permissions-Policy "geolocation=(self mydomain.com *.mydomain.com), microphone=()"
# }
# }
# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace ":80" below with your
# domain name.
home.mydomain.com {
# Set this path to your site's directory.
root * /usr/share/caddy
# Enable the static file server.
file_server
# Another common task is to set up a reverse proxy:
# reverse_proxy localhost:8080
# Or serve a PHP site through php-fpm:
# php_fastcgi localhost:9000
}
# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile
nc.mydomain.com {
reverse_proxy http://10.0.0.2
redir /.well-known/carddav /remote.php/dav 301
redir /.well-known/caldav /remote.php/dav 301
import hsts
}
vw.mydomain.com {
reverse_proxy http://192.168.178.227
import hsts
}
WG config:
[Interface]
Address = 10.0.0.1/24
#SaveConfig = true
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o ens6 -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o ens6 -j MASQUERADE
ListenPort = 55107
PrivateKey = <keyhere>
[Peer]
PublicKey = <keyhere>
AllowedIPs = 10.0.0.2/32, 192.168.178.0/24
PersistentKeepalive = 25
5. Links to relevant resources:
I tried the Caddy configs from the following page, but that did not result in any success, I only referred to the Caddyfiles, the WG tunnel is set up a bit different: Using Wireguard to Tunnel All Traffic through a VPS to Home
Hopefully, I have everything sorted out by now. If there are any questions, please let me know, otherwise I hope someone can help me because she/he had the same problem and found a solution to this.
its a complex setup. I would go from the lowest layers up to the highest. Without having the basic network working properly I wouldn’t try to troubleshoot a layer 7 application.