Running Subdomains Behind Tailscale

Help
1

1. Output of caddy version:

v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=

2. How I run Caddy:

a. System environment:

Raspberry Pi OS 11 through APT and systemD

b. Command:

● caddy.service - Caddy
     Loaded: loaded (/lib/systemd/system/caddy.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-11-10 14:21:45 EST; 1min 54s ago
       Docs: https://caddyserver.com/docs/
   Main PID: 19536 (caddy)
      Tasks: 11 (limit: 1830)
        CPU: 418ms
     CGroup: /system.slice/caddy.service
             └─19536 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108105.8305492,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"servarr.gappyland.com","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Nov 10 14:21:47 deb-rpi-rproxy caddy[19536]: {"level":"error","ts":1668108107.8311334,"logger":"http.acme_client","msg":"challenge failed","identifier":"servarr.gappyland.com","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","det>
Nov 10 14:21:47 deb-rpi-rproxy caddy[19536]: {"level":"error","ts":1668108107.83122,"logger":"http.acme_client","msg":"validating authorization","identifier":"servarr.gappyland.com","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: SE>
Nov 10 14:21:49 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108109.0362806,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"servarr.gappyland.com","challenge_type":"tls-alpn-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Nov 10 14:21:50 deb-rpi-rproxy caddy[19536]: {"level":"error","ts":1668108110.396576,"logger":"http.acme_client","msg":"challenge failed","identifier":"servarr.gappyland.com","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:dns","title":"",">
Nov 10 14:21:50 deb-rpi-rproxy caddy[19536]: {"level":"error","ts":1668108110.396922,"logger":"http.acme_client","msg":"validating authorization","identifier":"servarr.gappyland.com","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: S>
Nov 10 14:21:50 deb-rpi-rproxy caddy[19536]: {"level":"error","ts":1668108110.3972235,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"servarr.gappyland.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:p>
Nov 10 14:21:50 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108110.40008,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["servarr.gappyland.com"],"ca":"https://acme.zerossl.com/v2/DV90","account":"caddy@zerossl.com"}
Nov 10 14:21:50 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108110.4001958,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["servarr.gappyland.com"],"ca":"https://acme.zerossl.com/v2/DV90","account":"caddy@zerossl.com"}
Nov 10 14:21:52 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108112.25368,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"servarr.gappyland.com","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}

c. Service/unit/compose file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

gappyland.com {
        root * /var/www/html
        encode gzip
        file_server
        reverse_proxy /jellyfin/* 192.168.5.50:8096
        reverse_proxy /komga* 192.168.5.50:8080
}

servarr.gappyland.com {
        root * /var/www/html
        reverse_proxy /radarr* 192.168.5.50:7878
        reverse_proxy /sonarr* 192.168.5.50:8989
}

3. The problem I’m having:

I am trying to set up a subdomain which runs behind Tailscale, but I am getting “challenge failed” and “SERVFAIL” errors in the logs and the site shows “Error code: SSL_ERROR_INTERNAL_ERROR_ALERT”. I’ve set my Google Domain’s A and AAAA records like this:

gappyland.com - A - Public IPv4
gappyland.com - AAAA - Public IPv6
servarr.gappyland.com - A - Tailscale IPv4
servarr.gappyland.com - AAAA - Tailscale IPv6

So far as I understand, Caddy will request SSL certs for all domains defined in the Caddyfile but I don’t think that is happening properly. I’m not sure if the issue is with the Caddyfile (maybe I need to define a certain TLS option manually?), if it’s because I’m running the subdomain through Tailscale, or if my DNS records are not set up properly.

4. Error messages and/or full log output:

Nov 10 14:21:45 deb-rpi-rproxy systemd[1]: Starting Caddy...
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: caddy.HomeDir=/var/lib/caddy
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: caddy.Version=v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: runtime.GOOS=linux
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: runtime.GOARCH=arm64
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: runtime.Compiler=gc
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: runtime.NumCPU=4
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: runtime.GOMAXPROCS=4
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: runtime.Version=go1.19.2
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: os.Getwd=/
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: LANG=en_GB.UTF-8
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: NOTIFY_SOCKET=/run/systemd/notify
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: HOME=/var/lib/caddy
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: LOGNAME=caddy
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: USER=caddy
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: INVOCATION_ID=7f88f456fa194d238f69729cbeaad19d
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: JOURNAL_STREAM=8:42250
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108105.267036,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: {"level":"warn","ts":1668108105.2708948,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":34}
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108105.273863,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//[::1]:2019","//127.0.0.1:2019","//localhost:2019"]}
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108105.274323,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108105.2743795,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108105.2752733,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x4000700f50"}
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108105.2760944,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108105.276328,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108105.2764971,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108105.2766314,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108105.2766514,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["gappyland.com","servarr.gappyland.com"]}
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108105.2773557,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108105.2810223,"logger":"tls","msg":"finished cleaning storage units"}
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108105.2834857,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108105.2841089,"logger":"tls.obtain","msg":"acquiring lock","identifier":"servarr.gappyland.com"}
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108105.2847242,"msg":"serving initial configuration"}
Nov 10 14:21:45 deb-rpi-rproxy systemd[1]: Started Caddy.
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108105.2990992,"logger":"tls.obtain","msg":"lock acquired","identifier":"servarr.gappyland.com"}
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108105.2996101,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"servarr.gappyland.com"}
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108105.3021734,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["servarr.gappyland.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108105.3022444,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["servarr.gappyland.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
Nov 10 14:21:45 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108105.8305492,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"servarr.gappyland.com","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Nov 10 14:21:47 deb-rpi-rproxy caddy[19536]: {"level":"error","ts":1668108107.8311334,"logger":"http.acme_client","msg":"challenge failed","identifier":"servarr.gappyland.com","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: SERVFAIL looking up A for servarr.gappyland.com - the domain's nameservers may be malfunctioning; DNS problem: SERVFAIL looking up AAAA for servarr.gappyland.com - the domain's nameservers may be malfunctioning","instance":"","subproblems":[]}}
Nov 10 14:21:47 deb-rpi-rproxy caddy[19536]: {"level":"error","ts":1668108107.83122,"logger":"http.acme_client","msg":"validating authorization","identifier":"servarr.gappyland.com","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: SERVFAIL looking up A for servarr.gappyland.com - the domain's nameservers may be malfunctioning; DNS problem: SERVFAIL looking up AAAA for servarr.gappyland.com - the domain's nameservers may be malfunctioning","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/818117387/142816853477","attempt":1,"max_attempts":3}
Nov 10 14:21:49 deb-rpi-rproxy caddy[19536]: {"level":"info","ts":1668108109.0362806,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"servarr.gappyland.com","challenge_type":"tls-alpn-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}

5. What I already tried:

I set up my base domain on the Tailscale IP and it functioned as expect, my issue is only setting it up on a subdomain.

6. Links to relevant resources:

None

2

The crux of the problem is right in the logs:

Works fine for me though, and checks out at letsdebug.net, so maybe something has improved since then. Perhaps try again? (Use staging endpoint for testing.)

3

Did it really work? It shouldn’t, only members of my Tailscale network should be able to access it. I tried that site and I got “A private, inaccessible, IANA/IETF-reserved IP address was found”, so that does point to the Tailscale IP being unroutable on the internet. I did however use this guide to set up Tailscale with Caddy and I got it to work on the base domain, so what I am not understanding is what I need to do to achieve the same thing on a subdomain.

4

I didn’t try accessing it. To clarify, I was saying that the DNS looks set up properly, and that letsdebug.net doesn’t show any errors.

What happens if you try again? (Use the staging endpoint to test.)

5

Accessing https://servarr.gappyland.com directly via browser (Firefox) I get SSL_ERROR_INTERNAL_ERROR_ALERT

Using letsdebug.net I get a good check on DNS-1, but I receive an HTTP-01 error
A private, inaccessible, IANA/IETF-reserved IP address was found for servarr.gappyland.com. Let's Encrypt will always fail HTTP validation for any domain that is pointing to an address that is not routable on the internet. You should either remove this address and replace it with a public one or use the DNS validation method instead.

and a TLS-ALPN-01 error
A test authorization for servarr.gappyland.com to the Let's Encrypt staging service has revealed issues that may prevent any certificate for this domain being issued. DNS problem: SERVFAIL looking up A for servarr.gappyland.com - the domain's nameservers may be malfunctioning; DNS problem: SERVFAIL looking up AAAA for servarr.gappyland.com - the domain's nameservers may be malfunctioning

I also cannot connect via wget, it does not resolve the hostname.

I am going to try and use the Google DNS plugin with the TLS directive when I get a chance today, as I just noticed the or use the DNS validation method instead blurb in the error. Thank you for the help.

6

Your domain can ONLY get a cert from the DNS-01 challenge as expected, because of the internal-only IP address.

If you try the DNS challenge again, it should work IMO.

7

I think I understand what you are saying now. So the way to use a DNS challenge specifically is use the TLS directive with a plugin module? Because I am now realizing there is no caddy-dns module for Google Domains, just Google Cloud DNS and I don’t see it in this list so that may not be an option. Do you think there is any way to resolve this without switch domain providers?

1 Like
8

I was able to resolve it using the TLS directive and switching my domain to Cloudflare. Thanks for the help.

1 Like
9

Google Domains doesn’t have an API yet. (Ask me again about that in a couple months.)

Glad you got it working!

10

This topic was automatically closed after 30 days. New replies are no longer allowed.