Unable to resolve internal domains (302 redirect issue?)

Help
1

1. The problem I’m having:

I had a working Caddy instance with open 80/443 ports with cloudflare where all domains were internal resolving fine. However I decided to close the ports down and instead use xcaddy with cf plugin.
I managed to work through multiple issues and troubleshooting to get services installed & running fine, yet I cannot resolve my internal domains.

On Cloudflare I simply have A record with * pointing to my local lan IP where my caddy install is with DNS only reserved IP.

2. Error messages and/or full log output:

root@Caddy:~# 
root@Caddy:~# status caddy
-bash: status: command not found
root@Caddy:~# service caddy status
* caddy.service - Caddy
     Loaded: loaded (/lib/systemd/system/caddy.service; enabled; preset: enabled)
     Active: active (running) since Sat 2024-09-14 11:07:55 UTC; 3min 21s ago
       Docs: https://caddyserver.com/docs/
   Main PID: 324 (caddy)
      Tasks: 7 (limit: 9190)
     Memory: 10.9M
        CPU: 95ms
     CGroup: /system.slice/caddy.service
             `-324 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

Sep 14 11:07:55 Caddy caddy[324]: {"level":"info","ts":1726312075.7528937,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Sep 14 11:07:55 Caddy caddy[324]: {"level":"info","ts":1726312075.7529747,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Si>
Sep 14 11:07:55 Caddy caddy[324]: {"level":"info","ts":1726312075.7530856,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Sep 14 11:07:55 Caddy caddy[324]: {"level":"info","ts":1726312075.7531323,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
Sep 14 11:07:55 Caddy caddy[324]: {"level":"info","ts":1726312075.7531385,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["photos.nznux.com","titan.nznux.com","paperless.nznux.com","jellyfin.nznux.com","pr>
Sep 14 11:07:55 Caddy caddy[324]: {"level":"info","ts":1726312075.7564034,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/var/lib/caddy/.local/share/caddy","instance":"1913ad1d-6056->
Sep 14 11:07:55 Caddy caddy[324]: {"level":"info","ts":1726312075.7564645,"logger":"tls","msg":"finished cleaning storage units"}
Sep 14 11:07:55 Caddy caddy[324]: {"level":"info","ts":1726312075.7565536,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Sep 14 11:07:55 Caddy caddy[324]: {"level":"info","ts":1726312075.7567377,"msg":"serving initial configuration"}
Sep 14 11:07:55 Caddy systemd[1]: Started caddy.service - Caddy.

root@Caddy:~# curl -v https://jellyfin.nznux.com
*   Trying 192.168.1.11:443...
* Connected to jellyfin.nznux.com (192.168.1.11) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=jellyfin.nznux.com
*  start date: Sep 14 10:07:24 2024 GMT
*  expire date: Dec 13 10:07:23 2024 GMT
*  subjectAltName: host "jellyfin.nznux.com" matched cert's "jellyfin.nznux.com"
*  issuer: C=US; O=Let's Encrypt; CN=E5
*  SSL certificate verify ok.
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: jellyfin.nznux.com]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x6398a767ece0)
> GET / HTTP/2
> Host: jellyfin.nznux.com
> user-agent: curl/7.88.1
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 302 
< alt-svc: h3=":443"; ma=2592000
< date: Sat, 14 Sep 2024 11:11:51 GMT
< location: web/
< server: Caddy
< server: Kestrel
< content-length: 0
< 
* Connection #0 to host jellyfin.nznux.com left intact

Here are the logs from journalctl -xeu caddy.service

root@Caddy:/etc/caddy# journalctl -xeu caddy.service
-- A stop job for unit caddy.service has finished.
-- 
-- The job identifier is 118 and the job result is done.
Sep 14 11:07:55 Caddy systemd[1]: Starting caddy.service - Caddy...
-- Subject: A start job for unit caddy.service has begun execution
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit caddy.service has begun execution.
-- 
-- The job identifier is 119.
Sep 14 11:07:55 Caddy caddy[324]: caddy.HomeDir=/var/lib/caddy
Sep 14 11:07:55 Caddy caddy[324]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Sep 14 11:07:55 Caddy caddy[324]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Sep 14 11:07:55 Caddy caddy[324]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Sep 14 11:07:55 Caddy caddy[324]: caddy.Version=v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=
Sep 14 11:07:55 Caddy caddy[324]: runtime.GOOS=linux
Sep 14 11:07:55 Caddy caddy[324]: runtime.GOARCH=amd64
Sep 14 11:07:55 Caddy caddy[324]: runtime.Compiler=gc
Sep 14 11:07:55 Caddy caddy[324]: runtime.NumCPU=2
Sep 14 11:07:55 Caddy caddy[324]: runtime.GOMAXPROCS=2
Sep 14 11:07:55 Caddy caddy[324]: runtime.Version=go1.23.1
Sep 14 11:07:55 Caddy caddy[324]: os.Getwd=/
Sep 14 11:07:55 Caddy caddy[324]: LANG=C
Sep 14 11:07:55 Caddy caddy[324]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Sep 14 11:07:55 Caddy caddy[324]: NOTIFY_SOCKET=/run/systemd/notify
Sep 14 11:07:55 Caddy caddy[324]: HOME=/var/lib/caddy
Sep 14 11:07:55 Caddy caddy[324]: LOGNAME=caddy
Sep 14 11:07:55 Caddy caddy[324]: USER=caddy
Sep 14 11:07:55 Caddy caddy[324]: INVOCATION_ID=1f79bc3f59524c4694c3bb71199f7635
Sep 14 11:07:55 Caddy caddy[324]: JOURNAL_STREAM=8:43209984
Sep 14 11:07:55 Caddy caddy[324]: SYSTEMD_EXEC_PID=324
Sep 14 11:07:55 Caddy caddy[324]: {"level":"info","ts":1726312075.7455206,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
Sep 14 11:07:55 Caddy caddy[324]: {"level":"info","ts":1726312075.7491717,"msg":"adapted config to JSON","adapter":"caddyfile"}
Sep 14 11:07:55 Caddy caddy[324]: {"level":"warn","ts":1726312075.7492056,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":11}
Sep 14 11:07:55 Caddy caddy[324]: {"level":"info","ts":1726312075.7514281,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
Sep 14 11:07:55 Caddy caddy[324]: {"level":"info","ts":1726312075.7519143,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","htt>
Sep 14 11:07:55 Caddy caddy[324]: {"level":"info","ts":1726312075.7519479,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Sep 14 11:07:55 Caddy caddy[324]: {"level":"warn","ts":1726312075.751962,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":8>
Sep 14 11:07:55 Caddy caddy[324]: {"level":"info","ts":1726312075.7522135,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000376a80"}
Sep 14 11:07:55 Caddy caddy[324]: {"level":"info","ts":1726312075.7528937,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Sep 14 11:07:55 Caddy caddy[324]: {"level":"info","ts":1726312075.7529747,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Si>
Sep 14 11:07:55 Caddy caddy[324]: {"level":"info","ts":1726312075.7530856,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Sep 14 11:07:55 Caddy caddy[324]: {"level":"info","ts":1726312075.7531323,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
Sep 14 11:07:55 Caddy caddy[324]: {"level":"info","ts":1726312075.7531385,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["photos.nznux.com","titan.nznux.com","paperless.nznux.com","jellyfin.nznux.com","pr>
Sep 14 11:07:55 Caddy caddy[324]: {"level":"info","ts":1726312075.7564034,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/var/lib/caddy/.local/share/caddy","instance":"1913ad1d-6056->
Sep 14 11:07:55 Caddy caddy[324]: {"level":"info","ts":1726312075.7564645,"logger":"tls","msg":"finished cleaning storage units"}
Sep 14 11:07:55 Caddy caddy[324]: {"level":"info","ts":1726312075.7565536,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Sep 14 11:07:55 Caddy caddy[324]: {"level":"info","ts":1726312075.7567377,"msg":"serving initial configuration"}
Sep 14 11:07:55 Caddy systemd[1]: Started caddy.service - Caddy.
-- Subject: A start job for unit caddy.service has finished successfully
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit caddy.service has finished successfully.
-- 
-- The job identifier is 119.

3. Caddy version:

caddy version
v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy:

a. System environment:

Running Debian 12 on Proxmox and installed Caddy following guide here: Install — Caddy Documentation

Installed xcaddy following guide here: GitHub - caddy-dns/cloudflare: Caddy module: dns.providers.cloudflare
& this video https://youtu.be/9zm8d0jXY9o?si=rcBKCHDLBhImXVKI

b. Command:

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

c. Service/unit/compose file:

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

d. My complete Caddy config:

{
        acme_dns cloudflare myCFToken
}


:80 {
        # Set this path to your site's directory.
        root * /usr/share/caddy

        # Enable the static file server.
        file_server

        # Another common task is to set up a reverse proxy:
        # reverse_proxy localhost:8080

        # Or serve a PHP site through php-fpm:
        # php_fastcgi localhost:9000
}

# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile

#jellyfin.nznux.com {
#       reverse_proxy 192.168.1.2:8096
#       tls {
#               dns cloudflare {env.CLOUDFLARE_API_TOKEN}
#       }
#}

paperless.nznux.com {
        reverse_proxy 192.168.1.13:8000
}

jellyfin.nznux.com {
        reverse_proxy 192.168.1.2:8096
}

pihole.nznux.com {
        encode zstd gzip
        redir / /admin{uri}
        reverse_proxy 192.168.1.5:80
        # reverse_proxy 192.168.1.5:81
}

photos.nznux.com {
        reverse_proxy 192.168.1.2:5001
}

titan.nznux.com {
        reverse_proxy 192.168.1.2:5001 {
                transport http {
                        tls_insecure_skip_verify
                }
        }
}

proxmox.nznux.com {
        reverse_proxy 192.168.1.12:8006 {
                transport http {
                        tls_insecure_skip_verify
                }
        }
}

5. Links to relevant resources:

2

There appears to be some type of 302 redirect at each service e.g jellyfin and paperless-ngx below

root@Caddy:/etc/caddy# **wget -vv jellyfin.nznux.com**
--2024-09-15 03:40:31--  http://jellyfin.nznux.com/
Resolving jellyfin.nznux.com (jellyfin.nznux.com)... 192.168.1.11
Connecting to jellyfin.nznux.com (jellyfin.nznux.com)|192.168.1.11|:80... connected.
HTTP request sent, awaiting response... 308 Permanent Redirect
Location: https://jellyfin.nznux.com/ [following]
--2024-09-15 03:40:32--  https://jellyfin.nznux.com/
Connecting to jellyfin.nznux.com (jellyfin.nznux.com)|192.168.1.11|:443... connected.
HTTP request sent, awaiting response... **302 Found**
Location: web/ [following]
--2024-09-15 03:40:33--  https://jellyfin.nznux.com/web/
Reusing existing connection to jellyfin.nznux.com:443.
HTTP request sent, awaiting response... 200 OK
Length: 8893 (8.7K) [text/html]
Saving to: 'index.html'

index.html                                                  100%[==========================================================================================================================================>]   8.68K  --.-KB/s    in 0s     

root@Caddy:/etc/caddy# **wget -vv paperless.nznux.com**
--2024-09-15 03:41:40--  http://paperless.nznux.com/
Resolving paperless.nznux.com (paperless.nznux.com)... 192.168.1.11
Connecting to paperless.nznux.com (paperless.nznux.com)|192.168.1.11|:80... connected.
HTTP request sent, awaiting response... 308 Permanent Redirect
Location: https://paperless.nznux.com/ [following]
--2024-09-15 03:41:40--  https://paperless.nznux.com/
Connecting to paperless.nznux.com (paperless.nznux.com)|192.168.1.11|:443... connected.
HTTP request sent, awaiting response... **302 Found**
Location: /accounts/login/?next=/ [following]
--2024-09-15 03:41:40--  https://paperless.nznux.com/accounts/login/?next=/
Reusing existing connection to paperless.nznux.com:443.
HTTP request sent, awaiting response... 200 OK
Length: 8278 (8.1K) [text/html]
Saving to: 'index.html.1'

index.html.1                                                100%[==========================================================================================================================================>]   8.08K  --.-KB/s    in 0s      

2024-09-15 03:41:40 (80.4 MB/s) - 'index.html.1' saved [8278/8278]
3

Use curl -v for testing connections, it have much cleaner output.

I’m not sure I understand the problem you’re describing. I don’t really see any evidence of a problem.

1 Like
4

Your query is for HTTP as the verbose wget output shows, it receives a redirect from Caddy due to auto_https defaults so that you connect over HTTPS. That is then successful.

Is the content returned not correct? Both services adjust the subpath differently so it seems like they’re connecting to the services just fine? They definitely seem to be resolving DNS correctly.

If your issue is with the browser, it’s possible that it considers the connection suspicious. A public domain pointing to a private IP address is sometimes considered as a bad actor having compromised the DNS service.

Try a separate Caddyfile like this one:

example.localhost {
    tls internal
    respond "Hello world"
}

You should be able to access that via the browser, after allowing to continue pass the warning page presented from an untrusted certificate you would then get the “Hello World” text.

Similarly you could try the same via CLI with curl --insecure https://example.localhost

If that works, then try again with your Caddyfile shared, but change one of your site blocks like Jellyfin to just use the respond line I showed instead of reverse_proxy. If the CLI gives you the text response, but you can’t get that on the browser, then the public DNS to private subnet is the likely issue. It doesn’t affect localhost or other TLDs that are reserved for private use.

5

These all show that your request successfully resolved to 192.168.1.11, a HTTP request was made, upgraded to HTTPS, and then you successfully got an index file - that means DNS resolution, networking, Caddy, and your upstream server are all working perfectly fine.

That means whatever’s still going wrong is probably limited to your browser. Try in incognito, or with a different browser.

1 Like
6

@Whitestrake here is what I get opening jellyfin.nznux.com in incognito.
No change on Edge browser either

image
image967×627 11.1 KB

7 
root@Caddy:~# curl -v jellyfin.nznux.com
*   Trying 192.168.1.11:80...
* Connected to jellyfin.nznux.com (192.168.1.11) port 80 (#0)
> GET / HTTP/1.1
> Host: jellyfin.nznux.com
> User-Agent: curl/7.88.1
> Accept: */*
> 
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://jellyfin.nznux.com/
< Server: Caddy
< Date: Mon, 16 Sep 2024 08:20:37 GMT
< Content-Length: 0
< 
* Closing connection 0
root@Caddy:~#
8

You made an HTTP request with curl and got back a redirect (Location header). Make sure to use https:// in your command to actually make an HTTPS request, or use -L (or -vL to combine both) to follow the redirect (-L for Location).

1 Like
9

I replaced example.localhost with jellyfin.nznux.com and left tls internal and respond “HelloWorld”. No return on the browser but I attempted the curl option as below with following result

image

10

Interesting running curl with https now provides an error and closes the connection

root@Caddy:/etc/caddy# curl -v https://jellyfin.nznux.com
*   Trying 192.168.1.11:443...
* Connected to jellyfin.nznux.com (192.168.1.11) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS alert, internal error (592):
* OpenSSL/3.0.14: error:0A000438:SSL routines::tlsv1 alert internal error
* Closing connection 0
curl: (35) OpenSSL/3.0.14: error:0A000438:SSL routines::tlsv1 alert internal error
11

Browser result of DNS_PROBE_FINISHED_NXDOMAIN means your browser tried to resolve jellyfin.nznux.com and couldn’t for some reason.

That’s going to be a DNS issue.

Since you’re using public DNS for this, I can confirm even on my end that the records are set:

~/Projects/caddy
➜ nslookup jellyfin.nznux.com
Server:		100.100.100.100
Address:	100.100.100.100#53

Non-authoritative answer:
Name:	jellyfin.nznux.com
Address: 192.168.1.11

So what you need to troubleshoot next is whether you have a DNS cache returning a bad result, either on your PC or maybe on your router. You could try setting your local DNS to 1.1.1.1 or 8.8.8.8 or a similar service.

1 Like
12

I set DNS on my router to 8.8.8.8 & 1.1.1.1 (even turned DNS off at router) but no change. Browser still returns same message as before : Site cannot be reached.

Thinking it could be a browser issue, attempted on different browser on and on my cellphone (safari). Result is the same

Argh! I’m sure its a simple config issue just cannot figure it out

image
image849×597 10.8 KB

13

changing the DNS on the router to 8.8.8.8 however did change the curl result

root@Caddy:/etc/caddy# curl -v https://jellyfin.nznux.com
*   Trying 192.168.1.11:443...
* Connected to jellyfin.nznux.com (192.168.1.11) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=jellyfin.nznux.com
*  start date: Sep 14 10:07:24 2024 GMT
*  expire date: Dec 13 10:07:23 2024 GMT
*  subjectAltName: host "jellyfin.nznux.com" matched cert's "jellyfin.nznux.com"
*  issuer: C=US; O=Let's Encrypt; CN=E5
*  SSL certificate verify ok.
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: jellyfin.nznux.com]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x59f860075ce0)
> GET / HTTP/2
> Host: jellyfin.nznux.com
> user-agent: curl/7.88.1
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 302 
< alt-svc: h3=":443"; ma=2592000
< date: Mon, 16 Sep 2024 09:21:12 GMT
< location: web/
< server: Caddy
< server: Kestrel
< content-length: 0
< 
* Connection #0 to host jellyfin.nznux.com left intact
14

I had a quick search, the term I was thinking of is called “DNS Rebind Protection”, where a public DNS resolving to a private IP is flagged as suspicious (reference).

Might be related to the issue you’re experiencing?

UPDATE: Actually yes, your findings so far align with these results:

Tailscale has a dedicated docs page for the problem. The 2nd link notes browsers using DoH by default (which might be why you have a problem on browsers from the same machine curl was working?).

1 Like
15

This post helped with some additional learnings and led me to going back to my router and changing dns to pihole and 8.8.8.8.
I did a dns flush on my pc and also pointed its dns to 8.8.8.8 and 1.1.1.1.
Next I updated pihole and restarted the VM.
Then under Pihole → DNS → Advanced DNS setting → disable Never forward non-FQDN A and AAAA queries
It all works now and pages are resolving fine both via PC and mobile.
Thank you team!

1 Like
16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.