Help! Docker caddy suddenly no longer can find DNS records?

Help
17

I ran a very basic setup to proof-of-concept this. The machine is a NixOS VM running in Proxmox and it has Docker installed and Tailscale installed and joined to my Tailnet.

Here is the entire configuration:

whitestrake at 🌐 pascal in /opt/docker
❯ cat docker-compose.yml
configs:
  Caddyfile:
    content: |
      {
        tailscale {
          auth_key [snip]
          state_dir /tailscale
        }
      }
      https://whoami.fell-monitor.ts.net {
        bind tailscale/whoami
        reverse_proxy whoami
      }

volumes:
  caddy:
  tailscale:

services:
  caddy:
    build:
      dockerfile_inline: |
        FROM caddy:2-builder AS builder
        RUN xcaddy build latest \
          --with github.com/tailscale/caddy-tailscale
        FROM caddy:2
        COPY --from=builder /usr/bin/caddy /usr/bin/caddy
    restart: unless-stopped
    volumes:
      - caddy:/data
      - tailscale:/tailscale
    configs:
      - source: Caddyfile
        target: /etc/caddy/Caddyfile

  whoami:
    image: traefik/whoami
    restart: unless-stopped

Here is the logs from startup:

whitestrake at 🌐 pascal in /opt/docker
❯ docker compose up
Attaching to caddy-1, whoami-1
whoami-1  | 2024/08/01 00:02:09 Starting up on port 80
caddy-1   | {"level":"info","ts":1722470529.3187523,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
caddy-1   | {"level":"info","ts":1722470529.319486,"msg":"adapted config to JSON","adapter":"caddyfile"}
caddy-1   | {"level":"warn","ts":1722470529.3195,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
caddy-1   | {"level":"info","ts":1722470529.32001,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy-1   | {"level":"info","ts":1722470529.3201323,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy-1   | {"level":"info","ts":1722470529.3201504,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy-1   | {"level":"info","ts":1722470529.3202755,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000149100"}
caddy-1   | {"level":"info","ts":1722470529.3373108,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/data/caddy"}
caddy-1   | {"level":"info","ts":1722470529.3374925,"logger":"tls","msg":"finished cleaning storage units"}
caddy-1   | {"level":"info","ts":1722470529.3393836,"logger":"tailscale","msg":"tsnet running state path /tailscale/whoami/tailscaled.state"}
caddy-1   | {"level":"info","ts":1722470529.357314,"logger":"tailscale","msg":"tsnet starting with hostname \"whoami\", varRoot \"/tailscale/whoami\""}
caddy-1   | {"level":"info","ts":1722470530.3694124,"logger":"tailscale","msg":"LocalBackend state is NeedsLogin; running StartLoginInteractive..."}
caddy-1   | {"level":"info","ts":1722470530.3694654,"logger":"http","msg":"enabling HTTP/3 listener","addr":"whoami:443"}
caddy-1   | {"level":"info","ts":1722470533.2040713,"msg":"connection doesn't allow setting of receive buffer size. Not a *net.UDPConn?. See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
caddy-1   | {"level":"info","ts":1722470533.204147,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddy-1   | {"level":"info","ts":1722470533.2041695,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddy-1   | {"level":"info","ts":1722470533.2043512,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy-1   | {"level":"info","ts":1722470533.204362,"msg":"serving initial configuration"}
caddy-1   | {"level":"info","ts":1722470535.3704717,"logger":"tailscale","msg":"AuthLoop: state is Running; done"}

And here is the result from curling the new node:

whitestrake at 🌐 pascal in /opt/docker
❯ curl https://whoami.fell-monitor.ts.net
Hostname: 78f2a9f88e90
IP: 127.0.0.1
IP: 172.18.0.3
RemoteAddr: 172.18.0.2:42926
GET / HTTP/1.1
Host: whoami.fell-monitor.ts.net
User-Agent: curl/8.7.1
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 100.106.140.61
X-Forwarded-Host: whoami.fell-monitor.ts.net
X-Forwarded-Proto: https

I can also see that node keys were persisted:

whitestrake at 🌐 pascal in /opt/docker
❯ dc run --rm caddy ls -al /tailscale/whoami
total 24
drwx------    4 root     root          4096 Aug  1 00:09 .
drwxr-xr-x    3 root     root          4096 Aug  1 00:02 ..
drwx------    2 root     root          4096 Aug  1 00:02 certs
drwx------    3 root     root          4096 Aug  1 00:02 files
-rw-------    1 root     root           209 Aug  1 00:02 tailscaled.log.conf
-rw-------    1 root     root             0 Aug  1 00:09 tailscaled.log1.txt
-rw-------    1 root     root             0 Aug  1 00:09 tailscaled.log2.txt
-rw-------    1 root     root          2775 Aug  1 00:09 tailscaled.state
2 Likes