mDNS lookup failing for reverse_proxy when updating Caddy

1. Caddy version (caddy version):

v2.0.0-beta.14 h1:QX1hRMfTA5sel53o5SuON1ys50at6yuSAnPr56sLeK8=

and

v2.4.1 h1:kAJ0JB5Xk5gPdTH/27S5cyoMGqD5lBAe9yZ8zTjVJa0=

2. How I run Caddy:

a. System environment:

Debian vm running as a systemd service
Linux caddy 5.10.0-6-amd64 #1 SMP Debian 5.10.28-1 (2021-04-09) x86_64 GNU/Linux

b. Command:

systemctl start caddy

c. Service/unit/compose file:

[Unit]
Description=Caddy Web Server
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
User=caddy
Group=caddy
ExecStart=/usr/local/bin/caddy run --config /etc/caddy/Caddyfile --adapter caddyfile --environ
ExecReload=/usr/local/bin/caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile
TimeoutStopSec=5s
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

*.example.com {
    tls /etc/letsencrypt/live/example.com/fullchain.pem /etc/letsencrypt/live/example.com/privkey.pem {
        protocols tls1.2 tls1.3
        ciphers TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
    }

    @www {
        protocol https
        host www.example.com
    }

    handle @www {
        reverse_proxy nginx.local:443 {
            transport http {
                tls
            }
        }
    }

    @webmail {
        protocol https
        host webmail.example.com
    }

    handle @webmail {
        reverse_proxy roundcube.local:80
    }
}

https://example.com {
    tls /etc/letsencrypt/live/example.com/fullchain.pem /etc/letsencrypt/live/example.com/privkey.pem
    redir https://www.example.com{uri}
}

http://example.com http://www.example.com http://webmail.example.com {
    redir https://{host}{uri}
}

http://mail.example.com {
    reverse_proxy {
        to mail.local:80 192.168.1.6:80
    }
}

3. The problem I’m having:

mDNS resolution for the nginx.local address seems to fail with a 502 error on the later version of Caddy v2.4.1, but works as expected in the older Caddy v2.0.0 beta14

4. Error messages and/or full log output:

The log for v2.4.1 shows a DNS request going to the gateway and failing

Jun 09 00:39:08 caddy caddy[19199]: {"level":"error","ts":1623213548.9997115,"logger":"http.log.error","msg":"dial tcp: lookup nginx.local on 192.168.1.1:53: no such host","request":{"remote_addr":"1.2.3.4:49488","proto":"HTTP/2.0","method":"GET","host":"www.example.com","uri":"/","headers":{"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Gpc":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36 Edg/91.0.864.41"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Dnt":["1"],"Sec-Ch-Ua":["\" Not;A Brand\";v=\"99\", \"Microsoft Edge\";v=\"91\", \"Chromium\";v=\"91\""],"Sec-Ch-Ua-Mobile":["?0"],"Upgrade-Insecure-Requests":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"www.example.com"}},"duration":0.009322533,"status":502,"err_id":"pfus6m109","err_trace":"reverseproxy.statusError (reverseproxy.go:861)"}

Nothing is emitted in the log when running the older version because the request is resolving correctly.

5. What I already tried:

Staying on the old version works for now

6. Links to relevant resources:

N/A

That’s too wide of a range of versions to really have a clue what might have broken this for you. Could you try tracking down exactly the version where it stopped working?

Most efficient way to do this is to pick a version roughly in the middle between the two. If it works, then try a version that’s newer; if it didn’t work then try a version that’s older; both half way between the current version and the nearest in that direction. Basically a binary search.

The results are not what I was expecting or hoping. I’m getting the same error back through every version including 2.0.0 beta14. It seems the one that works for me also had the Cloudflare module built in. I forgot this was included because I had decided to use a different solution.

A list modules for the working version

caddy.logging.encoders.console
caddy.logging.encoders.filter
caddy.logging.encoders.filter.delete
caddy.logging.encoders.filter.ip_mask
caddy.logging.encoders.json
caddy.logging.encoders.logfmt
caddy.logging.encoders.string
caddy.logging.writers.discard
caddy.logging.writers.file
caddy.logging.writers.net
caddy.logging.writers.stderr
caddy.logging.writers.stdout
caddy.storage.file_system
http
http.authentication.hashes.bcrypt
http.authentication.hashes.scrypt
http.authentication.providers.http_basic
http.encoders.brotli
http.encoders.gzip
http.encoders.zstd
http.handlers.authentication
http.handlers.cache
http.handlers.encode
http.handlers.error
http.handlers.file_server
http.handlers.headers
http.handlers.request_body
http.handlers.reverse_proxy
http.handlers.rewrite
http.handlers.static_response
http.handlers.subroute
http.handlers.templates
http.handlers.vars
http.matchers.file
http.matchers.header
http.matchers.header_regexp
http.matchers.host
http.matchers.method
http.matchers.not
http.matchers.path
http.matchers.path_regexp
http.matchers.protocol
http.matchers.query
http.matchers.remote_ip
http.matchers.vars
http.matchers.vars_regexp
http.reverse_proxy.circuit_breakers.local
http.reverse_proxy.selection_policies.first
http.reverse_proxy.selection_policies.header
http.reverse_proxy.selection_policies.ip_hash
http.reverse_proxy.selection_policies.least_conn
http.reverse_proxy.selection_policies.random
http.reverse_proxy.selection_policies.random_choose
http.reverse_proxy.selection_policies.round_robin
http.reverse_proxy.selection_policies.uri_hash
http.reverse_proxy.transport.fastcgi
http.reverse_proxy.transport.http
http.reverse_proxy.transport.http_ntlm
tls
tls.certificate_selection.custom
tls.certificates.automate
tls.certificates.load_files
tls.certificates.load_folders
tls.certificates.load_pem
tls.dns.cloudflare
tls.handshake_match.sni
tls.management.acme
tls.stek.distributed
tls.stek.standard

And vanilla

caddy.logging.encoders.console
caddy.logging.encoders.filter
caddy.logging.encoders.filter.delete
caddy.logging.encoders.filter.ip_mask
caddy.logging.encoders.json
caddy.logging.encoders.logfmt
caddy.logging.encoders.string
caddy.logging.writers.discard
caddy.logging.writers.file
caddy.logging.writers.net
caddy.logging.writers.stderr
caddy.logging.writers.stdout
caddy.storage.file_system
http
http.authentication.hashes.bcrypt
http.authentication.hashes.scrypt
http.authentication.providers.http_basic
http.encoders.brotli
http.encoders.gzip
http.encoders.zstd
http.handlers.authentication
http.handlers.cache
http.handlers.encode
http.handlers.error
http.handlers.file_server
http.handlers.headers
http.handlers.request_body
http.handlers.reverse_proxy
http.handlers.rewrite
http.handlers.static_response
http.handlers.subroute
http.handlers.templates
http.handlers.vars
http.matchers.file
http.matchers.header
http.matchers.header_regexp
http.matchers.host
http.matchers.method
http.matchers.not
http.matchers.path
http.matchers.path_regexp
http.matchers.protocol
http.matchers.query
http.matchers.remote_ip
http.matchers.vars
http.matchers.vars_regexp
http.reverse_proxy.circuit_breakers.local
http.reverse_proxy.selection_policies.first
http.reverse_proxy.selection_policies.header
http.reverse_proxy.selection_policies.ip_hash
http.reverse_proxy.selection_policies.least_conn
http.reverse_proxy.selection_policies.random
http.reverse_proxy.selection_policies.random_choose
http.reverse_proxy.selection_policies.round_robin
http.reverse_proxy.selection_policies.uri_hash
http.reverse_proxy.transport.fastcgi
http.reverse_proxy.transport.http
http.reverse_proxy.transport.http_ntlm
tls
tls.certificate_selection.custom
tls.certificates.automate
tls.certificates.load_files
tls.certificates.load_folders
tls.certificates.load_pem
tls.handshake_match.sni
tls.management.acme
tls.stek.distributed
tls.stek.standard

Ping, of course, works correctly since the other build of caddy also works.

Comparing the build info shows the working version includes these packages that aren’t in the vanilla

github.com/caddyserver/tls.dns v0.0.0-20200209203058-83da3c680522 h1:QifITlpSKUi5+fuFtLrtCXnI+lTLWSPnTzIED+ASGW8=
github.com/cloudflare/cloudflare-go v0.10.2 h1:VBodKICVPnwmDxstcW3biKcDSpFIfS/RELUXsZSBYK4=
github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I=
golang.org/x/time v0.0.0-20190921001708-c4c64cad1fd0 h1:xQwXv67TxFo9nC1GJFyab5eq/5B590r6RlnL/G8Sz7w=

So it looks like one of those smuggled in support, but this doesn’t seem to work in any standard build.