Struggling to get TLS-APLN-01 challenge working behind another reverse proxy

Help
1

1. The problem I’m having:

Caddy does not seem to be able to complete the TLS-APLN-01 challenge with Let’s Encrypt’s staging server in my environment, and I can’t track down why.

I’m running Caddy on a Debian server, behind another server running HAProxy. I’m in a constrained environment where port 80 is not allowed through the network firewall, but port 443 is, so I’m relying on the TLS-APLN-01 challenge for Caddy to issue TLS certs. I’ve verified that traffic for port 443 with the SNI jellyfin.cjc4.com is successfully able to reach the server running HAProxy, is successfully proxied to the correct backend, and (verified with tcpdump on the server running Caddy) is reaching the server running Caddy. However, Caddy’s logs still say that the TLS-APLN-01 challenge fails, with the message “Error getting validation data.” How can I troubleshoot this? Thank you in advance.

For additional information, in this same environment, another server running Traefik is also located behind this same server running HAProxy. The server running Traefik is successfully able to complete the TLS-APLN-01 challenge for services it hosts. This leads me to believe that DNS and firewall rules are both working as expected.

2. Error messages and/or full log output:

Feb 05 12:24:28 adolin systemd[1]: Starting caddy.service - Caddy...
░░ Subject: A start job for unit caddy.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit caddy.service has begun execution.
░░ 
░░ The job identifier is 15125.
Feb 05 12:24:28 adolin caddy[70439]: caddy.HomeDir=/var/lib/caddy
Feb 05 12:24:28 adolin caddy[70439]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Feb 05 12:24:28 adolin caddy[70439]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Feb 05 12:24:28 adolin caddy[70439]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Feb 05 12:24:28 adolin caddy[70439]: caddy.Version=2.6.2
Feb 05 12:24:28 adolin caddy[70439]: runtime.GOOS=linux
Feb 05 12:24:28 adolin caddy[70439]: runtime.GOARCH=amd64
Feb 05 12:24:28 adolin caddy[70439]: runtime.Compiler=gc
Feb 05 12:24:28 adolin caddy[70439]: runtime.NumCPU=4
Feb 05 12:24:28 adolin caddy[70439]: runtime.GOMAXPROCS=4
Feb 05 12:24:28 adolin caddy[70439]: runtime.Version=go1.24.4
Feb 05 12:24:28 adolin caddy[70439]: os.Getwd=/
Feb 05 12:24:28 adolin caddy[70439]: LANG=en_US.UTF-8
Feb 05 12:24:28 adolin caddy[70439]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Feb 05 12:24:28 adolin caddy[70439]: NOTIFY_SOCKET=/run/systemd/notify
Feb 05 12:24:28 adolin caddy[70439]: USER=caddy
Feb 05 12:24:28 adolin caddy[70439]: LOGNAME=caddy
Feb 05 12:24:28 adolin caddy[70439]: HOME=/var/lib/caddy
Feb 05 12:24:28 adolin caddy[70439]: INVOCATION_ID=95325da0fa9c4648a4f14515aefbc4d1
Feb 05 12:24:28 adolin caddy[70439]: JOURNAL_STREAM=9:212014
Feb 05 12:24:28 adolin caddy[70439]: SYSTEMD_EXEC_PID=70439
Feb 05 12:24:28 adolin caddy[70439]: MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/system.slice/caddy.service/memory.pressure
Feb 05 12:24:28 adolin caddy[70439]: MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA=
Feb 05 12:24:28 adolin caddy[70439]: {"level":"info","ts":1770312268.3467023,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Feb 05 12:24:28 adolin caddy[70439]: {"level":"info","ts":1770312268.3508487,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
Feb 05 12:24:28 adolin caddy[70439]: {"level":"info","ts":1770312268.351041,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Feb 05 12:24:28 adolin caddy[70439]: {"level":"info","ts":1770312268.3510556,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Feb 05 12:24:28 adolin caddy[70439]: {"level":"info","ts":1770312268.351221,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00051d0a0"}
Feb 05 12:24:28 adolin caddy[70439]: {"level":"info","ts":1770312268.3512678,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Feb 05 12:24:28 adolin caddy[70439]: {"level":"info","ts":1770312268.351289,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Feb 05 12:24:28 adolin caddy[70439]: {"level":"info","ts":1770312268.3513322,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
Feb 05 12:24:28 adolin caddy[70439]: {"level":"info","ts":1770312268.3513927,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Feb 05 12:24:28 adolin caddy[70439]: {"level":"info","ts":1770312268.3513973,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["jellyfin.cjc4.com"]}
Feb 05 12:24:28 adolin caddy[70439]: {"level":"info","ts":1770312268.3516114,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Feb 05 12:24:28 adolin caddy[70439]: {"level":"info","ts":1770312268.3516853,"msg":"serving initial configuration"}
Feb 05 12:24:28 adolin caddy[70439]: {"level":"info","ts":1770312268.3517191,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Feb 05 12:24:28 adolin caddy[70439]: {"level":"info","ts":1770312268.3517334,"logger":"tls","msg":"finished cleaning storage units"}
Feb 05 12:24:28 adolin caddy[70439]: {"level":"info","ts":1770312268.3517685,"logger":"tls.obtain","msg":"acquiring lock","identifier":"jellyfin.cjc4.com"}
Feb 05 12:24:28 adolin systemd[1]: Started caddy.service - Caddy.
░░ Subject: A start job for unit caddy.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit caddy.service has finished successfully.
░░ 
░░ The job identifier is 15125.
Feb 05 12:24:28 adolin caddy[70439]: {"level":"info","ts":1770312268.354627,"logger":"tls.obtain","msg":"lock acquired","identifier":"jellyfin.cjc4.com"}
Feb 05 12:24:28 adolin caddy[70439]: {"level":"info","ts":1770312268.3547778,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"jellyfin.cjc4.com"}
Feb 05 12:24:28 adolin caddy[70439]: {"level":"info","ts":1770312268.3553956,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["jellyfin.cjc4.com"],"ca":"https://acme-staging-v02.api.letsencrypt.org/directory","account":"colin.j.crawford@gmail.com"}
Feb 05 12:24:28 adolin caddy[70439]: {"level":"info","ts":1770312268.355412,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["jellyfin.cjc4.com"],"ca":"https://acme-staging-v02.api.letsencrypt.org/directory","account":"colin.j.crawford@gmail.com"}
Feb 05 12:24:29 adolin caddy[70439]: {"level":"info","ts":1770312269.2907052,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"jellyfin.cjc4.com","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Feb 05 12:24:39 adolin caddy[70439]: {"level":"error","ts":1770312279.618702,"logger":"http.acme_client","msg":"challenge failed","identifier":"jellyfin.cjc4.com","challenge_type":"http-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:connection","error":"184.189.213.207: Fetching http://jellyfin.cjc4.com/.well-known/acme-challenge/bk_23iiFm5M9EqdhcWNqhkgBY_PUWj_bV2yKQWCAUq8: Timeout during connect (likely firewall problem)"}
Feb 05 12:24:39 adolin caddy[70439]: {"level":"error","ts":1770312279.6187437,"logger":"http.acme_client","msg":"validating authorization","identifier":"jellyfin.cjc4.com","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:connection - 184.189.213.207: Fetching http://jellyfin.cjc4.com/.well-known/acme-challenge/bk_23iiFm5M9EqdhcWNqhkgBY_PUWj_bV2yKQWCAUq8: Timeout during connect (likely firewall problem)","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/263632683/31494382473","attempt":1,"max_attempts":3}
Feb 05 12:24:40 adolin caddy[70439]: {"level":"info","ts":1770312280.8104937,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"jellyfin.cjc4.com","challenge_type":"tls-alpn-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Feb 05 12:24:41 adolin caddy[70439]: {"level":"error","ts":1770312281.2368424,"logger":"http.acme_client","msg":"challenge failed","identifier":"jellyfin.cjc4.com","challenge_type":"tls-alpn-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:connection","error":"184.189.213.207: Error getting validation data"}
Feb 05 12:24:41 adolin caddy[70439]: {"level":"error","ts":1770312281.2369077,"logger":"http.acme_client","msg":"validating authorization","identifier":"jellyfin.cjc4.com","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:connection - 184.189.213.207: Error getting validation data","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/263632683/31494387423","attempt":2,"max_attempts":3}
Feb 05 12:24:42 adolin caddy[70439]: {"level":"error","ts":1770312282.510033,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"jellyfin.cjc4.com","issuer":"acme-staging-v02.api.letsencrypt.org-directory","error":"[jellyfin.cjc4.com] solving challenges: jellyfin.cjc4.com: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[http-01 tls-alpn-01 dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/263632683/31494388133) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
Feb 05 12:24:42 adolin caddy[70439]: {"level":"error","ts":1770312282.5101092,"logger":"tls.obtain","msg":"will retry","error":"[jellyfin.cjc4.com] Obtain: [jellyfin.cjc4.com] solving challenges: jellyfin.cjc4.com: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[http-01 tls-alpn-01 dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/263632683/31494388133) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":14.155421664,"max_duration":2592000}

3. Caddy version:

2.6.2, the latest version packaged for Debian 13.

4. How I installed and ran Caddy:

Installed on Debian 13 with apt install caddy. Run with systemctl start caddy.

a. System environment:

Debian 13, amd64 architecture.

b. Command:

No relevant commands executed.

c. Service/unit/compose file:

No changes made to packaged service file(s).

d. My complete Caddy config:

My caddyfile located at /etc/caddy/Caddyfile is as follows:

{
        # General options

        # TLS options
        email REDACTED@gmail.com
        acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}

jellyfin.cjc4.com {
        reverse_proxy {
                to localhost:8096
                trusted_proxies 192.168.160.21/32
        }
}

5. Links to relevant resources:

N/A

2

Solved, though disappointed. The root issue was that HAProxy was configured to send proxy protocol to the backend server running Caddy, and Caddy does not seem to support that, at least not in the version packaged with Debian. Removing the send-proxy-v2 directive from the backend configuration in HAProxy allowed Caddy to successfully complete the TLS-ALPN-01 challenge.

However, this means I cannot preserve the client IP address for application use. I’ll likely need to replace Caddy with something else that supports the proxy protocol.

3

It is supported with a plugin:

1 Like