1. The problem I’m having:
I am working on setting up my reverse proxy to access some of my internal services with my domain name (brensinger.net) and subdomains. The first of these services is thelounge for IRC, but since I hit this problem I’ve even just reduced the footprint to static responses from caddy itself.
Caddy is in a docker container as are the other services.
The problem is that the SSL handshake isn’t working. I’m getting the age-old 400 timeout (likely firewall) situation. I can access everything just fine from my cell phone without wifi if I set the TLS to internal (of course with warnings about the untrustable cert), but SSL errors for days if I try to use actual certs.
The error is pretty adamant that this is a firewall issue but I don’t understand how that can be the case if my cell network can access the resources (with an untrusted cert) using tls internal. Still, I’m not sure how to 100% rule it out; my ISP claims they are not filtering any ports, I use firewalla as router and DNS server, the ports are forwarded in firewalla (80, 443, and 443/udp), and the issue is not resolved when I put the server into “emergency access” mode (disabling all firewalla routing rules). I have reset the DNS server to cloudflare and google just to rule out any weird filitering on firewalla’s part.
2. Error messages and/or full log output:
I don’t get any logs with the suggested command in the container nor on the host, so I exported them from Portainer. The complete logs I get that way are ridiculously redundant (like the error message is being built up step by step and each step is printed) so I tried to pare it down to what looks like the unique stuff. It was still too long to post so this is just through the end of the first errors - similar messaging repeats for retries. ZeroSSL might be worth posting separately as I don’t actually see any error coming from that side, but I still can’t connect.
DBG ts=1729622143.873199 logger=tls.issuance.acme.acme_client msg=http request method=POST url=https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/14544592223 headers={"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]} response_headers={"Boulder-Requester":["168116113"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["826"],"Content-Type":["application/json"],"Date":["Tue, 22 Oct 2024 18:35:43 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["AsXtR2E7JJT4QilCQ_e-4f_-9zgaLtM4TJg9IW9KAWfTCxJodL4"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]} status_code=200
DBG ts=1729622144.19578 logger=tls.issuance.acme.acme_client msg=http request method=POST url=https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/14544592223 headers={"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]} response_headers={"Boulder-Requester":["168116113"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["826"],"Content-Type":["application/json"],"Date":["Tue, 22 Oct 2024 18:35:44 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["AsXtR2E7Fa0ypMCv25zuWCT-xMhT2cGhYXxz1r9u0ElGLar9Dsk"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]} status_code=200
{"level":"debug","ts":1729622144.5189824,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/14544592223","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["168116113"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["826"],"Content-Type":["application/json"],"Date":["Tue, 22 Oct 2024 18:35:44 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["Mxxe5XbP3cPjeoTNmyWFIZMPJwhb7byK7bwcRODASfnOwoiOz4I"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1729622144.8469765,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/14544592223","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["168116113"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["826"],"Content-Type":["application/json"],"Date":["Tue, 22 Oct 2024 18:35:44 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["Mxxe5XbPA2uOp65YSHASjkgH-Gm1GAZr2bFg_eZxcTG_TouOpCM"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1729622145.1758244,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/14544592223","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["168116113"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["826"],"Content-Type":["application/json"],"Date":["Tue, 22 Oct 2024 18:35:45 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["AsXtR2E7tFNDzOsvnM4JOllcqC6nsiBfOaEkCjztzGeqSZ4oub0"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
DBG ts=1729622145.497003 logger=tls.issuance.acme.acme_client msg=http request method=POST url=https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/14544592223 headers={"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]} response_headers={"Boulder-Requester":["168116113"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["826"],"Content-Type":["application/json"],"Date":["Tue, 22 Oct 2024 18:35:45 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["Mxxe5XbPta6AOmjpp8OgUHeM2RJDGDlpaIAJQ0Q4ceKpVoxDuxk"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]} status_code=200
{"level":"debug","ts":1729622145.8214753,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/14544592223","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["168116113"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["826"],"Content-Type":["application/json"],"Date":["Tue, 22 Oct 2024 18:35:45 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["AsXtR2E70-wu4cL1t7vQc5Zzfo9atZnEzJomIx-o1yxJFeZEIV8"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1729622146.1445563,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/14544592223","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["168116113"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["826"],"Content-Type":["application/json"],"Date":["Tue, 22 Oct 2024 18:35:46 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["AsXtR2E7beZ_E5yAZ7uMEIlb9_l0v47bPUd9T2LPlm9gxEKa7lg"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1729622146.4665804,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/14544592223","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["168116113"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["826"],"Content-Type":["application/json"],"Date":["Tue, 22 Oct 2024 18:35:46 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["AsXtR2E7DV_xz3xEnrYRq5XMTWq9OnMHQI6zcnL6C1mih7zUu8M"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1729622146.7889798,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/14544592223","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["168116113"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["826"],"Content-Type":["application/json"],"Date":["Tue, 22 Oct 2024 18:35:46 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["AsXtR2E7oMQ5xH7xR-RImhpvShx8nxUJYpqJHZCK2gqnsT4NQWY"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1729622147.1226642,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/14544592223","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["168116113"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["826"],"Content-Type":["application/json"],"Date":["Tue, 22 Oct 2024 18:35:47 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["Mxxe5XbPoQ3_vOu_dM6Bt6SJeXRqxJxkDVRkziJN1Ui5lftbGpw"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1729622147.4486928,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/14544592223","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["168116113"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["826"],"Content-Type":["application/json"],"Date":["Tue, 22 Oct 2024 18:35:47 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["Mxxe5XbP1uvqWmbIcnjaM_9zF3pE6tUijcUP1-X_hi1yKFhatZE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1729622147.7718122,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/14544592223","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["168116113"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["826"],"Content-Type":["application/json"],"Date":["Tue, 22 Oct 2024 18:35:47 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["AsXtR2E7vr6fwHTuyVBrGMpcGxac9T1SudmAh9CKmdO02aGSLpY"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1729622148.1051173,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/14544592223","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["168116113"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["826"],"Content-Type":["application/json"],"Date":["Tue, 22 Oct 2024 18:35:48 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["Mxxe5XbPn88sVXVdqeBe6CNDpc43vEE8a7W4NdBuh1nOiMaBuDw"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1729622148.4413064,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/14544592223","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["168116113"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["826"],"Content-Type":["application/json"],"Date":["Tue, 22 Oct 2024 18:35:48 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["AsXtR2E7bk-zmaao0tJt2OkTWO9Jo3xrfJD7SqB3K_BE8yuuOZg"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
DBG ts=1729622148.7614417 logger=tls.issuance.acme.acme_client msg=http request method=POST url=https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/14544592223 headers={"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]} response_headers={"Boulder-Requester":["168116113"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["1118"],"Content-Type":["application/json"],"Date":["Tue, 22 Oct 2024 18:35:48 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["AsXtR2E7m7ZHhpDOFHqb_Y5q_vljUFDEssJlq2heli3lfKWirOo"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]} status_code=200
ERR ts=1729622148.761981 logger=tls.issuance.acme.acme_client msg=challenge failed identifier=lounge.brensinger.net challenge_type=http-01 problem={"type":"urn:ietf:params:acme:error:connection","title":"","detail":"During secondary validation: 32.216.175.43: Fetching http://lounge.brensinger.net/.well-known/acme-challenge/PwcDxOfYWTzIHeszVJyDXn_roUbzptfIdLg9TDzjWgY: Timeout during connect (likely firewall problem)","instance":"","subproblems":[]}
ERR ts=1729622148.7620494 logger=tls.issuance.acme.acme_client msg=validating authorization identifier=lounge.brensinger.net problem={"type":"urn:ietf:params:acme:error:connection","title":"","detail":"During secondary validation: 32.216.175.43: Fetching http://lounge.brensinger.net/.well-known/acme-challenge/PwcDxOfYWTzIHeszVJyDXn_roUbzptfIdLg9TDzjWgY: Timeout during connect (likely firewall problem)","instance":"","subproblems":[]} order=https://acme-staging-v02.api.letsencrypt.org/acme/order/168116113/19915238683 attempt=1 max_attempts=3
3. Caddy version:
v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=
4. How I installed and ran Caddy:
a. System environment:
Headless ubuntu amd64 host
Docker version 25.0.3, build 4debf41
DockSTARTer
DockSTARTer manages compose files for common services automatically. Caddy is not one of those services, but I can add “unsupported” services using docker-compose.override.yml which is where caddy lives
b. Command:
ds -c up caddy
This is supposed to be equivalent to docker compose up caddy. Running it through dockSTARTer makes sure environment variables are correct/shared across the docker network.
c. Service/unit/compose file:
name: compose
services:
caddy:
image: caddy
container_name: caddy
hostname: caddy
restart: unless-stopped
env_file: /home/charlie/.docker/compose/.env
ports:
- "80:80"
- "443:443"
- "443:443/udp"
volumes:
- /home/charlie/caddy:/srv
- /home/charlie/caddy/Caddyfile:/etc/caddy/Caddyfile
- /home/charlie/.config/appdata/caddy:/config
- /home/charlie/storage/caddy:/data
network_mode: host
networks:
default:
name: $DOCKER_MY_NETWORK
external: true
DOCKER_MY_NETWORK is caddy_net and Portainer indeed shows all the other services on caddy_net. Perhaps worth noting that if I enable the reverse_proxy to thelounge with internal tls, everything works as expected, so that’s why I’ve pared this down to just the caddy config.
d. My complete Caddy config:
{
acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
debug
}
brensinger.net {
tls internal
respond "Welcome!! @"
}
www.brensinger.net {
tls internal
redir https://brensinger.net{uri}
}
jf.brensinger.net {
tls internal
respond "jf"
}
media.brensinger.net {
tls internal
respond "media"
}
lounge.brensinger.net {
tls admin@brensinger.net
# reverse_proxy thelounge:9000
respond "TLS?"
}