Caddyfile reverse_proxy stopped working

Help
1

1. The problem I’m having:

Within the past 2 days, the reverse proxy I had set up for my media server via the Caddyfile “reverse_proxy” directive is no longer working.

2. Error messages and/or full log output:

Excerpt of output (parts I thought would be relevant) since pasting all output after “Started Caddy” line exceeded post character limit.

Dec 31 12:25:42 ubuntu-pc caddy[1140]: {"level":"error","ts":1704043542.7140281,"logger":"http.acme_client","msg":"challenge failed","identifier":"mydnsname.com","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"97.70.224.137: Fetching http://mydnsname.com/.well-known/acme-challenge/Pch-0_839TcTv1Pj-dzLIApgUGhv6S2zpdyzlU2PXdA: Timeout during connect (likely firewall problem)","instance":"","subproblems":[]}}
Dec 31 12:25:42 ubuntu-pc caddy[1140]: {"level":"error","ts":1704043542.7140665,"logger":"http.acme_client","msg":"validating authorization","identifier":"mydnsname.com","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"97.70.224.137: Fetching http://mydnsname.com/.well-known/acme-challenge/Pch-0_839TcTv1Pj-dzLIApgUGhv6S2zpdyzlU2PXdA: Timeout during connect (likely firewall problem)","instance":"","subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/127331144/13341318304","attempt":2,"max_attempts":3}
Dec 31 12:25:42 ubuntu-pc caddy[1140]: {"level":"error","ts":1704043542.714109,"logger":"tls.renew","msg":"could not get certificate from issuer","identifier":"mydnsname.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:connection - 97.70.224.137: Fetching http://mydnsname.com/.well-known/acme-challenge/Pch-0_839TcTv1Pj-dzLIApgUGhv6S2zpdyzlU2PXdA: Timeout during connect (likely firewall problem)"}
Dec 31 12:25:44 ubuntu-pc caddy[1140]: {"level":"debug","ts":1704043544.0504794,"logger":"http.acme_client","msg":"http request","method":"HEAD","url":"https://acme.zerossl.com/v2/DV90/newNonce","headers":{"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Type":["application/octet-stream"],"Date":["Sun, 31 Dec 2023 17:25:44 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["_BDjpgCaj8agxgyacdrG1V2e5Z4spPpEThkResa-sTc"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]},"status_code":200}
Dec 31 12:25:44 ubuntu-pc caddy[1140]: {"level":"debug","ts":1704043544.906995,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/newOrder","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["282"],"Content-Type":["application/json"],"Date":["Sun, 31 Dec 2023 17:25:44 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/LpphbjpY-PJdTeTVXlea9g"],"Replay-Nonce":["5nQ4Gm7nKMQtukZvy32026sBYNp-sLB8MY5OYWn7oy0"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]},"status_code":201}
Dec 31 12:25:45 ubuntu-pc caddy[1140]: {"level":"debug","ts":1704043545.6767008,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/c3gcgRBNVf_4EMHrdsgmSQ","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["300"],"Content-Type":["application/json"],"Date":["Sun, 31 Dec 2023 17:25:45 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["Gea4FLalGDouKVj9W8CLfA57Skor4Qwf_a_5Xi2jNOg"],"Retry-After":["86400"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]},"status_code":200}
Dec 31 12:25:45 ubuntu-pc caddy[1140]: {"level":"error","ts":1704043545.6769145,"logger":"tls.renew","msg":"could not get certificate from issuer","identifier":"mydnsname.com","issuer":"acme.zerossl.com-v2-DV90","error":"[mydnsname.com] solving challenges: authz https://acme.zerossl.com/v2/DV90/authz/c3gcgRBNVf_4EMHrdsgmSQ has unexpected status; order will fail: invalid (order=https://acme.zerossl.com/v2/DV90/order/LpphbjpY-PJdTeTVXlea9g) (ca=https://acme.zerossl.com/v2/DV90)"}
Dec 31 12:25:45 ubuntu-pc caddy[1140]: {"level":"debug","ts":1704043545.6769593,"logger":"events","msg":"event","name":"cert_failed","id":"da4486ff-555f-4f3a-a1ed-cc2be08047b2","origin":"tls","data":{"error":{},"identifier":"mydnsname.com","issuers":["acme-v02.api.letsencrypt.org-directory","acme.zerossl.com-v2-DV90"],"remaining":-81506650983212,"renewal":true}}
Dec 31 12:25:45 ubuntu-pc caddy[1140]: {"level":"error","ts":1704043545.6769953,"logger":"tls.renew","msg":"will retry","error":"[mydnsname.com] Renew: [mydnsname.com] solving challenges: authz https://acme.zerossl.com/v2/DV90/authz/c3gcgRBNVf_4EMHrdsgmSQ has unexpected status; order will fail: invalid (order=https://acme.zerossl.com/v2/DV90/order/LpphbjpY-PJdTeTVXlea9g) (ca=https://acme.zerossl.com/v2/DV90)","attempt":4,"retrying_in":300,"elapsed":401.608607709,"max_duration":2592000}

3. Caddy version:

v2.7.6

4. How I installed and ran Caddy:

Followed the guide Install — Caddy Documentation for Ubuntu install

a. System environment:

Ubuntu 22.04.3 LTS
Kernel: 6.2.0-39-generic
systemd 249 (249.11-0ubuntu3.11)

b. Command:

Running the “reverse-proxy” command as shown below works successfully.

caddy reverse-proxy --from mydnsname.com --to 127.0.0.1:8096

c. Service/unit/compose file:

d. My complete Caddy config:

{
  debug
}
mydnsname.com {
  reverse_proxy 127.0.0.1:8096
}

The reverse proxy was working normally until a day or two ago using the above Caddyfile. The only change made since is the addition of the “debug” global option.

Now, the only way the reverse proxy is working (e.g. allowing access to my media server via https://mydnsname.com) is to stop the Caddy service that runs on system startup

sudo systemctl stop caddy

and manually issue the reverse-proxy command

caddy reverse-proxy --from mydnsname.com --to 127.0.0.1:8096

Seemingly, the command is doing the same thing as the Caddyfile, but the result is different.

5. Links to relevant resources:

2

Looks like TLS cert renewal is failing because ACME issuers aren’t able to connect to your server on port 80.

Are you sure your DNS is still correct (your IP address hasn’t changed)? Are you sure your firewall/router are still allowing connections on port 80 and 443?

3

Originally I was forwarding ports “8080” and “8443” because my previous ISP prohibited forwarding of common ports. I had my Caddyfile configured with:

{
  http_port 8080
  https_port 8443
}

And I would access the media server at address https://mydnsname.com:8443

This was working until a few days ago, so maybe something changed on the TLS certificate issuer side.

While trying to debug the issue yesterday, I eliminated the custom http and https ports from the Caddyfile right away to isolate the problem. Then I replaced the port forwarding rule to only forward port 443. I believe this was my problem as you identified - there was no longer any port forwarding for http. Although I still don’t understand why executing the reverse-proxy command from terminal still worked fine.

After editing the rule to additionally forward port 80 and rebooting, Caddy was establishing the reverse proxy successfully and I could access the media server via https://mydnsname.com.

However, since I already have all devices set up to access via https://mydnsname.com:8443, I changed the port forwarding rules such that external ports 8443 and 8080 forward to internal ports 443 and 80. This is also working. If any problems arise, I know this is the first place to check.

Thank you for your help with this! I appreciate you.

4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.