Single service no CGNAT Timeout during connect (likely firewall problem)

Help
1

1. The problem I’m having:

After finally getting internet that isn’t behind a CGNAT, I am trying to simplify my homelab setup by moving my Caddy from a VPS to a server on my local network. In doing this, I am getting challenge fail errors on certificate renewal. Not sure how to proceed.

107.178.7.78: Fetching http://gitea.veritablevalor.com/.well-known/acme-challenge/WBv1QzhiKUk0x6W8ajCfb06agvOPB06swofi1GyN0h4: Timeout during connect (likely firewall problem)

2. Error messages and/or full log output:

caddy  | {"level":"info","ts":1735310073.1551025,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
caddy  | {"level":"info","ts":1735310073.1591182,"msg":"adapted config to JSON","adapter":"caddyfile"}
caddy  | {"level":"warn","ts":1735310073.1623573,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
caddy  | {"level":"info","ts":1735310073.175995,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy  | {"level":"info","ts":1735310073.1765828,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy  | {"level":"info","ts":1735310073.1766567,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy  | {"level":"info","ts":1735310073.1792445,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
caddy  | {"level":"info","ts":1735310073.1799774,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
caddy  | {"level":"info","ts":1735310073.1807303,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddy  | {"level":"info","ts":1735310073.1811168,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddy  | {"level":"info","ts":1735310073.1816185,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["gitea.veritablevalor.com"]}
caddy  | {"level":"info","ts":1735310073.1834373,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00068ba00"}
caddy  | {"level":"info","ts":1735310073.1901941,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy  | {"level":"info","ts":1735310073.1902661,"msg":"serving initial configuration"}
caddy  | {"level":"info","ts":1735310073.1958888,"logger":"tls.obtain","msg":"acquiring lock","identifier":"gitea.veritablevalor.com"}
caddy  | {"level":"info","ts":1735310073.2298613,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"b9d91aef-ff93-4d8e-b144-97df91becd43","try_again":1735396473.2298536,"try_again_in":86399.99999865}
caddy  | {"level":"info","ts":1735310073.2299924,"logger":"tls.obtain","msg":"lock acquired","identifier":"gitea.veritablevalor.com"}
caddy  | {"level":"info","ts":1735310073.230255,"logger":"tls","msg":"finished cleaning storage units"}
caddy  | {"level":"info","ts":1735310073.2305427,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"gitea.veritablevalor.com"}
caddy  | {"level":"info","ts":1735310073.244029,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["gitea.veritablevalor.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
caddy  | {"level":"info","ts":1735310073.244157,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["gitea.veritablevalor.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
caddy  | {"level":"info","ts":1735310073.2449074,"logger":"http","msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/2132604505","account_contact":[]}
caddy  | {"level":"info","ts":1735310074.1342561,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"gitea.veritablevalor.com","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
caddy  | {"level":"error","ts":1735310084.5404093,"logger":"http.acme_client","msg":"challenge failed","identifier":"gitea.veritablevalor.com","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"107.178.7.78: Fetching http://gitea.veritablevalor.com/.well-known/acme-challenge/WBv1QzhiKUk0x6W8ajCfb06agvOPB06swofi1GyN0h4: Timeout during connect (likely firewall problem)","instance":"","subproblems":[]}}
caddy  | {"level":"error","ts":1735310084.540601,"logger":"http.acme_client","msg":"validating authorization","identifier":"gitea.veritablevalor.com","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"107.178.7.78: Fetching http://gitea.veritablevalor.com/.well-known/acme-challenge/WBv1QzhiKUk0x6W8ajCfb06agvOPB06swofi1GyN0h4: Timeout during connect (likely firewall problem)","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/2132604505/337818400695","attempt":1,"max_attempts":3}
caddy  | {"level":"info","ts":1735310086.3745348,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"gitea.veritablevalor.com","challenge_type":"tls-alpn-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
caddy  | {"level":"info","ts":1735310093.214666,"msg":"shutting down apps, then terminating","signal":"SIGTERM"}
caddy  | {"level":"warn","ts":1735310093.2150981,"msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
caddy  | {"level":"info","ts":1735310093.2155733,"logger":"http","msg":"servers shutting down with eternal grace period"}
caddy  | {"level":"warn","ts":1735310093.2173066,"logger":"http.acme_client","msg":"HTTP request failed; retrying","url":"https://acme-v02.api.letsencrypt.org/acme/authz/2132604505/451363512625","error":"performing request: Post \"https://acme-v02.api.letsencrypt.org/acme/authz/2132604505/451363512625\": context canceled"}
caddy  | {"level":"error","ts":1735310093.217379,"logger":"http.acme_client","msg":"deactivating authorization","identifier":"gitea.veritablevalor.com","authz":"https://acme-v02.api.letsencrypt.org/acme/authz/2132604505/451363512625","error":"attempt 1: https://acme-v02.api.letsencrypt.org/acme/authz/2132604505/451363512625: context canceled"}
caddy  | {"level":"error","ts":1735310093.2174602,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"gitea.veritablevalor.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[gitea.veritablevalor.com] solving challenges: [gitea.veritablevalor.com] context canceled (order=https://acme-v02.api.letsencrypt.org/acme/order/2132604505/337818457925) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
caddy  | {"level":"info","ts":1735310093.2176192,"logger":"tls.obtain","msg":"releasing lock","identifier":"gitea.veritablevalor.com"}
caddy  | {"level":"error","ts":1735310093.2178595,"logger":"tls.obtain","msg":"unable to unlock","identifier":"gitea.veritablevalor.com","lock_key":"issue_cert_gitea.veritablevalor.com","error":"remove /data/caddy/locks/issue_cert_gitea.veritablevalor.com.lock: no such file or directory"}
caddy  | {"level":"error","ts":1735310093.2179327,"logger":"tls","msg":"job failed","error":"gitea.veritablevalor.com: obtaining certificate: [gitea.veritablevalor.com] Obtain: [gitea.veritablevalor.com] solving challenges: [gitea.veritablevalor.com] context canceled (order=https://acme-v02.api.letsencrypt.org/acme/order/2132604505/337818457925) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
caddy  | {"level":"info","ts":1735310093.2192986,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
caddy  | {"level":"info","ts":1735310093.2193553,"msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
caddy exited with code 0

3. Caddy version: 2.8.4-alpine (with cloudflare dns plugin)

4. How I installed and ran Caddy:

Caddy is installed on a headless arch VM running through docker-compose.

a. System environment:

My public IP is 107.178.7.78
My VM IP is 10.0.10.20
I have confirmed that my ISP is NOT blocking ports 80 and 443 and that my port forwarding works - tested it with an http site in a different docker container.

Docker versions

docker 1:24.0.2-1
docker-compose 2.19.1-1

b. Command:

N/A I think

c. Service/unit/compose file:

version: "3.7"
services:
  caddy:
    logging:
      driver: loki
      options:
        loki-url: http://10.0.10.23:3100/loki/api/v1/push
    container_name: caddy
    image: ghcr.io/caddybuilds/caddy-cloudflare:2.8.4-alpine
    restart: unless-stopped
    ports:
      - 80:80
      - 443:443
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./data:/data
      - ./config:/config
    environment:
      - CLOUDFLARE_API_TOKEN=<my eyes only>
networks: {}

d. My complete Caddy config:

Normally I have more, but in trying to resolve the issue, my Caddyfile has been reduced to one service.

gitea.veritablevalor.com {
    reverse_proxy 10.0.10.21:3003
}

5. Links to relevant resources:

2

I find that message is very true.

3

Do you know of any checks past open ports that I could be doing to narrow down the search?

4

Try these

Make sure Stockholm, Sweden & Singapore can connect as Let’s Encrypt uses Multi-Perspective Validation Improves Domain Validation Security - Let’s Encrypt