[Solved] Changing VPS and caddy not succeeding with certs

Help
1

1. The problem I’m having:

Forgive me, I don’t fully understand how TLS really works. I am moving VPS and want to redirect example.com to point to the new server. At the DNS level I’ve done this, but caddy won’t serve me anything at example.com now, from what I presume is it failing to get an https certificate.

I saw a couple other topics on moving VPS, and I copied all the files from my old VPS’s certificates directory over to the new server, but I think that’s contributing to more problems.

I should also mention that caddy is working find for some of the domains that are redirecting to services, but it isn’t working on any endpoints with file_server directive.

2. Error messages and/or full log output:

This is before I copied certs over.

{"level":"info","ts":1701792958.8249846,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1701792958.8311405,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1701792958.8371565,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1701792958.8378112,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1701792958.8379698,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1701792958.8380525,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv1"}
{"level":"info","ts":1701792958.8537703,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000652700"}
{"level":"info","ts":1701792958.854257,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
{"level":"info","ts":1701792958.8569999,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1701792958.89402,"logger":"pki.ca.local","msg":"root certificate is already trusted by system","path":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":1701792958.8945768,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1701792958.8949215,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
{"level":"info","ts":1701792958.895278,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1701792958.8955452,"logger":"http","msg":"enabling HTTP/3 listener","addr":":8443"}
{"level":"info","ts":1701792958.8959994,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
{"level":"info","ts":1701792958.8963084,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1701792958.896459,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["stringcase.org","mn.rutrum.net","45.63.65.162","cloud2.rutrum.net","anki.rutrum.net","www.rutrum.net","rutrum.net"]}
{"level":"info","ts":1701792958.9024584,"logger":"tls.obtain","msg":"acquiring lock","identifier":"stringcase.org"}
{"level":"info","ts":1701792958.9037902,"logger":"tls.obtain","msg":"lock acquired","identifier":"stringcase.org"}
{"level":"info","ts":1701792958.9053376,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"stringcase.org"}
{"level":"info","ts":1701792958.9068308,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["stringcase.org"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"davepurdum@pm.me"}
{"level":"info","ts":1701792958.9069858,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["stringcase.org"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"davepurdum@pm.me"}
{"level":"warn","ts":1701792959.0552335,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [45.63.65.162]: no OCSP server specified in certificate","identifiers":["45.63.65.162"]}
{"level":"info","ts":1701792959.0558188,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1701792959.0560489,"msg":"serving initial configuration"}
{"level":"info","ts":1701792959.1545413,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"stringcase.org","challenge_type":"tls-alpn-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"error","ts":1701792959.7733595,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"stringcase.org","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge","instance":"","subproblems":[]}}
{"level":"error","ts":1701792959.7736707,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"stringcase.org","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/1446778836/227075318996","attempt":1,"max_attempts":3}
{"level":"info","ts":1701792960.9103184,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"stringcase.org","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"error","ts":1701792961.239697,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"stringcase.org","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"45.63.65.16: Invalid response from http://stringcase.org/.well-known/acme-challenge/a7xpkF4b6QDGqntlR1V3YeR9HY9afexoY2Z4QS167Ws: 404","instance":"","subproblems":[]}}
{"level":"error","ts":1701792961.2401223,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"stringcase.org","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"45.63.65.16: Invalid response from http://stringcase.org/.well-known/acme-challenge/a7xpkF4b6QDGqntlR1V3YeR9HY9afexoY2Z4QS167Ws: 404","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/1446778836/227075324876","attempt":2,"max_attempts":3}
{"level":"error","ts":1701792961.2404008,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"stringcase.org","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 403 urn:ietf:params:acme:error:unauthorized - 45.63.65.16: Invalid response from http://stringcase.org/.well-known/acme-challenge/a7xpkF4b6QDGqntlR1V3YeR9HY9afexoY2Z4QS167Ws: 404"}
{"level":"info","ts":1701792961.2411015,"logger":"tls.issuance.zerossl","msg":"waiting on internal rate limiter","identifiers":["stringcase.org"],"ca":"https://acme.zerossl.com/v2/DV90","account":"davepurdum@pm.me"}
{"level":"info","ts":1701792961.2412953,"logger":"tls.issuance.zerossl","msg":"done waiting on internal rate limiter","identifiers":["stringcase.org"],"ca":"https://acme.zerossl.com/v2/DV90","account":"davepurdum@pm.me"}
{"level":"error","ts":1701792961.640618,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"stringcase.org","issuer":"acme.zerossl.com-v2-DV90","error":"[stringcase.org] creating new order: fetching new nonce from server: HTTP 504:  (ca=https://acme.zerossl.com/v2/DV90)"}
{"level":"error","ts":1701792961.6407313,"logger":"tls.obtain","msg":"will retry","error":"[stringcase.org] Obtain: [stringcase.org] creating new order: fetching new nonce from server: HTTP 504:  (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":2.736753416,"max_duration":2592000}
{"level":"info","ts":1701793021.64127,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"stringcase.org"}
{"level":"info","ts":1701793021.915543,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"stringcase.org","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
{"level":"error","ts":1701793022.5375707,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"stringcase.org","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"45.63.65.16: Invalid response from http://stringcase.org/.well-known/acme-challenge/ftKVpUsjW09p-VADWMOAsNLha2KO1NtBImyBjavF6vE: 404","instance":"","subproblems":[]}}
{"level":"error","ts":1701793022.5378966,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"stringcase.org","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"45.63.65.16: Invalid response from http://stringcase.org/.well-known/acme-challenge/ftKVpUsjW09p-VADWMOAsNLha2KO1NtBImyBjavF6vE: 404","instance":"","subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/127644284/12768769804","attempt":1,"max_attempts":3}
{"level":"info","ts":1701793023.6464005,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"stringcase.org","challenge_type":"tls-alpn-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
{"level":"error","ts":1701793023.9950287,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"stringcase.org","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge","instance":"","subproblems":[]}}
{"level":"error","ts":1701793023.9950871,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"stringcase.org","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge","instance":"","subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/127644284/12768770414","attempt":2,"max_attempts":3}
{"level":"error","ts":1701793023.995122,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"stringcase.org","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 403 urn:ietf:params:acme:error:unauthorized - Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge"}
{"level":"error","ts":1701793024.0136251,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"stringcase.org","issuer":"acme.zerossl.com-v2-DV90","error":"[stringcase.org] creating new order: fetching new nonce from server: HTTP 504:  (ca=https://acme.zerossl.com/v2/DV90)"}
{"level":"error","ts":1701793024.0137148,"logger":"tls.obtain","msg":"will retry","error":"[stringcase.org] Obtain: [stringcase.org] creating new order: fetching new nonce from server: HTTP 504:  (ca=https://acme.zerossl.com/v2/DV90)","attempt":2,"retrying_in":120,"elapsed":65.109736803,"max_duration":2592000}
{"level":"info","ts":1701793067.0382657,"msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":1701793067.0383441,"msg":"exiting; byeee!! 👋","signal":"SIGTERM"

This is after I moved certs over.

{"level":"info","ts":1701795758.3542159,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1701795758.3592465,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1701795758.3638856,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//[::1]:2019","//127.0.0.1:2019","//localhost:2019"]}
{"level":"info","ts":1701795758.364507,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv1"}
{"level":"info","ts":1701795758.3647475,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1701795758.3648446,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1701795758.3729105,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
{"level":"info","ts":1701795758.3768833,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1701795758.373466,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000668d80"}
{"level":"info","ts":1701795758.3989737,"logger":"pki.ca.local","msg":"root certificate is already trusted by system","path":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":1701795758.3995905,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1701795758.3998492,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
{"level":"info","ts":1701795758.400291,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1701795758.4004915,"logger":"http","msg":"enabling HTTP/3 listener","addr":":8443"}
{"level":"info","ts":1701795758.4007652,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
{"level":"info","ts":1701795758.4009483,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1701795758.4010582,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["stringcase.org","mn.rutrum.net","45.63.65.162","anki.rutrum.net","www.rutrum.net","cloud2.rutrum.net","rutrum.net"]}
{"level":"warn","ts":1701795758.4048553,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [45.63.65.162]: no OCSP server specified in certificate","identifiers":["45.63.65.162"]}
{"level":"info","ts":1701795758.4053142,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1701795758.4054346,"msg":"serving initial configuration"}

3. Caddy version:

v2.7.5 h1:HoysvZkLcN2xJExEepaFHK92Qgs7xAiCFydN5x5Hs6Q

4. How I installed and ran Caddy:

Docker image.

a. System environment:

Debian 6.1.55-1 (2023-09-29) x86_64 GNU/Linux
Docker version 24.0.7, build afdd53b

b. Command:

docker compose up -d

c. Service/unit/compose file:

services:
  caddy:
    image: caddy:alpine
    restart: unless-stopped
    container_name: caddy
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./config:/config
      - ./data:/data
    network_mode: "host"

d. My complete Caddy config:

{
    email <my email>
}
https://cloud2.rutrum.net:443 {
    header Strict-Transport-Security max-age=31536000;
    reverse_proxy localhost:11000
}
https://cloud2.rutrum.net:8443 {
    reverse_proxy https://localhost:8080 {
        transport http {
            tls_insecure_skip_verify
        }
    }
}
anki.rutrum.net {
    reverse_proxy :27701
}
www.rutrum.net, rutrum.net {
    root * /home/rutrum/src/rutrum.net/public
    file_server
}
stringcase.org {
    root * /home/rutrum/src/stringcase.org/public
    file_server
}
mn.rutrum.net {
    root * /home/rutrum/src/mn-svelte/build
    file_server
}

5. Links to relevant resources:

2

Who’s your DNS provider? Are you sure you pointed your DNS to your Caddy server’s IP address?

Does your DNS provider have somekind of proxying enabled (e.g. Cloudflare)? If so, turn that off.

Make sure ports 80 and 443 are open on your VPS (check the firewall, check if ufw is running and if so open ports 80 & 443).

There’s not much benefit to using Docker with host networking rather than simply running Caddy on the host machine. If everything you want to serve is running on the host and not in Docker, then you’d be better off installing Caddy on the host. See Install — Caddy Documentation

3

DNS is NameCheap. Yes it is pointing the right address. I do have other DNS subdomains working. It’s just the https that is going wrong on domains that are currently just acting as file servers. For instance, I had an anki sync server running on the old server, and its working on the new instance just fine, before I tried messing with certificates.

I am not using cloudflare, and ufw isn’t even enabled.

4

Alright I’m an idiot. Why would this only fail on file_server directives and not on port forwarding directives? It has nothing to do with https. I’m pointing the local directories on the host machine, but I forgot that, unlike my last VPS, on this one I’m running caddy inside a docker container. So I need to just mount my files inside the container and point to them…problem solved! How silly of me!

5

Well it did, because TLS issuance was failing.

You must have changed something if it’s suddenly working now.

TLS issuance is at a layer above HTTP routing, so the fact that root/file_server were wrong would not have affected TLS issuance.

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.