TLS "invalid answer" when trying to get initial LetsEncrypt certificate

Help
1

1. The problem I’m having:

I’m setting up caddy as a reverse proxy on a copy of a VM (but with a different hostname). I deleted the caddy user’s home (the whole caddy config) and made sure the Caddyfile points to the new hostname. I have a caddy 2.7.6 with the caddy-security plugin.

Caddy fails to get its certificate from letsencrypt. The reported error is

HTTP 400 urn:ietf:params:acme:error:tls - 34.27.15.232: Fetching https://central2.wisebridge.tw/.well-known/acme-challenge/JrQGUXLjkgepeXZa-PgV6UurF141EXR6XPX3JCP__58: remote error: tls: internal error

Indeed, running curl on central2.wisebridge.tw reports a TLS internal error as well. But then, there is no certificate … so isn’t that expected ? Can letsencrypt read the challenge from HTTPS even though there is no certificate yet

2. Error messages and/or full log output:

caddy.HomeDir=/home/caddy
caddy.AppDataDir=/home/caddy/.local/share/caddy
caddy.AppConfigDir=/home/caddy/.config/caddy
caddy.ConfigAutosavePath=/home/caddy/.config/caddy/autosave.json
caddy.Version=v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=
runtime.GOOS=linux
runtime.GOARCH=amd64
runtime.Compiler=gc
runtime.NumCPU=1
runtime.GOMAXPROCS=1
runtime.Version=go1.21.4
os.Getwd=/
LANG=C.UTF-8
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
NOTIFY_SOCKET=/run/systemd/notify
HOME=/home/caddy
LOGNAME=caddy
USER=caddy
INVOCATION_ID=c3f562fba0fc4e6bb5f0e44828087aa9
JOURNAL_STREAM=8:18135
SYSTEMD_EXEC_PID=2077
{"level":"info","ts":1704375766.3538609,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
{"level":"warn","ts":1704375766.3555255,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1704375766.3568468,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1704375766.357068,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1704375766.357234,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"debug","ts":1704375766.3573096,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{}]}},"http":{"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"authentication","providers":{"authorizer":{"gatekeeper_name":"tw","route_matcher":"*"}}}]},{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"rewrite","strip_path_prefix":"/assets"}]},{"handle":[{"handler":"vars","root":"/home/rails/wisebridge-central/public/assets"},{"handler":"file_server","hide":["/etc/caddy/Caddyfile"]}]}]}],"match":[{"path":["/assets/*"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"127.0.0.1:3000"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}}
{"level":"info","ts":1704375766.3577561,"logger":"security","msg":"provisioning app instance","app":"security"}
{"level":"debug","ts":1704375766.3579302,"logger":"security","msg":"Configured gatekeeper","gatekeeper_name":"tw","gatekeeper_id":"4d6ef339-32ac-46ef-861f-227d66b5c3b4","auth_url_path":"https://auth.wisebridge.tw/oauth2/google","token_sources":"cookie header query","token_validator_options":{"validate_bearer_header":true},"access_list_rules":[{"conditions":["match roles tw/ops"],"action":"allow log debug"}],"forbidden_path":""}
{"level":"info","ts":1704375766.3580072,"logger":"security","msg":"provisioned app instance","app":"security"}
{"level":"debug","ts":1704375766.3581812,"logger":"security","msg":"started app instance","app":"security"}
{"level":"info","ts":1704375766.3582983,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1704375766.3584216,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
{"level":"debug","ts":1704375766.3586235,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
{"level":"info","ts":1704375766.358682,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"debug","ts":1704375766.3587465,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
{"level":"info","ts":1704375766.3587925,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1704375766.3588295,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["central2.wisebridge.tw"]}
{"level":"info","ts":1704375766.3590398,"msg":"autosaved config (load with --resume flag)","file":"/home/caddy/.config/caddy/autosave.json"}
tarted caddy.service - Caddy.
{"level":"info","ts":1704375766.361648,"msg":"serving initial configuration"}
{"level":"info","ts":1704375766.361907,"logger":"tls.obtain","msg":"acquiring lock","identifier":"central2.wisebridge.tw"}
{"level":"info","ts":1704375766.3664305,"logger":"tls.obtain","msg":"lock acquired","identifier":"central2.wisebridge.tw"}
{"level":"info","ts":1704375766.366589,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"central2.wisebridge.tw"}
{"level":"debug","ts":1704375766.3666458,"logger":"events","msg":"event","name":"cert_obtaining","id":"7425f630-7f48-4844-a7dc-f3266eafe48e","origin":"tls","data":{"identifier":"central2.wisebridge.tw"}}
{"level":"debug","ts":1704375766.3669817,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
{"level":"info","ts":1704375766.3673294,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["central2.wisebridge.tw"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1704375766.3674092,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["central2.wisebridge.tw"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1704375766.368102,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0004c5180"}
{"level":"info","ts":1704375766.3722286,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/home/caddy/.local/share/caddy"}
{"level":"info","ts":1704375766.372475,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"debug","ts":1704375766.5253923,"logger":"http.acme_client","msg":"http request","method":"GET","url":"https://acme-v02.api.letsencrypt.org/directory","headers":{"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["752"],"Content-Type":["application/json"],"Date":["Thu, 04 Jan 2024 13:42:46 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1704375766.568485,"logger":"http.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Thu, 04 Jan 2024 13:42:46 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["zXX7izQwzeJWKDjg3Am5pfQ2h5Ro6eFV8SvVoibqZcx7gJB78hY"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1704375766.6472158,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1499325266"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["348"],"Content-Type":["application/json"],"Date":["Thu, 04 Jan 2024 13:42:46 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/1499325266/234195407396"],"Replay-Nonce":["zXX7izQw2rtONKWCs8dQccSEp5130JpZNGqZUFKVrrvoDShL_W4"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
{"level":"debug","ts":1704375766.6952698,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/300680151216","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1499325266"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["806"],"Content-Type":["application/json"],"Date":["Thu, 04 Jan 2024 13:42:46 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["zXX7izQw81A6I-EFZtWTO4AEllanmz9F4_3-BRgcV33jEPr5npY"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"info","ts":1704375766.6956224,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"central2.wisebridge.tw","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"debug","ts":1704375766.69602,"logger":"http.acme_client","msg":"waiting for solver before continuing","identifier":"central2.wisebridge.tw","challenge_type":"http-01"}
{"level":"debug","ts":1704375766.6961913,"logger":"http.acme_client","msg":"done waiting for solver","identifier":"central2.wisebridge.tw","challenge_type":"http-01"}
{"level":"debug","ts":1704375766.7460277,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/300680151216/E2HRLw","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1499325266"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["187"],"Content-Type":["application/json"],"Date":["Thu, 04 Jan 2024 13:42:46 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/authz-v3/300680151216>;rel=\"up\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/chall-v3/300680151216/E2HRLw"],"Replay-Nonce":["I5zfHDkYB9x0K-h_zNlQGlxAkG2y9ARQXwpbyUzuIkRuSFuyYh4"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1704375766.746381,"logger":"http.acme_client","msg":"challenge accepted","identifier":"central2.wisebridge.tw","challenge_type":"http-01"}
{"level":"debug","ts":1704375767.0431077,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/300680151216","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1499325266"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["806"],"Content-Type":["application/json"],"Date":["Thu, 04 Jan 2024 13:42:47 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["I5zfHDkYv6RO4iv9a9idmBA4bseqW59eYGBfyTQbzQ44BtXOEjo"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1704375767.3416102,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/300680151216","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1499325266"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["806"],"Content-Type":["application/json"],"Date":["Thu, 04 Jan 2024 13:42:47 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["I5zfHDkYnLIziK5fuhIr_hpm9GyW3sDePeU8CO2l-z1niybbEz8"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1704375767.6383233,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/300680151216","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1499325266"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["806"],"Content-Type":["application/json"],"Date":["Thu, 04 Jan 2024 13:42:47 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["I5zfHDkYr6ubJh_RW7YTsn7JM7u7YVGAO6UvRX5cYtmrHzmrp9U"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1704375767.9347703,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/300680151216","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1499325266"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["806"],"Content-Type":["application/json"],"Date":["Thu, 04 Jan 2024 13:42:47 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["I5zfHDkYvxFZIBvkmvugItLYq25QAdACRrMNmyAC4XagUh-aEOY"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1704375768.2314026,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/300680151216","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1499325266"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["806"],"Content-Type":["application/json"],"Date":["Thu, 04 Jan 2024 13:42:48 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["zXX7izQwBBySPekRFboZl0I-CEklrGEuCEVwixZIvEvxseTL3Lg"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1704375768.5266325,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/300680151216","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1499325266"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["806"],"Content-Type":["application/json"],"Date":["Thu, 04 Jan 2024 13:42:48 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["I5zfHDkYvqZ5q3HYA36PE81xCagLUZ1rLfz-6uFu13AmMSfEQEs"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1704375768.8234296,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/300680151216","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1499325266"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["1391"],"Content-Type":["application/json"],"Date":["Thu, 04 Jan 2024 13:42:48 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["zXX7izQwjxW-j4y6l-EqJZ2AAsREVXDGjZy0_Hy3-Y7JIomIoKE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"error","ts":1704375768.8247075,"logger":"http.acme_client","msg":"challenge failed","identifier":"central2.wisebridge.tw","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"34.27.15.232: Fetching https://central2.wisebridge.tw/.well-known/acme-challenge/CniTlqPeRAZi7xvpQRB5BtpDP3yl-soNNKvWOzFAb5E: remote error: tls: internal error","instance":"","subproblems":[]}}
{"level":"error","ts":1704375768.824862,"logger":"http.acme_client","msg":"validating authorization","identifier":"central2.wisebridge.tw","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"34.27.15.232: Fetching https://central2.wisebridge.tw/.well-known/acme-challenge/CniTlqPeRAZi7xvpQRB5BtpDP3yl-soNNKvWOzFAb5E: remote error: tls: internal error","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/1499325266/234195407396","attempt":1,"max_attempts":3}

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

a. System environment:

Distributor ID: Debian
Description:    Debian GNU/Linux 12 (bookworm)
Release:        12
Codename:       bookworm

Using systemd. Nothing special on the systemd side. Running as a separate user. /home/caddy is writeable and persistent.

b. Command:

c. Service/unit/compose file:

d. My complete Caddy config:

{
  debug
  order authenticate before respond
  order authorize before basicauth

  security {
    authorization policy tw {
      bypass uri prefix /wiselogs/api/v1
      validate bearer header
      set auth url https://auth.wisebridge.tw/oauth2/google
      crypto key verify from file /etc/caddy/wisebridge-pub.pem
      allow roles tw/ops
    }
  }
}

central2.wisebridge.tw {
  authorize with tw

  handle_path /assets/* {
    root * /home/rails/wisebridge-central/public/assets
    file_server
  }

  reverse_proxy 127.0.0.1:3000
}

5. Links to relevant resources:

2

Yes – that’s the way the ACME TLS-ALPN challenge works, via some special data in the TLS handshake. See Challenge Types - Let's Encrypt

Is that all the logs you have? I’m confused because your logs show Caddy is trying the ACME HTTP challenge as well, but I don’t see the conclusion of it.

3

Oh wait – the TLS error is coming from the HTTP challenge. That’s wild.

Do you have somekind of proxy in front of Caddy which might be redirecting HTTP to HTTPS? Are you using Cloudflare for example?

4

Right … so it was all my fault as expected. Thanks for asking about CloudFlare, it got me looking into the DNS zone and seeing the error.

I got the DNS setup in terraform wrong, and it was still pointing to the old server… Why is that causing a TLS internal error, I don’t know …

Fixing the DNS entry solved the issue right away.

1 Like
5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.