Repeated SSL failures when trying to get certs for subdomains

1. My Caddy version (caddy -version):

Caddy v1.0.4 (h1:wwuGSkUHo6RZ3oMpeTt7J09WBB87X5o+IZN4dKehcQE=)

2. How I run Caddy:

Please provide all of the relevant information and DO NOT REDACT anything except passwords/keys. Thank you!

a. System environment:

OS, relevant versions, systemd? docker? etc.

docker, alpine aarch64, raspberry pi 4

b. Command:

ENTRYPOINT ["/usr/bin/caddy"]
CMD ["--conf", "/etc/Caddyfile", "--log", "stdout", "--agree=true"]

c. Service/unit/compose file:

services:
  caddy:
    container_name: caddy
    image: novachat/pi-caddy
    restart: unless-stopped
    ports:
    - "80:80"
    - "443:443"
    volumes:
    - ./caddy/Caddyfile:/etc/Caddyfile:ro
    - ./caddy/letsencrypt:/root/.caddy
    - /srv/web:/srv
    networks:
    - default

d. My complete Caddyfile:

anthony.nova.chat {
        tls email@gmail.com
        proxyprotocol
        root /srv

        proxy / http://synapse:8008 {
                transparent
                except /im
        }
        proxy /hangouts http://mautrix-hangouts:29320 {
                transparent
                without /hangouts
        }
        proxy /telegram http://mautrix-telegram:29317/telegram {
                transparent
                without /telegram
        }
        proxy /slack http://mx-puppet-slack:8434 {
                transparent
                without /slack
        }
        proxy /twitter http://mx-puppet-twitter:4567 {
                transparent
                without /twitter
        }
        proxy /facebook http://mautrix-facebook:29319 {
                transparent
                without /facebook
        }
}

3. The problem I’m having:

Please describe the issue thoroughly enough so that anyone can reproduce the exact behavior you’re seeing. Be as specific as possible.

4. Error messages and/or full log output:

Please DO NOT REDACT any information except passwords/keys.

caddy                  | 2020/01/14 05:35:19 [INFO] Caddy version: v1.0.4
caddy                  | Activating privacy features...2020/01/14 05:35:19 [INFO][cache:0x40001d2140] Started certificate maintenance routine
caddy                  | 2020/01/14 05:35:20 [INFO] acme: Registering account for ericmigi@gmail.com
caddy                  | 2020/01/14 05:35:20 [INFO][anthony.nova.chat] Obtain certificate
caddy                  | 2020/01/14 05:35:20 [INFO][anthony.nova.chat] Obtain: Waiting on rate limiter...
caddy                  | 2020/01/14 05:35:20 [INFO][anthony.nova.chat] Obtain: Done waiting
caddy                  | 2020/01/14 05:35:20 [INFO] [anthony.nova.chat] acme: Obtaining bundled SAN certificate
caddy                  | 2020/01/14 05:35:20 [INFO] [anthony.nova.chat] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2277759143
caddy                  | 2020/01/14 05:35:20 [INFO] [anthony.nova.chat] acme: Could not find solver for: tls-alpn-01
caddy                  | 2020/01/14 05:35:20 [INFO] [anthony.nova.chat] acme: use http-01 solver
caddy                  | 2020/01/14 05:35:20 [INFO] [anthony.nova.chat] acme: Trying to solve HTTP-01
caddy                  | 2020/01/14 05:35:25 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2277759143
caddy                  | 2020/01/14 05:35:25 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2277759143
caddy                  | 2020/01/14 05:35:25 [ERROR][anthony.nova.chat] failed to obtain certificate: acme: Error -> One or more domains had a problem:
caddy                  | [anthony.nova.chat] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Invalid response from http://anthony.nova.chat/.well-known/acme-challenge/ULmwc5xRNdl-29eEVOv9DyHK8dMn7zWRQh5TiJ9c7K8 [68.183.251.172]: 503, url: (attempt 1/3; challenge=http-01)
caddy                  | 2020/01/14 05:35:26 [INFO] [anthony.nova.chat] acme: Obtaining bundled SAN certificate
caddy                  | 2020/01/14 05:35:26 [INFO] [anthony.nova.chat] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2277760423
caddy                  | 2020/01/14 05:35:26 [INFO] [anthony.nova.chat] acme: Could not find solver for: tls-alpn-01
caddy                  | 2020/01/14 05:35:26 [INFO] [anthony.nova.chat] acme: use http-01 solver
caddy                  | 2020/01/14 05:35:26 [INFO] [anthony.nova.chat] acme: Trying to solve HTTP-01
caddy                  | 2020/01/14 05:35:30 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2277760423
caddy                  | 2020/01/14 05:35:30 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2277760423
caddy                  | 2020/01/14 05:35:30 [ERROR][anthony.nova.chat] failed to obtain certificate: acme: Error -> One or more domains had a problem:
caddy                  | [anthony.nova.chat] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Invalid response from http://anthony.nova.chat/.well-known/acme-challenge/6lKiuaX3KHs2N7BYyZsouk000WwUtN9gkxYXTjXD9N0 [68.183.251.172]: 503, url: (attempt 2/3; challenge=http-01)
caddy                  | 2020/01/14 05:35:31 [INFO] [anthony.nova.chat] acme: Obtaining bundled SAN certificate
caddy                  | 2020/01/14 05:35:31 [INFO] [anthony.nova.chat] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2277761639
caddy                  | 2020/01/14 05:35:31 [INFO] [anthony.nova.chat] acme: Could not find solver for: tls-alpn-01
caddy                  | 2020/01/14 05:35:31 [INFO] [anthony.nova.chat] acme: use http-01 solver
caddy                  | 2020/01/14 05:35:31 [INFO] [anthony.nova.chat] acme: Trying to solve HTTP-01
caddy                  | 2020/01/14 05:35:32 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2277761639

5. What I already tried:

I am provisioning several raspberry pis per day. Running script from https://gitlab.com/nova/pi-matrix

This happens almost every time I set up a new Pi. Sometimes waiting an hour resolves it. but this error keeps coming back. Any idea what I’ve got set up wrong?

6. Links to relevant resources:

Hi @ericmigi, welcome to the Caddy community! This one’s interesting.

LetsEncrypt is connecting to something, but it’s getting a bad response:

Invalid response from http://anthony.nova.chat/.well-known/acme-challenge/aFxJhmHkG9356RIdxWWh0tANF0iJhBSsl2VrHdqEb20 [68.183.251.172]: 503

Whatever is at that IP address, it’s probably not Caddy. At no point would Caddy be issuing 503s in response to well-known challenges.

I’m using haproxy here as a reverse proxy. Here’s the config

defaults
  maxconn 1000
  mode http
  log global
  option dontlognull
  timeout http-request 5s
  timeout connect 5000
  timeout client 2000000 # ddos protection
  timeout server 2000000 # stick-table type ip size 100k expire 30s store conn_cur

frontend https
  bind *:443
  mode tcp
  option tcplog
  tcp-request inspect-delay 5s
  tcp-request content accept if { req_ssl_hello_type 1 }

#addhttpsfrontend
  use_backend anthony if { req.ssl_sni -i anthony.nova.chat }

frontend http
  bind *:80
  acl anthony_request hdr(host) -i anthony.nova.chat
  use_backend anthony_pub if anthony_request


backend anthony
  mode tcp
  balance roundrobin
  option ssl-hello-chk
  server pi9 10.0.0.9:443 send-proxy

backend anthony_pub
  server pi9 10.0.0.9:80

For some reason, HAProxy is giving LetsEncrypt those 503 statuses, then. You’ll need to troubleshoot that - HAProxy needs to be sending that traffic on to Caddy straight away.

yeah I think it’s something weird with it. but this is the strange part…i just tried again after waiting an hour for the blocks to reset and it worked

Attaching to caddy
caddy                  | 2020/01/14 06:43:25 [INFO] Caddy version: v1.0.4
caddy                  | Activating privacy features...2020/01/14 06:43:25 [INFO][cache:0x4000032190] Started certificate maintenance routine
caddy                  | 2020/01/14 06:43:25 [INFO][anthony.nova.chat] Obtain certificate
caddy                  | 2020/01/14 06:43:25 [INFO][anthony.nova.chat] Obtain: Waiting on rate limiter...
caddy                  | 2020/01/14 06:43:25 [INFO][anthony.nova.chat] Obtain: Done waiting
caddy                  | 2020/01/14 06:43:25 [INFO] [anthony.nova.chat] acme: Obtaining bundled SAN certificate
caddy                  | 2020/01/14 06:43:26 [INFO] [anthony.nova.chat] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2278548117
caddy                  | 2020/01/14 06:43:26 [INFO] [anthony.nova.chat] acme: use tls-alpn-01 solver
caddy                  | 2020/01/14 06:43:26 [INFO] [anthony.nova.chat] acme: Trying to solve TLS-ALPN-01
caddy                  | 2020/01/14 06:43:27 http: TLS handshake error from 10.0.0.1:44100: tls: first record does not look like a TLS handshake
caddy                  | 2020/01/14 06:43:27 http: TLS handshake error from 10.0.0.1:44102: tls: first record does not look like a TLS handshake
caddy                  | 2020/01/14 06:43:27 http: TLS handshake error from 10.0.0.1:44104: tls: first record does not look like a TLS handshake
caddy                  | 2020/01/14 06:43:33 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2278548117
caddy                  | 2020/01/14 06:43:33 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2278548117
caddy                  | 2020/01/14 06:43:33 [ERROR][anthony.nova.chat] failed to obtain certificate: acme: Error -> One or more domains had a problem:
caddy                  | [anthony.nova.chat] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Errorgetting validation data, url: (attempt 1/3; challenge=tls-alpn-01)
caddy                  | 2020/01/14 06:43:34 [INFO] [anthony.nova.chat] acme: Obtaining bundled SAN certificate
caddy                  | 2020/01/14 06:43:34 [INFO] [anthony.nova.chat] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2278549512
caddy                  | 2020/01/14 06:43:34 [INFO] [anthony.nova.chat] acme: use tls-alpn-01 solver
caddy                  | 2020/01/14 06:43:34 [INFO] [anthony.nova.chat] acme: Trying to solve TLS-ALPN-01
caddy                  | 2020/01/14 06:43:36 http: TLS handshake error from 10.0.0.1:44106: tls: first record does not look like a TLS handshake
caddy                  | 2020/01/14 06:43:36 http: TLS handshake error from 10.0.0.1:44108: tls: first record does not look like a TLS handshake
caddy                  | 2020/01/14 06:43:36 http: TLS handshake error from 10.0.0.1:44110: tls: first record does not look like a TLS handshake
caddy                  | 2020/01/14 06:43:36 http: TLS handshake error from 10.0.0.1:44112: tls: first record does not look like a TLS handshake
caddy                  | 2020/01/14 06:43:41 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2278549512
caddy                  | 2020/01/14 06:43:41 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2278549512
caddy                  | 2020/01/14 06:43:41 [ERROR][anthony.nova.chat] failed to obtain certificate: acme: Error -> One or more domains had a problem:
caddy                  | [anthony.nova.chat] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Errorgetting validation data, url: (attempt 2/3; challenge=tls-alpn-01)
caddy                  | 2020/01/14 06:43:42 [INFO] [anthony.nova.chat] acme: Obtaining bundled SAN certificate
caddy                  | 2020/01/14 06:43:42 [INFO] [anthony.nova.chat] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2278550805
caddy                  | 2020/01/14 06:43:42 [INFO] [anthony.nova.chat] acme: use tls-alpn-01 solver
caddy                  | 2020/01/14 06:43:42 [INFO] [anthony.nova.chat] acme: Trying to solve TLS-ALPN-01
caddy                  | 2020/01/14 06:43:43 http: TLS handshake error from 10.0.0.1:44114: tls: first record does not look like a TLS handshake
caddy                  | 2020/01/14 06:43:43 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2278550805
caddy                  | 2020/01/14 06:43:44 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2278550805
caddy                  | 2020/01/14 06:43:44 [ERROR][anthony.nova.chat] failed to obtain certificate: acme: Error -> One or more domains had a problem:
caddy                  | [anthony.nova.chat] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Errorgetting validation data, url: (attempt 3/3; challenge=tls-alpn-01)
caddy                  | 2020/01/14 06:43:45 [INFO] [anthony.nova.chat] acme: Obtaining bundled SAN certificate
caddy                  | 2020/01/14 06:43:45 [INFO] [anthony.nova.chat] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2278551249
caddy                  | 2020/01/14 06:43:45 [INFO] [anthony.nova.chat] acme: Could not find solver for: tls-alpn-01
caddy                  | 2020/01/14 06:43:45 [INFO] [anthony.nova.chat] acme: use http-01 solver
caddy                  | 2020/01/14 06:43:45 [INFO] [anthony.nova.chat] acme: Trying to solve HTTP-01
caddy                  | 2020/01/14 06:43:45 [INFO] [anthony.nova.chat] Served key authentication
caddy                  | 2020/01/14 06:43:45 [INFO] [anthony.nova.chat] Served key authentication
caddy                  | 2020/01/14 06:43:50 [INFO] [anthony.nova.chat] The server validated our request
caddy                  | 2020/01/14 06:43:50 [INFO] [anthony.nova.chat] acme: Validations succeeded; requesting certificates
caddy                  | 2020/01/14 06:43:50 [INFO] [anthony.nova.chat] Server responded with a certificate.
caddy                  | done.
caddy                  |
caddy                  | Serving HTTP on port 80
caddy                  | http://anthony.nova.chat
caddy                  |
caddy                  | 2020/01/14 06:43:50 [INFO] Serving http://anthony.nova.chat
caddy                  |
caddy                  | Serving HTTPS on port 443
caddy                  | https://anthony.nova.chat
caddy                  |
caddy                  | 2020/01/14 06:43:50 [INFO] Serving https://anthony.nova.chat

Ok, I think I fixed the haproxy problem. But occasionally I still get this error when trying to start caddy for first time on a new subdomain. Any suggestions for things I can do to debug? Thanks for the tips so far @Whitestrake!

Attaching to caddy
caddy                  | 2020/01/19 07:07:19 [INFO] Caddy version: v1.0.4
caddy                  | Activating privacy features... 2020/01/19 07:07:19 [INFO][cache:0x40001e0140] Started certificate maintenance routine
caddy                  | 2020/01/19 07:07:19 [INFO] acme: Registering account for ericmigi@gmail.com
caddy                  | 2020/01/19 07:07:19 [INFO][tidjane.nova.chat] Obtain certificate
caddy                  | 2020/01/19 07:07:19 [INFO][tidjane.nova.chat] Obtain: Waiting on rate limiter...
caddy                  | 2020/01/19 07:07:19 [INFO][tidjane.nova.chat] Obtain: Done waiting
caddy                  | 2020/01/19 07:07:19 [INFO] [tidjane.nova.chat] acme: Obtaining bundled SAN certificate
caddy                  | 2020/01/19 07:07:20 [INFO] [tidjane.nova.chat] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2362104287
caddy                  | 2020/01/19 07:07:20 [INFO] [tidjane.nova.chat] acme: use tls-alpn-01 solver
caddy                  | 2020/01/19 07:07:20 [INFO] [tidjane.nova.chat] acme: Trying to solve TLS-ALPN-01
caddy                  | 2020/01/19 07:07:26 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2362104287
caddy                  | 2020/01/19 07:07:26 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2362104287
caddy                  | 2020/01/19 07:07:26 [ERROR][tidjane.nova.chat] failed to obtain certificate: acme: Error -> One or more domains had a problem:
caddy                  | [tidjane.nova.chat] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Error getting validation data, url: (attempt 1/3; challenge=tls-alpn-01)
caddy                  | 2020/01/19 07:07:27 [INFO] [tidjane.nova.chat] acme: Obtaining bundled SAN certificate
caddy                  | 2020/01/19 07:07:28 [INFO] [tidjane.nova.chat] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2362105487
caddy                  | 2020/01/19 07:07:28 [INFO] [tidjane.nova.chat] acme: use tls-alpn-01 solver
caddy                  | 2020/01/19 07:07:28 [INFO] [tidjane.nova.chat] acme: Trying to solve TLS-ALPN-01
caddy                  | 2020/01/19 07:07:29 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2362105487
caddy                  | 2020/01/19 07:07:29 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2362105487
caddy                  | 2020/01/19 07:07:29 [ERROR][tidjane.nova.chat] failed to obtain certificate: acme: Error -> One or more domains had a problem:
caddy                  | [tidjane.nova.chat] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Error getting validation data, url: (attempt 2/3; challenge=tls-alpn-01)
caddy                  | 2020/01/19 07:07:30 [INFO] [tidjane.nova.chat] acme: Obtaining bundled SAN certificate
caddy                  | 2020/01/19 07:07:30 [INFO] [tidjane.nova.chat] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2362105885
caddy                  | 2020/01/19 07:07:30 [INFO] [tidjane.nova.chat] acme: use tls-alpn-01 solver
caddy                  | 2020/01/19 07:07:30 [INFO] [tidjane.nova.chat] acme: Trying to solve TLS-ALPN-01
caddy                  | 2020/01/19 07:07:31 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2362105885
caddy                  | 2020/01/19 07:07:31 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2362105885
caddy                  | 2020/01/19 07:07:31 [ERROR][tidjane.nova.chat] failed to obtain certificate: acme: Error -> One or more domains had a problem:
caddy                  | [tidjane.nova.chat] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Error getting validation data, url: (attempt 3/3; challenge=tls-alpn-01)
caddy                  | 2020/01/19 07:07:32 [INFO] [tidjane.nova.chat] acme: Obtaining bundled SAN certificate
caddy                  | 2020/01/19 07:07:33 [INFO] [tidjane.nova.chat] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2362106227
caddy                  | 2020/01/19 07:07:33 [INFO] [tidjane.nova.chat] acme: Could not find solver for: tls-alpn-01
caddy                  | 2020/01/19 07:07:33 [INFO] [tidjane.nova.chat] acme: use http-01 solver
caddy                  | 2020/01/19 07:07:33 [INFO] [tidjane.nova.chat] acme: Trying to solve HTTP-01
caddy                  | 2020/01/19 07:07:33 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2362106227
caddy                  | 2020/01/19 07:07:33 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2362106227
caddy                  | 2020/01/19 07:07:33 [ERROR][tidjane.nova.chat] failed to obtain certificate: acme: Error -> One or more domains had a problem:
caddy                  | [tidjane.nova.chat] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Invalid response from http://tidjane.nova.chat/.well-known/acme-challenge/MNVTLu_1gk5AXf0m4x5XAoZhUq8GZrkivNpbBHEvoUA [68.183.251.172]: 503, url: (attempt 1/3; challenge=http-01)
caddy                  | 2020/01/19 07:07:34 [INFO] [tidjane.nova.chat] acme: Obtaining bundled SAN certificate
caddy                  | 2020/01/19 07:07:35 [INFO] [tidjane.nova.chat] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2362106466
caddy                  | 2020/01/19 07:07:35 [INFO] [tidjane.nova.chat] acme: Could not find solver for: tls-alpn-01
caddy                  | 2020/01/19 07:07:35 [INFO] [tidjane.nova.chat] acme: use http-01 solver
caddy                  | 2020/01/19 07:07:35 [INFO] [tidjane.nova.chat] acme: Trying to solve HTTP-01
caddy                  | 2020/01/19 07:07:35 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2362106466
caddy                  | 2020/01/19 07:07:35 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/2362106466
caddy                  | 2020/01/19 07:07:35 [ERROR][tidjane.nova.chat] failed to obtain certificate: acme: Error -> One or more domains had a problem:
caddy                  | [tidjane.nova.chat] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Invalid response from http://tidjane.nova.chat/.well-known/acme-challenge/TJsyn5WpF5Mzp_kmztuYH9IF9uAZHtlXef1sVTK2dKU [68.183.251.172]: 503, url: (attempt 2/3; challenge=http-01)
caddy                  | 2020/01/19 07:07:36 [INFO] [tidjane.nova.chat] acme: Obtaining bundled SAN certificate
caddy                  | 2020/01/19 07:07:36 [ERROR][tidjane.nova.chat] failed to obtain certificate: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/, url: (attempt 3/3; challenge=http-01)
caddy                  | 2020/01/19 07:07:38 failed to obtain certificate: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/, url: 
caddy exited with code 1

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.