Cloudflare DNS challenge for TLS -- some subdomains work, others fail DNS challenge

[evamvid (Evamvid Sharma)]

1. Caddy version (2.0.0-rc3):

2. How I run Caddy:

I’m running Caddy from a docker container, using docker-compose.

a. System environment:

Ubuntu Server 20.04, Docker version 19.03.6

b. Command:

cd caddy && docker-compose up -d

c. Service/unit/compose file:

version: "3"
services:
  caddy:
    build: .
    container_name: caddy
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/Berlin
      - CLOUDFLARE_EMAIL=[email]
     - CLOUDFLARE_API_TOKEN=[token]
      - ACME_AGREE=true
    volumes:
      - /home/user/caddy/Caddyfile:/etc/caddy/Caddyfile
      - /home/user/caddy/site:/usr/share/caddy
      - /home/user/caddy/caddy_data:/data
      - /home/user/caddy/caddy_config:/config
    ports:
      - 80:80
      - 443:443
    networks:
      - t2_proxy
    restart: unless-stopped
networks:
  t2_proxy:
    external: true

d. My complete Caddyfile or JSON config:

{
  email username@gmail.com
}

test.my.domain {
    tls { 
      dns cloudflare {env.CLOUDFLARE_API_TOKEN}
      }
     respond "Hello, World!"
}

jellyfin.my.domain {
    encode gzip
        log {
            output file /data/jellyfin.log {
                roll true               # Rotate logs, enabled by default
                roll_size_mb 5          # Set max size 5 MB
                roll_gzip true          # Whether to compress rolled files
                roll_local_time true    # Use localhost time
                roll_keep 2             # Keep at most 2 log files
                roll_keep_days 7        # Keep log files for 7 days
            }
        }
	tls {
        dns cloudflare {env.CLOUDFLARE_API_TOKEN}
        }
        reverse_proxy 172.23.0.5:8096
}

firefly.my.domain {
    encode gzip
        log {
            output file /data/fireflyiii.log {
                roll true               # Rotate logs, enabled by default
                roll_size_mb 5          # Set max size 5 MB
                roll_gzip true          # Whether to compress rolled files
                roll_local_time true    # Use localhost time
                roll_keep 2             # Keep at most 2 log files
                roll_keep_days 7        # Keep log files for 7 days
            }
        }
	tls {
        dns cloudflare {env.CLOUDFLARE_API_TOKEN}
        }
        reverse_proxy 172.23.0.6:80 {
	   header_up X-Forwarded-Proto "https"
	}
}

portainer.my.domain {
encode gzip
        log {
            output file /data/fireflyiii.log {
                roll true               # Rotate logs, enabled by default
                roll_size_mb 5          # Set max size 5 MB
                roll_gzip true          # Whether to compress rolled files
                roll_local_time true    # Use localhost time
                roll_keep 2             # Keep at most 2 log files
                roll_keep_days 7        # Keep log files for 7 days
            }
        }
        tls {
        dns cloudflare {env.CLOUDFLARE_API_TOKEN}
        }
        reverse_proxy 172.23.0.3:9000
}

my.domain {
	tls {
  	dns cloudflare {env.CLOUDFLARE_API_TOKEN}
	}
	respond "Hello, world!"
}

3. The problem I’m having:

I use Caddy as a reverse proxy for different subdomains to different docker containers. The problem is that TLS only works for the subdomains that I had already been using with Traefik2 (I switched a few weeks ago). So the root domain works, the jellyfin subdomain works, and the firefly subdomain works. In other words, only domains that I had already successfully used the Cloudflare DNS challenge with.

The problem is that all of the others give SSL errors. The speicif errors are in the log below from the server side; trying to access the pages from Firefox gives a “Secure Connection Failed” error with error code: SSL_ERROR_INTERNAL_ERROR_ALERT (which makes sense).

My DNS in Cloudflare has a A record to the root domain, and a wildcard A record for the subdomains, both pointing to my IP.

4. Error messages and/or full log output:

{"level":"info","ts":1594733449.2572584,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"},
{"level":"info","ts":1594733449.2897034,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]},
{"level":"info","ts":1594733449.311245,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"},
2020/07/14 13:30:49 [INFO][cache:0xc0003113b0] Started certificate maintenance routine,
{"level":"info","ts":1594733449.459557,"logger":"tls","msg":"cleaned up storage units"},
{"level":"info","ts":1594733449.4636476,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["jellyfin.my.domain","firefly.my.domain","test.my.domain","my.domain","portainer.my.domain"]},
{"level":"info","ts":1594733449.5477698,"msg":"autosaved config","file":"/config/caddy/autosave.json"},
{"level":"info","ts":1594733449.5482585,"msg":"serving initial configuration"},
2020/07/14 13:30:49 [INFO][portainer.my.domain] Obtain certificate; acquiring lock...,
2020/07/14 13:30:49 [INFO][portainer.my.domain] Obtain: Lock acquired; proceeding...,
2020/07/14 13:30:49 [INFO][test.my.domain] Obtain certificate; acquiring lock...,
2020/07/14 13:30:49 [INFO][test.my.domain] Obtain: Lock acquired; proceeding...,
2020/07/14 13:30:50 [INFO][portainer.my.domain] Waiting on rate limiter...,
2020/07/14 13:30:50 [INFO][portainer.my.domain] Done waiting,
2020/07/14 13:30:50 [INFO] [portainer.my.domain] acme: Obtaining bundled SAN certificate given a CSR,
2020/07/14 13:30:50 [INFO][test.my.domain] Waiting on rate limiter...,
2020/07/14 13:30:50 [INFO][test.my.domain] Done waiting,
2020/07/14 13:30:50 [INFO] [test.my.domain] acme: Obtaining bundled SAN certificate given a CSR,
2020/07/14 13:30:51 [INFO] [test.my.domain] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5876525814,
2020/07/14 13:30:51 [INFO] [test.my.domain] acme: Could not find solver for: tls-alpn-01,
2020/07/14 13:30:51 [INFO] [test.my.domain] acme: Could not find solver for: http-01,
2020/07/14 13:30:51 [INFO] [test.my.domain] acme: use dns-01 solver,
2020/07/14 13:30:51 [INFO] [test.my.domain] acme: Preparing to solve DNS-01,
2020/07/14 13:30:51 [INFO] [portainer.my.domain] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5876525805,
2020/07/14 13:30:51 [INFO] [portainer.my.domain] acme: Could not find solver for: tls-alpn-01,
2020/07/14 13:30:51 [INFO] [portainer.my.domain] acme: Could not find solver for: http-01,
2020/07/14 13:30:51 [INFO] [portainer.my.domain] acme: use dns-01 solver,
2020/07/14 13:30:51 [INFO] [portainer.my.domain] acme: Preparing to solve DNS-01,
2020/07/14 13:30:51 [INFO] [test.my.domain] acme: Cleaning DNS-01 challenge,
2020/07/14 13:30:51 [WARN] [test.my.domain] acme: cleaning up failed: no memory of presenting a DNS record for test.my.domain ,
2020/07/14 13:30:52 [INFO] [portainer.my.domain] acme: Cleaning DNS-01 challenge,
2020/07/14 13:30:52 [WARN] [portainer.my.domain] acme: cleaning up failed: no memory of presenting a DNS record for portainer.my.domain ,
2020/07/14 13:30:52 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5876525814,
2020/07/14 13:30:52 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5876525805,
2020/07/14 13:30:52 [ERROR] error: one or more domains had a problem:,
[test.my.domain] [test.my.domain] acme: error presenting token: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}],
 (challenge=dns-01 remaining=[]),
2020/07/14 13:30:52 [ERROR] error: one or more domains had a problem:,
[portainer.my.domain] [portainer.my.domain] acme: error presenting token: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}],
 (challenge=dns-01 remaining=[]),
2020/07/14 13:30:54 [ERROR] attempt 1: [test.my.domain] Obtain: [test.my.domain] error: one or more domains had a problem:,
[test.my.domain] [test.my.domain] acme: error presenting token: got error status: HTTP 400: [{Code:6003 Messge:Invalid request headers}],
 - retrying in 1m0s (4.748711077s/720h0m0s elapsed)...,
2020/07/14 13:30:54 [ERROR] attempt 1: [portainer.my.domain] Obtain: [portainer.my.domain] error: one or more domains had a problem:,
[portainer.my.domain] [portainer.my.domain] acme: error presenting token: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}],
 - retrying in 1m0s (5.038788328s/720h0m0s elapsed)...,
2020/07/14 13:31:55 [INFO] [test.my.domain] acme: Obtaining bundled SAN certificate given a CSR,
2020/07/14 13:31:55 [INFO] [portainer.my.domain] acme: Obtaining bundled SAN certificate given a CSR,
2020/07/14 13:31:55 [INFO] [test.my.domain] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/77245172,
2020/07/14 13:31:55 [INFO] [test.my.domain] acme: Could not find solver for: tls-alpn-01,
2020/07/14 13:31:55 [INFO] [test.my.domain] acme: Could not find solver for: http-01,
2020/07/14 13:31:55 [INFO] [test.my.domain] acme: use dns-01 solver,
2020/07/14 13:31:55 [INFO] [test.my.domain] acme: Preparing to solve DNS-01,
2020/07/14 13:31:55 [INFO] [portainer.my.domain] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/77245174,
2020/07/14 13:31:55 [INFO] [portainer.my.domain] acme: Could not find solver for: tls-alpn-01,
2020/07/14 13:31:55 [INFO] [portainer.my.domain] acme: Could not find solver for: http-01,
2020/07/14 13:31:55 [INFO] [portainer.my.domain] acme: use dns-01 solver,
2020/07/14 13:31:55 [INFO] [portainer.my.domain] acme: Preparing to solve DNS-01,
2020/07/14 13:31:56 [INFO] [test.my.domain] acme: Cleaning DNS-01 challenge,
2020/07/14 13:31:56 [WARN] [test.my.domain] acme: cleaning up failed: no memory of presenting a DNS record for test.my.domain ,
2020/07/14 13:31:56 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/77245172,
2020/07/14 13:31:56 [INFO] [portainer.my.domain] acme: Cleaning DNS-01 challenge,
2020/07/14 13:31:56 [WARN] [portainer.my.domain] acme: cleaning up failed: no memory of presenting a DNS record for portainer.my.domain ,
2020/07/14 13:31:56 [ERROR] error: one or more domains had a problem:,
[test.my.domain] [test.my.domain] acme: error presenting token: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}],
 (challenge=dns-01 remaining=[]),
2020/07/14 13:31:56 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/77245174,
2020/07/14 13:31:56 [ERROR] error: one or more domains had a problem:,
[portainer.my.domain] [portainer.my.domain] acme: error presenting token: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}],
 (challenge=dns-01 remaining=[]),
2020/07/14 13:31:58 [ERROR] attempt 2: [test.my.domain] Obtain: [test.my.domain] error: one or more domains had a problem:,
[test.my.domain] [test.my.domain] acme: error presenting token: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}],
 - retrying in 2m0s (1m8.791609463s/720h0m0s elapsed)...,
2020/07/14 13:31:58 [ERROR] attempt 2: [portainer.my.domain] Obtain: [portainer.my.domain] error: one or more domains had a problem:,
[portainer.my.domain] [portainer.my.domain] acme: error presenting token: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}],
 - retrying in 2m0s (1m9.054859565s/720h0m0s elapsed)...,
2020/07/14 13:33:58 [INFO] [test.my.domain] acme: Obtaining bundled SAN certificate given a CSR,
2020/07/14 13:33:58 [INFO] [portainer.my.domain] acme: Obtaining bundled SAN certificate given a CSR,
2020/07/14 13:33:58 [INFO] [test.my.domain] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/77245663,
2020/07/14 13:33:58 [INFO] [test.my.domain] acme: Could not find solver for: tls-alpn-01,
2020/07/14 13:33:58 [INFO] [test.my.domain] acme: Could not find solver for: http-01,
2020/07/14 13:33:58 [INFO] [test.my.domain] acme: use dns-01 solver,
2020/07/14 13:33:58 [INFO] [test.my.domain] acme: Preparing to solve DNS-01,
2020/07/14 13:33:59 [INFO] [portainer.my.domain] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/77245667,
2020/07/14 13:33:59 [INFO] [portainer.my.domain] acme: Could not find solver for: tls-alpn-01,
2020/07/14 13:33:59 [INFO] [portainer.my.domain] acme: Could not find solver for: http-01,
2020/07/14 13:33:59 [INFO] [portainer.my.domain] acme: use dns-01 solver,
2020/07/14 13:33:59 [INFO] [portainer.my.domain] acme: Preparing to solve DNS-01,
2020/07/14 13:33:59 [INFO] [test.my.domain] acme: Cleaning DNS-01 challenge,
2020/07/14 13:33:59 [WARN] [test.my.domain] acme: cleaning up failed: no memory of presenting a DNS record for test.my.domain ,
2020/07/14 13:33:59 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/77245663,
2020/07/14 13:33:59 [INFO] [portainer.my.domain] acme: Cleaning DNS-01 challenge,
2020/07/14 13:33:59 [WARN] [portainer.my.domain] acme: cleaning up failed: no memory of presenting a DNS record for portainer.my.domain ,
2020/07/14 13:33:59 [ERROR] error: one or more domains had a problem:,
[test.my.domain] [test.my.domain] acme: error presenting token: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}],
 (challenge=dns-01 remaining=[]),
2020/07/14 13:33:59 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/77245667,
2020/07/14 13:33:59 [ERROR] error: one or more domains had a problem:,
[portainer.my.domain] [portainer.my.domain] acme: error presenting token: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}],
 (challenge=dns-01 remaining=[]),
2020/07/14 13:34:01 [ERROR] attempt 3: [test.my.domain] Obtain: [test.my.domain] error: one or more domains had a problem:,
[test.my.domain] [test.my.domain] acme: error presenting token: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}],
 - retrying in 2m0s (3m11.896682731s/720h0m0s elapsed)...,
2020/07/14 13:34:01 [ERROR] attempt 3: [portainer.my.domain] Obtain: [portainer.my.domain] error: one or more domains had a problem:,
[portainer.my.domain] [portainer.my.domain] acme: error presenting token: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}],
 - retrying in 2m0s (3m12.13822479s/720h0m0s elapsed)...,
2020/07/14 13:36:01 [INFO] [test.my.domain] acme: Obtaining bundled SAN certificate given a CSR,
2020/07/14 13:36:01 [INFO] [portainer.my.domain] acme: Obtaining bundled SAN certificate given a CSR,
2020/07/14 13:36:01 [INFO] [test.my.domain] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/77246102,
2020/07/14 13:36:01 [INFO] [test.my.domain] acme: Could not find solver for: tls-alpn-01,
2020/07/14 13:36:01 [INFO] [test.my.domain] acme: Could not find solver for: http-01,
2020/07/14 13:36:01 [INFO] [test.my.domain] acme: use dns-01 solver,
2020/07/14 13:36:01 [INFO] [test.my.domain] acme: Preparing to solve DNS-01,
2020/07/14 13:36:02 [INFO] [test.my.domain] acme: Cleaning DNS-01 challenge,
2020/07/14 13:36:02 [WARN] [test.my.domain] acme: cleaning up failed: no memory of presenting a DNS record for test.my.domain ,
2020/07/14 13:36:02 [INFO] [portainer.my.domain] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/77246107,
2020/07/14 13:36:02 [INFO] [portainer.my.domain] acme: Could not find solver for: tls-alpn-01,
2020/07/14 13:36:02 [INFO] [portainer.my.domain] acme: Could not find solver for: http-01,
2020/07/14 13:36:02 [INFO] [portainer.my.domain] acme: use dns-01 solver,
2020/07/14 13:36:02 [INFO] [portainer.my.domain] acme: Preparing to solve DNS-01,
2020/07/14 13:36:02 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/77246102,
2020/07/14 13:36:02 [INFO] [portainer.my.domain] acme: Cleaning DNS-01 challenge,
2020/07/14 13:36:02 [WARN] [portainer.my.domain] acme: cleaning up failed: no memory of presenting a DNS record for portainer.my.domain ,
2020/07/14 13:36:02 [ERROR] error: one or more domains had a problem:,
[test.my.domain] [test.my.domain] acme: error presenting token: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}],
 (challenge=dns-01 remaining=[]),
2020/07/14 13:36:02 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/77246107,
2020/07/14 13:36:02 [ERROR] error: one or more domains had a problem:,
[portainer.my.domain] [portainer.my.domain] acme: error presenting token: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}],
 (challenge=dns-01 remaining=[]),
2020/07/14 13:36:04 [ERROR] attempt 4: [test.my.domain] Obtain: [test.my.domain] error: one or more domains had a problem:,
[test.my.domain] [test.my.domain] acme: error presenting token: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}],
 - retrying in 5m0s (5m14.95093963s/720h0m0s elapsed)...,
2020/07/14 13:36:04 [ERROR] attempt 4: [portainer.my.domain] Obtain: [portainer.my.domain] error: one or more domains had a problem:,
[portainer.my.domain] [portainer.my.domain] acme: error presenting token: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}],
 - retrying in 5m0s (5m15.234327361s/720h0m0s elapsed)...

5. What I already tried:

I tried switching to ALPN authentication on some subdomains, but that gave similar errors (and my understanding is that that doesn’t play nice with Cloudflare’s orange cloud proxy anyway). I’d prefer to use DNS authentication if at all possible.

Other than that, I haven’t really tried much, for the simple reason that I have no idea what I should try. I found similar issues on this forum, but they all seemed to be caused by different things that aren’t applicable to my setup. I’m not even quite sure if this a caddy issue or something with Let’s Encrypt!

(The Cloudflare API token is the global account one – using the wrong API token was an issue in some of the other posts I found, but should be correct here.

6. Links to relevant resources:

One particular domain refuses to get a proper SSL cert using Cloudflare DNS, while others work fine

We don’t know where the issue lies or how to fix it, either. Generally the best way to find out is to reproduce the issue and investigate it in place. That means if you want help, you must help yourself in some way. Either by hiring a professional to investigate privately, or sharing something we can use to reproduce the issue ourselves and study it, or simply continuing to troubleshoot it on your own with what guidance we can provide (as you have been doing up to this point). It seems like th…

This issue was similar, but seems to have been solved with a rebuild. This (as far as I know) isn’t something I can do since I’m just pulling the container from DockerHub.

[francislavoie]

You’re using a pretty old version of Caddy, the latest is 2.1.1, which is definitely available on docker hub. Could you try again with the latest version to see if it works, to narrow things down?

[matt (Matt Holt)]

I seem to recall seeing that error before… and it was really obscure… but I think it was fixed. I don’t recall the details because the first commit of libdns/cloudflare was just a few days before the 2.0 release. Still, if it was a bug (and is not in fact a misconfiguration), then I think it was fixed…

[evamvid (Evamvid Sharma)]

I ddin’t realize I had an outdated version of Caddy! I pulled v2.1.1 from DockerHub, but the problem persists with exactly the same error.

[matt (Matt Holt)]

Oh… are you sure you’re using an API token and not an API key? GitHub - libdns/cloudflare: Cloudflare provider implementation for libdns

- CLOUDFLARE_EMAIL=[email]

Because that looks suspect (like from their old API).

[evamvid (Evamvid Sharma)]

That was it, thanks a lot! I was using the global API key – after creating a scoped API token and rebooting caddy, all of the subdomains are working.