Caddy Cloudflare SSL Cert Challenge doesn't work on 1 Domain

isikerkan · March 21, 2021, 1:46pm

1. Caddy version (`caddy version`):

v2.3.0 h1:fnrqJLa3G5vfxcxmOH/+kJOcunPLhSBnjgIvjXV/QTA=

2. How I run Caddy:

a. System environment:

Ubuntu 20.04, Caddy native on OS, systemd 245
DNS + Caching trough Cloudflare with Strict SSL

b. Command:

caddy run Caddyfile

c. Service/unit/compose file:

d. My complete Caddyfile or JSON config:

{
        http_port 80
        https_port 443
        email {email}
}

sasf.ch {
        root * /var/www/sasf
        file_server

        log {
                output file /var/log/caddy/sasf.ch.access.log {
                        roll_size 3MiB
                        roll_keep 5
                        roll_keep_for 48h
                }
                format console
        }

        encode gzip zstd
        php_fastcgi unix//run/php/php7.4-fpm.sock

        tls {email} {
                dns cloudflare {API_KEY}
        }
}

asvd.sasf.ch {
        root * /var/www/asvd
        file_server

        log {
                output file /var/log/caddy/asvd.sasf.ch.access.log {
                        roll_size 3MiB
                        roll_keep 5
                        roll_keep_for 48h
                }
                format console
        }

        basicauth /uploads/* {
                {User} {Hash}
        }

        encode gzip zstd
        php_fastcgi unix//run/php/php7.4-fpm.sock

        tls {email} {
                dns cloudflare {API_KEY}
        }
}

discourse.asvd.sasf.ch {
        reverse_proxy 127.0.0.1:12080 {
                header_up Host {http.request.host}
                header_up X-Real-IP {http.request.remote}
                header_up X-Forwarded-For {http.request.remote}
                header_up X-Forwarded-Port {http.request.port}
                header_up X-Forwarded-Proto {http.request.scheme}
        }

        log {
                output file /var/log/caddy/discourse.access.log {
                        roll_size 3MiB
                        roll_keep 5
                        roll_keep_for 48h
                }
                format console
        }

        tls {email} {
                dns cloudflare {API_KEY}
        }
}

as-starter.ch {
        root * /var/www/as-starter
        file_server

        log {
                output file /var/log/caddy/as-starter.ch.access.log {
                        roll_size 3MiB
                        roll_keep 5
                        roll_keep_for 48h
                }
                format console
                #       format single_field common_log
        }

        encode gzip zstd
        php_fastcgi unix//run/php/php8.0-fpm.sock

        tls {email} {
                dns cloudflare {API_KEY}
        }
}

grafana.sasf.ch {
        reverse_proxy 127.0.0.1:3000
        file_server

        log {
                output file /var/log/caddy/grafana.access.log {
                        roll_size 3MiB
                        roll_keep 5
                        roll_keep_for 48h
                }
                format console
        }

        tls {email} {
                dns cloudflare {API_KEY}
        }
}

php.sasf.ch {
        root * /var/www/phpmyadmin
        file_server

        log {
                output file /var/log/caddy/php.sasf.ch.access.log {
                        roll_size 3MiB
                        roll_keep 5
                        roll_keep_for 48h
                }
                format console
        }

        encode gzip zstd
        php_fastcgi unix//run/php/php7.4-fpm.sock

        tls {email} {
                dns cloudflare {API_KEY}
        }

        basicauth {
                {User} {Hash}
        }
}

stage.sasf.ch {
        root * /var/www/stage/asvd
        file_server

        log {
                output file /var/log/caddy/stage.sasf.ch.access.log {
                        roll_size 3MiB
                        roll_keep 5
                        roll_keep_for 24h
                }
                format console
        }

        encode gzip zstd
        php_fastcgi unix//run/php/php7.4-fpm.sock

        tls {email} {
                dns cloudflare {API_KEY}
        }

        basicauth {
                {User} {Hash}
        }
}

webmin.sasf.ch {
        reverse_proxy http://127.0.0.1:10000 {
                header_up Host {http.request.host}
                header_up X-Real-IP {http.request.remote}
                header_up X-Forwarded-For {http.request.remote}
                header_up X-Forwarded-Port {http.request.port}
                header_up X-Forwarded-Proto {http.request.scheme}
        }

        file_server

        log {
                output file /var/log/caddy/webmin.access.log {
                        roll_size 3MiB
                        roll_keep 5
                        roll_keep_for 48h
                }
                format console
        }

        tls {email} {
                dns cloudflare {API_KEY}
        }
}

fullet.store {
        root * /var/www/fullet
        file_server

        log {
                output file /var/log/caddy/fullet.access.log {
                        roll_size 3MiB
                        roll_keep 5
                        roll_keep_for 48h
                }
                format console
        }

        encode gzip zstd
        php_fastcgi unix//run/php/php7.4-fpm.sock

        tls {email} {
                dns cloudflare {API_KEY}
        }
}

ascon.sasf.ch {
        root * /var/www/airsoftcon
        file_server

        log {
                output file /var/log/caddy/airsoftcon.access.log {
                        roll_size 3MiB
                        roll_keep 5
                        roll_keep_for 48h
                }
                format console
        }

        encode gzip zstd
        php_fastcgi unix//run/php/php7.4-fpm.sock

        tls {email} {
                dns cloudflare {API_KEY}
        }
}

safe.sasf.ch {
        root * /usr/share/php/passbolt/webroot
        file_server

        log {
                output file /var/log/caddy/safe.access.log {
                        roll_size 3MiB
                        roll_keep 5
                        roll_keep_for 48h
                }
                format console
        }

        encode gzip zstd
        php_fastcgi unix//run/php/php7.4-fpm.sock

        tls {email} {
                dns cloudflare {API_KEY}
        }
}

www.sasf.ch {
        redir https://sasf.ch{uri}

        tls {email} {
                dns cloudflare {API_KEY}
        }
}

www.asvd.sasf.ch {
        redir https://asvd.sasf.ch{uri}

        tls {email} {
                dns cloudflare {API_KEY}
        }
}
www.discourse.asvd.sasf.ch {
        redir https://discourse.asvd.sasf.ch{uri}

        tls {email} {
                dns cloudflare {API_KEY}
        }
}

www.as-starter.ch {
        redir https://as-starter.ch{uri}

        tls {email} {
                dns cloudflare {API_KEY}
        }
}

www.grafana.sasf.ch {
        redir https://grafana.sasf.ch{uri}

        tls {email} {
                dns cloudflare {API_KEY}
        }
}

www.php.sasf.ch {
        redir https://php.sasf.ch{uri}

        tls {email} {
                dns cloudflare {API_KEY}
        }
}

www.stage.sasf.ch {
        redir https://stage.sasf.ch{uri}

        tls {email} {
                dns cloudflare {API_KEY}
        }
}

www.webmin.sasf.ch {
        redir https://webmin.sasf.ch{uri}

        tls {email} {
                dns cloudflare {API_KEY}
        }
}

www.fullet.store {
        redir https://fullet.store{uri}

        tls {email} {
                dns cloudflare {API_KEY}
        }
}

www.ascon.sasf.ch {
        redir https://ascon.sasf.ch{uri}

        tls {email} {
                dns cloudflare {API_KEY}
        }
}

www.safe.sasf.ch {
        redir https://safe.sasf.ch{uri}

        tls {email} {
                dns cloudflare {API_KEY}
        }
}

3. The problem I’m having:

I’ve routed my Origin Server trough Cloudflare with the CDN Service activated. The Problem I’m facing is, when I try to obtain a certificate for *sasf.ch with Cloudflare active, the Challenges can’t be solved. I’ve built Caddy with the Cloudflare Plugin and created an API Key with Read&Write Permissions for the whole Zone. For every other Site obtaining a Cert works with Cloudflare active.

The Strange thing is when I switch Cloudflare to only DNS for *sasf.ch, I can obtain the SSL Certificates and the DNS Challenges gets written _acme-challenge.

4. Error messages and/or full log output:

Mar 21 13:42:35 asvd caddy[2385887]: {“level”:“info”,“ts”:1616330555.8242142,“logger”:“tls.issuance.acme”,“msg”:“done waiting on internal rate limiter”,“identifiers”:[“www.sasf.ch”]}
Mar 21 13:42:37 asvd caddy[2385887]: {“level”:“info”,“ts”:1616330557.022536,“logger”:“tls.issuance.acme.acme_client”,“msg”:“trying to solve challenge”,“identifier”:“www.ascon.sasf.ch”,“challenge_type”:“dns-01”,“ca”:“https://acme-v02.api.letsencrypt.org/directory”}
Mar 21 13:42:37 asvd caddy[2385887]: {“level”:“info”,“ts”:1616330557.4230819,“logger”:“tls.issuance.acme.acme_client”,“msg”:“trying to solve challenge”,“identifier”:“www.php.sasf.ch”,“challenge_type”:“dns-01”,“ca”:“https://acme-v02.api.letsencrypt.org/directory”}
Mar 21 13:42:37 asvd caddy[2385887]: {“level”:“info”,“ts”:1616330557.4250066,“logger”:“tls.issuance.acme.acme_client”,“msg”:“trying to solve challenge”,“identifier”:“www.sasf.ch”,“challenge_type”:“dns-01”,“ca”:“https://acme-v02.api.letsencrypt.org/directory”}
Mar 21 13:42:37 asvd caddy[2385887]: {“level”:“info”,“ts”:1616330557.4319265,“logger”:“tls.issuance.acme.acme_client”,“msg”:“trying to solve challenge”,“identifier”:“www.asvd.sasf.ch”,“challenge_type”:“tls-alpn-01”,“ca”:“https://acme-v02.api.letsencrypt.org/directory”}
Mar 21 13:42:37 asvd caddy[2385887]: {“level”:“info”,“ts”:1616330557.4746373,“logger”:“tls.issuance.acme.acme_client”,“msg”:“trying to solve challenge”,“identifier”:“www.webmin.sasf.ch”,“challenge_type”:“dns-01”,“ca”:“https://acme-v02.api.letsencrypt.org/directory”}
Mar 21 13:42:38 asvd caddy[2385887]: {“level”:“error”,“ts”:1616330558.0501175,“logger”:“tls.issuance.acme.acme_client”,“msg”:“challenge failed”,“identifier”:“www.asvd.sasf.ch”,“challenge_type”:“tls-alpn-01”,“status_code”:400,“problem_type”:“urn:ietf:params:acme:error:tls”,“error”:“remote error: tls: handshake failure”}
Mar 21 13:42:38 asvd caddy[2385887]: {“level”:“error”,“ts”:1616330558.0501742,“logger”:“tls.issuance.acme.acme_client”,“msg”:“validating authorization”,“identifier”:“www.asvd.sasf.ch”,“error”:“authorization failed: HTTP 400 urn:ietf:params:acme:error:tls - remote error: tls: handshake failure”,“order”:"https://acme-v02.api.letsencrypt.org/acme/order/94169920/>Mar 21 13:42:39 asvd caddy[2385887]: {“level”:“info”,“ts”:1616330559.3053281,“logger”:“tls.issuance.acme”,“msg”:“waiting on internal rate limiter”,“identifiers”:[“www.asvd.sasf.ch”]}
Mar 21 13:42:39 asvd caddy[2385887]: {“level”:“info”,“ts”:1616330559.3053942,“logger”:“tls.issuance.acme”,“msg”:“done waiting on internal rate limiter”,“identifiers”:[“www.asvd.sasf.ch”]}

5. What I already tried:

Created SSL Certs for other Site with Cloudflare active, worked without a Problem.
Controlled the Config Files. Disabled Cloudflare, then obtaining a Cert for sasf.ch worked.

6. Links to relevant resources:

francislavoie · March 21, 2021, 9:06pm

I notice that you’re repeating these lines a lot.

I recommend using the Caddyfile snippets feature so that you can just define that once then reuse it with an import in each of your sites.

isikerkan:

                header_up Host {http.request.host}
                header_up X-Real-IP {http.request.remote}
                header_up X-Forwarded-For {http.request.remote}
                header_up X-Forwarded-Port {http.request.port}
                header_up X-Forwarded-Proto {http.request.scheme}

You should remove all these lines. Caddy sets the relevant headers appropriately automatically.

That said, how did you build Caddy with Cloudflare? There’s a known issue involving recent versions of Caddy and the plugin. I just want to make sure you actually have a working build.

isikerkan · March 21, 2021, 9:30pm

Thanks for the Snippet/Import Tip. Didn’t know that, but after writing a Snippet for the repeating Lines, my Caddyfile got compact (YAY )

The Build Im using was downloaded from the official Caddy Site and I’ve replaced /usr/bin/caddy with it: Download Caddy (caddyserver.com)
Should I try to build it with xcaddy to troubleshoot?

Or could it be that I would need to add disable_tlsalpn_challenge, because my Origin IP is hidden? Caddy 2 behind Cloudflare CDN - Help - Caddy Community

isikerkan · March 21, 2021, 10:43pm

Just a heads up, after including disable_tlsalpn_challenge in the SSL Snippet, obtaining the Certs worked with Cloudflare enabled and SSL in Strict Mode

(ssl) {
    tls {email} {
        dns cloudflare {API Key}
        alpn disable_tlsalpn_challenge
    }
}

system · April 20, 2021, 1:47pm

This topic was automatically closed after 30 days. New replies are no longer allowed.