V2: Automatic https certificate errors

Continued from https://caddy.community/t/re-v2-config-doesnt-run-after-beta-9/6822.

Two more things to add before I create an issue on Github:

My automatic https setting for three domains is not working as expected.

All of the three dollowing sites have static htmls.

Caddyfile:

weather-frogs.de, www.weather-frogs.de {
        root * /etc/caddy/html/weather-frogs.de
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}
die-reikiquellen.de, www.die-reikiquellen.de {
        root * /etc/caddy/html/die-reikiquellen.de
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}
harmoniks.de, www.harmoniks.de {
        root * /etc/caddy/html/harmoniks.de
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}

Complete Caddy json config (extracted from autosave.json):

will give me this message in chrome:

This site can’t provide a secure connection
www.die-reikiquellen.de sent an invalid response.
Try running Windows Network Diagnostics.
ERR_SSL_PROTOCOL_ERROR

and this one in logs:

● v2caddy.service - Caddy v2
   Loaded: loaded (/lib/systemd/system/v2caddy.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2020-01-20 15:06:42 CET; 19min ago
     Docs: https://caddyserver.com/docs/
 Main PID: 1432 (caddy)
    Tasks: 15 (limit: 4915)
   Memory: 20.5M
   CGroup: /system.slice/v2caddy.service
           └─1432 /etc/v2caddy/caddy run --config /etc/v2caddy/Caddyfile --adapter caddyfile

Jan 20 15:06:42 hnrk.io caddy[1432]: 2020/01/20 14:06:42.269        INFO        http        skipping automatic certificate management because one or more matching certificates are already loaded        {"domain": "hnrk.io", "server_name": "srv1"}
Jan 20 15:06:42 hnrk.io caddy[1432]: 2020/01/20 14:06:42.269        INFO        http        skipping automatic certificate management because one or more matching certificates are already loaded        {"domain": "ip.hnrk.io", "server_name": "srv1"}
Jan 20 15:06:42 hnrk.io caddy[1432]: 2020/01/20 14:06:42.269        INFO        http        enabling automatic TLS certificate management        {"domains": ["die-reikiquellen.de", "www.harmoniks.de", "www.weather-frogs.de", "harmoniks.de", "www.die-reikiquellen.de", "weather-frogs.de"]}
Jan 20 15:06:42 hnrk.io caddy[1432]: 2020/01/20 14:06:42.270        INFO        http        enabling experimental HTTP/3 listener        {"addr": ":443"}
Jan 20 15:06:42 hnrk.io caddy[1432]: 2020/01/20 14:06:42.271        INFO        autosaved config        {"file": "/root/.config/caddy/autosave.json"}
Jan 20 15:06:42 hnrk.io caddy[1432]: 2020/01/20 14:06:42.271        INFO        serving initial configuration
Jan 20 15:06:43 hnrk.io caddy[1432]: 2020/01/20 15:06:43 http: TLS handshake error from 62.157.168.126:37494: no server TLS configuration available for ClientHello: &{CipherSuites:[14906 4865 4866 4867 49195 49199 49196 49200 52393 52392 49171 49172 156 157 47 53 10] ServerName:www.die-reikiquellen.de SupportedCurves:[10794 29 23 24] SupportedPoints:[0] SignatureSchemes:[1027 2052 1025 1283 2053 1281 2054 1537 513] SupportedProtos:[h2 http/1.1] SupportedVersions:[6682 772 771 770 769] Conn:0xc0000c6720}
Jan 20 15:06:44 hnrk.io caddy[1432]: 2020/01/20 15:06:44 http: TLS handshake error from 62.157.168.126:56285: no server TLS configuration available for ClientHello: &{CipherSuites:[2570 4865 4866 4867 49195 49199 49196 49200 52393 52392 49171 49172 156 157 47 53 10] ServerName:www.die-reikiquellen.de SupportedCurves:[35466 29 23 24] SupportedPoints:[0] SignatureSchemes:[1027 2052 1025 1283 2053 1281 2054 1537 513] SupportedProtos:[h2 http/1.1] SupportedVersions:[56026 772 771 770 769] Conn:0xc000652030}
Jan 20 15:06:45 hnrk.io caddy[1432]: 2020/01/20 15:06:45 http: TLS handshake error from 62.157.168.126:37524: no server TLS configuration available for ClientHello: &{CipherSuites:[51914 4865 4866 4867 49195 49199 49196 49200 52393 52392 49171 49172 156 157 47 53 10] ServerName:www.die-reikiquellen.de SupportedCurves:[19018 29 23 24] SupportedPoints:[0] SignatureSchemes:[1027 2052 1025 1283 2053 1281 2054 1537 513] SupportedProtos:[h2 http/1.1] SupportedVersions:[6682 772 771 770 769] Conn:0xc000652038}
Jan 20 15:20:22 hnrk.io caddy[1432]: 2020/01/20 15:20:22 http: TLS handshake error from 66.249.75.140:58356: no server TLS configuration available for ClientHello: &{CipherSuites:[60138 4865 4866 4867 49195 49199 52393 52392 49196 49200 49161 49171 49162 49172 156 157 47 53 10] ServerName:harmoniks.de SupportedCurves:[56026 29 23 24] SupportedPoints:[0] SignatureSchemes:[1027 2052 1025 1283 2053 1281 2054 1537 513] SupportedProtos:[] SupportedVersions:[14906 772 771 770 769] Conn:0xc00054e258}

Is there something missing when Caddy v2 adapts the config from Caddyfile style?

  1. Question: Is there a directive to configure markdown via Caddyfile style configuration?

Thank you!

Hi HNRK,
Looking at your past posts, the actual Caddyfile appears to have more settings.

Some of them are left in .json, which seems to be bad.

What I checked was the status of the certificate:
harmoniks. de’s certificate has not been issued recently,
but it was only issued last month and has not been renewed since it has not been 30 days old.

https://crt.sh/?q=weather-frogs.de
https://crt.sh/?q=die-reikiquellen.de
https://crt.sh/?q=harmoniks.de

Probably there is something wrong with such a long Caddyfile.
Or you need a mechanism to reset the json settings.

I know the implementation of markdown, but it leaves others to help.

1 Like

Hi,
thank you for your help!

Do you think it could help to flush the stored autosave.json and let Caddy generate a new one?
Tried this one, but it doesn’t change the behaviour.

With beta 9 it worked perfectly fine, or did I miss any breaking change for beta13?

I was able to refer to the file a bit more specifically.

From log file:

Jan 20 15:06:43 hnrk.io caddy[1432]: 2020/01/20 15:06:43 http: TLS handshake error from 62.157.168.126:37494: no server TLS configuration available for ClientHello: &{CipherSuites:[14906 4865 4866 4867 49195 49199 49196 49200 52393 52392 49171 49172 156 157 47 53 10] ServerName:www.die-reikiquellen.de SupportedCurves:[10794 29 23 24] SupportedPoints:[0] SignatureSchemes:[1027 2052 1025 1283 2053 1281 2054 1537 513] SupportedProtos:[h2 http/1.1] SupportedVersions:[6682 772 771 770 769] Conn:0xc0000c6720}
Jan 20 15:06:44 hnrk.io caddy[1432]: 2020/01/20 15:06:44 http: TLS handshake error from 62.157.168.126:56285: no server TLS configuration available for ClientHello: &{CipherSuites:[2570 4865 4866 4867 49195 49199 49196 49200 52393 52392 49171 49172 156 157 47 53 10] ServerName:www.die-reikiquellen.de SupportedCurves:[35466 29 23 24] SupportedPoints:[0] SignatureSchemes:[1027 2052 1025 1283 2053 1281 2054 1537 513] SupportedProtos:[h2 http/1.1] SupportedVersions:[56026 772 771 770 769] Conn:0xc000652030}
Jan 20 15:06:45 hnrk.io caddy[1432]: 2020/01/20 15:06:45 http: TLS handshake error from 62.157.168.126:37524: no server TLS configuration available for ClientHello: &{CipherSuites:[51914 4865 4866 4867 49195 49199 49196 49200 52393 52392 49171 49172 156 157 47 53 10] ServerName:www.die-reikiquellen.de SupportedCurves:[19018 29 23 24] SupportedPoints:[0] SignatureSchemes:[1027 2052 1025 1283 2053 1281 2054 1537 513] SupportedProtos:[h2 http/1.1] SupportedVersions:[6682 772 771 770 769] Conn:0xc000652038}
Jan 20 15:20:22 hnrk.io caddy[1432]: 2020/01/20 15:20:22 http: TLS handshake error from 66.249.75.140:58356: no server TLS configuration available for ClientHello: &{CipherSuites:[60138 4865 4866 4867 49195 49199 52393 52392 49196 49200 49161 49171 49162 49172 156 157 47 53 10] ServerName:harmoniks.de SupportedCurves:[56026 29 23 24] SupportedPoints:[0] SignatureSchemes:[1027 2052 1025 1283 2053 1281 2054 1537 513] SupportedProtos:[] SupportedVersions:[14906 772 771 770 769] Conn:0xc00054e258}

no server TLS configuration available was output there.

On the other hand, there is only tls here in the caddyv2.json you pasted.

      "tls":{
         "certificates":{
            "load_files":[
               {
                  "certificate":"/etc/caddy/hnrk.io.crt",
                  "key":"/etc/caddy/hnrk.io.key"
               }
            ]
         }

Could it be a solution hint?

1 Like

I happened to add a new domain to Caddyfile and restart Caddy.
Immediately after browsing the web browser, I got ERR_SSL_PROTOCOL_ERROR.
It appeared fine in a few seconds.
Meanwhile, Caddy should have issued a new Let’s Encrypt certificate

Perhaps something went wrong while trying to issue a new or renewed certificate.
For example, you may have reached the issuance limit because you have tried to issue many certificates in the (subdomain).hnrk.io in the past.

https://letsencrypt.org/docs/rate-limits/

@matt ,
I want to check one of the Caddy specifications.
Is it correct that Caddy issues / updates to Let’s Encrypt for each domain / subdomain listed in Caddyfile?

If it is correct, this rate limit will be reached when adding more than 11 new domains / subdomains simultaneously.

https://letsencrypt.org/docs/rate-limits/ :

You can create a maximum of 10 Accounts per IP Address per 3 hours.

To circumvent this rate limit, it is necessary to add a function so that multi-domain certificates (SANs) and wildcard certificates (*.example.com) can be issued.

@HNRK What is the full log output? It looks like you posted a few arbitrary lines but we need the full output. There’s important things missing.

What do you mean “configure markdown”? What do you want to do?

Caddy obtains certificates for each domain and subdomain listed in the Caddyfile, when they do not already exist or when they are expiring soon.

No, because only one ACME account is used for all domains (unless a separate email is used for each domain, which is not a good practice).

1 Like

@matt Sorry, and thank you. I was convinced that this could only be answered by you.

Okay, so first of all, thank you balloon for your time and help!

So let me do this step by step:

I am starting a fresh instance of Caddy v2 beta 13 (the amd64 one attached to the beta 13 release).

I am using the following Caddyfile style configuration:

{
        "experimental_http3": true
}
167.86.123.102:80, [2a02:c207:3004:1207:be:a:bad:babe]:80 {
        redir https://hnrk.io{uri} 301
}
hnrk.io, www.hnrk.io {
        root * /etc/caddy/html
        tls /etc/caddy/hnrk.io.crt /etc/caddy/hnrk.io.key
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server /md* browse
        file_server
}
weather-frogs.de, www.weather-frogs.de {
        tls samplemail@hnrk.io
        root * /etc/caddy/html/weather-frogs.de
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}
die-reikiquellen.de, www.die-reikiquellen.de {
        tls samplemail@hnrk.io
        root * /etc/caddy/html/die-reikiquellen.de
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}
harmoniks.de, www.harmoniks.de {
        tls samplemail@hnrk.io
        root * /etc/caddy/html/harmoniks.de
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}
ip.hnrk.io {
        root * /etc/caddy/html/myip
        tls /etc/caddy/hnrk.io.crt /etc/caddy/hnrk.io.key
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}
sql.hnrk.io {
        root * /etc/caddy/html/pma
        tls /etc/caddy/hnrk.io.crt /etc/caddy/hnrk.io.key
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}
postfix.hnrk.io {
        root * /etc/caddy/html/pfa/public
        tls /etc/caddy/hnrk.io.crt /etc/caddy/hnrk.io.key
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}
draw.hnrk.io {
        root * /etc/caddy/html/draw
        tls /etc/caddy/hnrk.io.crt /etc/caddy/hnrk.io.key
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}
mail.hnrk.io {
        root * /etc/caddy/html/rl
        tls /etc/caddy/hnrk.io.crt /etc/caddy/hnrk.io.key
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server * {
                        hide /data
        }
}
fotos.hnrk.io {
        root * /etc/caddy/html/fotos
        tls /etc/caddy/hnrk.io.crt /etc/caddy/hnrk.io.key
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}
proxy.hnrk.io {
        root * /etc/caddy/html/proxy
        tls /etc/caddy/hnrk.io.crt /etc/caddy/hnrk.io.key
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}
skip.hnrk.io {
        root * /etc/caddy/html/skip
        tls /etc/caddy/hnrk.io.crt /etc/caddy/hnrk.io.key
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}
zeit.hnrk.io {
        root * /etc/caddy/html/time
        tls /etc/caddy/hnrk.io.crt /etc/caddy/hnrk.io.key
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}
frequencies.hnrk.io {
        root * /etc/caddy/html/niviuk.free.fr/niviuk.free.fr
        tls /etc/caddy/hnrk.io.crt /etc/caddy/hnrk.io.key
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}
dns.hnrk.io {
        root * /etc/caddy/html/null
        reverse_proxy * localhost:10001
        tls /etc/caddy/hnrk.io.crt /etc/caddy/hnrk.io.key
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}

Notice the three domains:

These should receive LE Certificates automatically from Caddy v2, because the other domains have Cert + Keyfile already given as path (/etc/caddy/hnrk.io.[crt|key])

Now when I try to navigate to these three domains via my browser, unfortunately I get this error message in Brave:
image

These are the logs that Caddy v2 spits out when navigating to one of them:

Jan 22 19:46:10 hnrk.io systemd[1]: Started Caddy v2.
Jan 22 19:46:10 hnrk.io caddy[11773]: 2020/01/22 18:46:10.638        INFO        using provided configuration        {"config_file": "/etc/v2caddy/Caddyfile", "config_adapter": "caddyfile"}
Jan 22 19:46:10 hnrk.io caddy[11773]: 2020/01/22 18:46:10.657        INFO        admin        admin endpoint started        {"address": "localhost:2019", "enforce_origin": false, "origins": ["localhost:2019"]}
Jan 22 19:46:10 hnrk.io caddy[11773]: 2020/01/22 18:46:10.658        INFO        http        enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
Jan 22 19:46:10 hnrk.io caddy[11773]: 2020/01/22 18:46:10.659        INFO        http        server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server        {"server_name": "srv1", "http_port": 80}
Jan 22 19:46:10 hnrk.io caddy[11773]: 2020/01/22 18:46:10.659        WARN        http        server is listening on same interface as redirects, so automatic HTTP->HTTPS redirects might be overridden by your own configuration        {"server_name": "srv1", "interface": ":80"}
Jan 22 19:46:10 hnrk.io caddy[11773]: 2020/01/22 19:46:10 [INFO][cache:0xc0005bc190] Started certificate maintenance routine
Jan 22 19:46:10 hnrk.io caddy[11773]: 2020/01/22 18:46:10.699        INFO        tls        cleaned up storage units
Jan 22 19:46:10 hnrk.io caddy[11773]: 2020/01/22 18:46:10.699        INFO        http        skipping automatic certificate management because one or more matching certificates are already loaded        {"domain": "draw.hnrk.io", "server_name": "srv0"}
Jan 22 19:46:10 hnrk.io caddy[11773]: 2020/01/22 18:46:10.699        INFO        http        skipping automatic certificate management because one or more matching certificates are already loaded        {"domain": "skip.hnrk.io", "server_name": "srv0"}
Jan 22 19:46:10 hnrk.io caddy[11773]: 2020/01/22 18:46:10.699        INFO        http        skipping automatic certificate management because one or more matching certificates are already loaded        {"domain": "sql.hnrk.io", "server_name": "srv0"}
Jan 22 19:46:10 hnrk.io caddy[11773]: 2020/01/22 18:46:10.699        INFO        http        skipping automatic certificate management because one or more matching certificates are already loaded        {"domain": "frequencies.hnrk.io", "server_name": "srv0"}
Jan 22 19:46:10 hnrk.io caddy[11773]: 2020/01/22 18:46:10.699        INFO        http        skipping automatic certificate management because one or more matching certificates are already loaded        {"domain": "fotos.hnrk.io", "server_name": "srv0"}
Jan 22 19:46:10 hnrk.io caddy[11773]: 2020/01/22 18:46:10.699        INFO        http        skipping automatic certificate management because one or more matching certificates are already loaded        {"domain": "proxy.hnrk.io", "server_name": "srv0"}
Jan 22 19:46:10 hnrk.io caddy[11773]: 2020/01/22 18:46:10.699        INFO        http        skipping automatic certificate management because one or more matching certificates are already loaded        {"domain": "ip.hnrk.io", "server_name": "srv0"}
Jan 22 19:46:10 hnrk.io caddy[11773]: 2020/01/22 18:46:10.699        INFO        http        skipping automatic certificate management because one or more matching certificates are already loaded        {"domain": "zeit.hnrk.io", "server_name": "srv0"}
Jan 22 19:46:10 hnrk.io caddy[11773]: 2020/01/22 18:46:10.699        INFO        http        skipping automatic certificate management because one or more matching certificates are already loaded        {"domain": "postfix.hnrk.io", "server_name": "srv0"}
Jan 22 19:46:10 hnrk.io caddy[11773]: 2020/01/22 18:46:10.699        INFO        http        skipping automatic certificate management because one or more matching certificates are already loaded        {"domain": "mail.hnrk.io", "server_name": "srv0"}
Jan 22 19:46:10 hnrk.io caddy[11773]: 2020/01/22 18:46:10.699        INFO        http        skipping automatic certificate management because one or more matching certificates are already loaded        {"domain": "hnrk.io", "server_name": "srv0"}
Jan 22 19:46:10 hnrk.io caddy[11773]: 2020/01/22 18:46:10.699        INFO        http        skipping automatic certificate management because one or more matching certificates are already loaded        {"domain": "www.hnrk.io", "server_name": "srv0"}
Jan 22 19:46:10 hnrk.io caddy[11773]: 2020/01/22 18:46:10.699        INFO        http        skipping automatic certificate management because one or more matching certificates are already loaded        {"domain": "dns.hnrk.io", "server_name": "srv0"}
Jan 22 19:46:10 hnrk.io caddy[11773]: 2020/01/22 18:46:10.699        INFO        http        enabling automatic TLS certificate management        {"domains": ["www.die-reikiquellen.de", "www.weather-frogs.de", "weather-frogs.de", "www.harmoniks.de", "die-reikiquellen.de", "harmoniks.de"]}
Jan 22 19:46:10 hnrk.io caddy[11773]: 2020/01/22 18:46:10.703        INFO        http        enabling experimental HTTP/3 listener        {"addr": ":443"}
Jan 22 19:46:10 hnrk.io caddy[11773]: 2020/01/22 18:46:10.704        INFO        autosaved config        {"file": "/root/.config/caddy/autosave.json"}
Jan 22 19:46:10 hnrk.io caddy[11773]: 2020/01/22 18:46:10.705        INFO        serving initial configuration
Jan 22 19:46:31 hnrk.io caddy[11773]: 2020/01/22 19:46:31 http: TLS handshake error from 2.202.73.167:17276: no server TLS configuration available for ClientHello: &{CipherSuites:[60138 4865 4866 4867 49195 49199 49196 49200 52393 52392 49171 49172 156 157 47 53 10] ServerName:harmoniks.de SupportedCurves:[2570 29 23 24] SupportedPoints:[0] SignatureSchemes:[1027 2052 1025 1283 2053 1281 2054 1537 513] SupportedProtos:[h2 http/1.1] SupportedVersions:[23130 772 771 770 769] Conn:0xc00028c0f8}
Jan 22 19:46:31 hnrk.io caddy[11773]: 2020/01/22 19:46:31 http: TLS handshake error from 2.202.73.167:17277: no server TLS configuration available for ClientHello: &{CipherSuites:[60138 4865 4866 4867 49195 49199 49196 49200 52393 52392 49171 49172 156 157 47 53 10] ServerName:harmoniks.de SupportedCurves:[47802 29 23 24] SupportedPoints:[0] SignatureSchemes:[1027 2052 1025 1283 2053 1281 2054 1537 513] SupportedProtos:[h2 http/1.1] SupportedVersions:[43690 772 771 770 769] Conn:0xc0000104a0}
Jan 22 19:46:35 hnrk.io caddy[11773]: 2020/01/22 19:46:35 http: TLS handshake error from 2.202.73.167:17279: no server TLS configuration available for ClientHello: &{CipherSuites:[23130 4865 4866 4867 49195 49199 49196 49200 52393 52392 49171 49172 156 157 47 53 10] ServerName:die-reikiquellen.de SupportedCurves:[6682 29 23 24] SupportedPoints:[0] SignatureSchemes:[1027 2052 1025 1283 2053 1281 2054 1537 513] SupportedProtos:[h2 http/1.1] SupportedVersions:[39578 772 771 770 769] Conn:0xc000122220}

So the question is: Did I make a configuration mistake or is this a bug in Caddy v2?
Again, thank you guys for your time! :slight_smile:

@balloon No worries, thanks for your help! It’s great when others get involved.

@HNRK Thanks, that’s much clearer. Have you tried fiddling with anything (e.g. turning of HTTP/3) or removing sites from the config? What is the most minimal steps I would need to perform to reproduce the issue?

You might also try converting the Caddyfile to JSON, then set manage_sync in the automation policy to true: https://caddyserver.com/docs/json/apps/tls/automation/policies/#manage_sync – that should stop your server from continuing until it has something to say about certificates.

Let me know how it goes!

1 Like

Hmm, I think your use case touches on a TODO in certmagic, let me see what I can do about it. In the meantime, any debugging updates from your end would be helpful! Especially the manage_sync stuff.

Edit: @HNRK would you be able to build Caddy beta 13 against this commit of CertMagic: https://github.com/mholt/certmagic/commit/7f077781d2a26bd5a6528fd877848d86b552470a

Hi @matt, I was able to track down the issue. (I didn’t yet consider checking out the certmagic commit. Will do shortly.

I tried setting up Caddy v2 with the following config (one single domain that receives LE certificate by Caddy v2)

harmoniks.de, www.harmoniks.de {
       tls henrik@hnrk.io
       root * /etc/caddy/html/harmoniks.de
       encode brotli zstd gzip
       php_fastcgi unix//run/php/php7.3-fpm.sock
       try_files {path} /index.php?{query}
       file_server
}

This works perfectly fine.

Having all three (automatically generated LE certified) domains in the config also works fine:

harmoniks.de, www.harmoniks.de {
        tls henrik@hnrk.io
        root * /etc/caddy/html/harmoniks.de
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}
weather-frogs.de, www.weather-frogs.de {
        tls henrik@hnrk.io
        root * /etc/caddy/html/weather-frogs.de
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}
die-reikiquellen.de, www.die-reikiquellen.de {
        tls henrik@hnrk.io
        root * /etc/caddy/html/die-reikiquellen.de
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}

Now if I add my main domain (hnrk.io) on top of the other three domains, that has cert and key files in a folder, Caddy v2 stops serving the other three domains and returns a TLS Error every second time I visit the page, however hnrk.io is served fine without issues.

hnrk.io, www.hnrk.io {
        root * /etc/caddy/html
        tls /etc/caddy/hnrk.io.crt /etc/caddy/hnrk.io.key
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server /md* browse
        file_server
}
harmoniks.de, www.harmoniks.de {
        tls henrik@hnrk.io
        root * /etc/caddy/html/harmoniks.de
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}
weather-frogs.de, www.weather-frogs.de {
        tls henrik@hnrk.io
        root * /etc/caddy/html/weather-frogs.de
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}
die-reikiquellen.de, www.die-reikiquellen.de {
        tls henrik@hnrk.io
        root * /etc/caddy/html/die-reikiquellen.de
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}

Caddy v2 behaves like this: When visiting my main domain hnrk.io with the config above, it loads fine everytime, but when visiting my other three domains, the site loads successfully every second time. Little bit weird, but I tried to capture this behaviour here: Harmoniks.de for example loads successfully every second time:

The entry for tls looks strange:
https://caddyserver.com/docs/caddyfile/options

@balloon I cannot see any tls at your link, do you mean https://caddyserver.com/docs/caddyfile/directives/tls?

Thanks for the details – can you also help by posting the full log output? From the very moment you start the process all the way until it is closed…

Also, are you using the API / caddy reload for config changes, or are you stopping and restarting the process entirely?

Log output from my last mentioned config setting where the LE domains fail (hnrk.io + three automatically LE certified domains):

Jan 23 16:17:32 hnrk.io systemd[1]: Stopped Caddy v2.
Jan 23 16:18:41 hnrk.io systemd[1]: Started Caddy v2.
Jan 23 16:18:41 hnrk.io caddy[16928]: 2020/01/23 15:18:41.128        INFO        using provided configuration        {"config_file": "/etc/v2caddy/Caddyfile", "config_adapter": "caddyfile"}
Jan 23 16:18:41 hnrk.io caddy[16928]: 2020/01/23 15:18:41.138        INFO        admin        admin endpoint started        {"address": "localhost:2019", "enforce_origin": false, "origins": ["localhost:2019"]}
Jan 23 16:18:41 hnrk.io caddy[16928]: 2020/01/23 15:18:41.138        INFO        http        enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
Jan 23 16:18:41 hnrk.io caddy[16928]: 2020/01/23 16:18:41 [INFO][cache:0xc0004cccd0] Started certificate maintenance routine
Jan 23 16:18:41 hnrk.io caddy[16928]: 2020/01/23 15:18:41.156        INFO        http        skipping automatic certificate management because one or more matching certificates are already loaded        {"domain": "hnrk.io", "server_name": "srv0"}
Jan 23 16:18:41 hnrk.io caddy[16928]: 2020/01/23 15:18:41.156        INFO        http        skipping automatic certificate management because one or more matching certificates are already loaded        {"domain": "www.hnrk.io", "server_name": "srv0"}
Jan 23 16:18:41 hnrk.io caddy[16928]: 2020/01/23 15:18:41.156        INFO        http        enabling automatic TLS certificate management        {"domains": ["www.weather-frogs.de", "harmoniks.de", "www.harmoniks.de", "die-reikiquellen.de", "www.die-reikiquellen.de", "weather-frogs.de"]}
Jan 23 16:18:41 hnrk.io caddy[16928]: 2020/01/23 15:18:41.177        INFO        tls        cleaned up storage units
Jan 23 16:18:41 hnrk.io caddy[16928]: 2020/01/23 15:18:41.177        INFO        autosaved config        {"file": "/root/.config/caddy/autosave.json"}
Jan 23 16:18:41 hnrk.io caddy[16928]: 2020/01/23 15:18:41.177        INFO        serving initial configuration
Jan 23 16:18:53 hnrk.io caddy[16928]: 2020/01/23 16:18:53 http: TLS handshake error from 2.202.73.167:18209: no server TLS configuration available for ClientHello: &{CipherSuites:[19018 4865 4866 4867 49195 49199 49196 49200 52393 52392 49171 49172 156 157 47 53 10] ServerName:harmoniks.de SupportedCurves:[51914 29 23 24] SupportedPoints:[0] SignatureSchemes:[1027 2052 1025 1283 2053 1281 2054 1537 513] SupportedProtos:[h2 http/1.1] SupportedVersions:[56026 772 771 770 769] Conn:0xc0008850a0}
Jan 23 16:18:54 hnrk.io caddy[16928]: 2020/01/23 16:18:54 http: TLS handshake error from 2.202.73.167:18210: no server TLS configuration available for ClientHello: &{CipherSuites:[31354 4865 4866 4867 49195 49199 49196 49200 52393 52392 49171 49172 156 157 47 53 10] ServerName:harmoniks.de SupportedCurves:[35466 29 23 24] SupportedPoints:[0] SignatureSchemes:[1027 2052 1025 1283 2053 1281 2054 1537 513] SupportedProtos:[h2 http/1.1] SupportedVersions:[43690 772 771 770 769] Conn:0xc00059a0a0}
Jan 23 16:18:54 hnrk.io caddy[16928]: 2020/01/23 16:18:54 http: TLS handshake error from 2.202.73.167:18211: no server TLS configuration available for ClientHello: &{CipherSuites:[6682 4865 4866 4867 49195 49199 49196 49200 52393 52392 49171 49172 156 157 47 53 10] ServerName:harmoniks.de SupportedCurves:[6682 29 23 24] SupportedPoints:[0] SignatureSchemes:[1027 2052 1025 1283 2053 1281 2054 1537 513] SupportedProtos:[h2 http/1.1] SupportedVersions:[47802 772 771 770 769] Conn:0xc00059a0a8}
Jan 23 16:18:55 hnrk.io caddy[16928]: 2020/01/23 16:18:55 http: TLS handshake error from 2.202.73.167:18212: no server TLS configuration available for ClientHello: &{CipherSuites:[14906 4865 4866 4867 49195 49199 49196 49200 52393 52392 49171 49172 156 157 47 53 10] ServerName:harmoniks.de SupportedCurves:[2570 29 23 24] SupportedPoints:[0] SignatureSchemes:[1027 2052 1025 1283 2053 1281 2054 1537 513] SupportedProtos:[h2 http/1.1] SupportedVersions:[14906 772 771 770 769] Conn:0xc0008850a8}
Jan 23 16:18:55 hnrk.io caddy[16928]: 2020/01/23 16:18:55 http: TLS handshake error from 2.202.73.167:18214: no server TLS configuration available for ClientHello: &{CipherSuites:[60138 4865 4866 4867 49195 49199 49196 49200 52393 52392 49171 49172 156 157 47 53 10] ServerName:harmoniks.de SupportedCurves:[10794 29 23 24] SupportedPoints:[0] SignatureSchemes:[1027 2052 1025 1283 2053 1281 2054 1537 513] SupportedProtos:[h2 http/1.1] SupportedVersions:[2570 772 771 770 769] Conn:0xc0008850b0}
Jan 23 16:18:55 hnrk.io caddy[16928]: 2020/01/23 16:18:55 http: TLS handshake error from 2.202.73.167:18213: no server TLS configuration available for ClientHello: &{CipherSuites:[39578 4865 4866 4867 49195 49199 49196 49200 52393 52392 49171 49172 156 157 47 53 10] ServerName:harmoniks.de SupportedCurves:[31354 29 23 24] SupportedPoints:[0] SignatureSchemes:[1027 2052 1025 1283 2053 1281 2054 1537 513] SupportedProtos:[h2 http/1.1] SupportedVersions:[10794 772 771 770 769] Conn:0xc00059a1e8}
Jan 23 16:18:55 hnrk.io caddy[16928]: 2020/01/23 16:18:55 http: TLS handshake error from 2.202.73.167:18216: no server TLS configuration available for ClientHello: &{CipherSuites:[35466 4865 4866 4867 49195 49199 49196 49200 52393 52392 49171 49172 156 157 47 53 10] ServerName:harmoniks.de SupportedCurves:[43690 29 23 24] SupportedPoints:[0] SignatureSchemes:[1027 2052 1025 1283 2053 1281 2054 1537 513] SupportedProtos:[h2 http/1.1] SupportedVersions:[14906 772 771 770 769] Conn:0xc0008850b8}

I completely stop and start the process via systemd file.

1 Like

Thanks; do you have anything longer than 10 seconds worth of logs? At least a minute, maybe?

For troubleshooting, I think it’ll be helpful to eliminate as many variables as possible. Can you please try just running it on the command line directly without systemd?

EDIT: Yeah, sure: https://gist.github.com/Henrocker/172bc21eeb78dbc30785cf05fe9ce547

1 Like

Thanks! I’ll try to look into this more. Might be a while though as I recover from being sick :mask: