Acme handshake failing

1. The problem I’m having:

I am trying to set up Caddy to provide reverse proxy and HTTPS for Foundry VTT. Specifically I am following this guide. Going to vtt.masterflinter.net times out (as does curling it).

I have my router forwarding ports 443 and 80 to my home-server. I believe this is working because if I set up port forwarding to the foundry port (30000), turn off Caddy, and change the Foundry config file to remove the proxy info I can hit foundry using my ip:port. I could be wrong though since Caddy is having trouble.

If my network setup is relevant then it is Internet → ISP Modem/Router in Bridge mode → Router → home-server.

I have a porkbun domain name that I created an A-record for my subdomain (vtt.masterflinter.net). I also believe I did this correctly because if you curl vtt.masterflinter.net the correct ip shoes up.

The acme challenge is failing. If I copy the link from the caddy output and paste it in a browser I see what looks like json contents.

In the course of troubleshooting I have tried allowing caddy to bind ports 80 and 443, deleting the contents of /var/lib/caddy/.local/share/caddy/, and verifying that my firewall rules are not blocking 80/443 based on other posts in caddy.community.

2. Error messages and/or full log output:

flint@flint-home-server:/etc/caddy$ caddy run
2023/10/08 19:37:55.090 INFO    using adjacent Caddyfile
2023/10/08 19:37:55.091 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2023/10/08 19:37:55.091 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc000372600"}
2023/10/08 19:37:55.091 INFO    http.auto_https enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2023/10/08 19:37:55.091 DEBUG   http.auto_https adjusted config {"tls": {"automation":{"policies":[{"subjects":["vtt.masterflinter.net"]},{"subjects":["10.0.0.23"]},{}]}}, "http": {"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"encodings":{"gzip":{},"zstd":{}},"handler":"encode","prefer":["zstd","gzip"]},{"handler":"reverse_proxy","upstreams":[{"dial":"localhost:30000"}]}]}]}],"terminal":true},{"handle":[{"handler":"subroute","routes":[{"handle":[{"encodings":{"gzip":{},"zstd":{}},"handler":"encode","prefer":["zstd","gzip"]},{"handler":"reverse_proxy","upstreams":[{"dial":"localhost:30000"}]}]}]}],"terminal":true}],"tls_connection_policies":[{"match":{"sni":["10.0.0.23",""]},"default_sni":"10.0.0.23"},{"default_sni":"10.0.0.23"}],"automatic_https":{}}}}}
2023/10/08 19:37:55.097 INFO    pki.ca.local    root certificate is already trusted by system   {"path": "storage:pki/authorities/local/root.crt"}
2023/10/08 19:37:55.097 INFO    http    enabling HTTP/3 listener        {"addr": ":443"}
2023/10/08 19:37:55.097 INFO    tls     cleaning storage unit   {"description": "FileStorage:/home/flint/.local/share/caddy"}
2023/10/08 19:37:55.097 INFO    failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
2023/10/08 19:37:55.098 DEBUG   http    starting server loop    {"address": "[::]:443", "tls": true, "http3": true}
2023/10/08 19:37:55.098 INFO    http.log        server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2023/10/08 19:37:55.098 DEBUG   http    starting server loop    {"address": "[::]:80", "tls": false, "http3": false}
2023/10/08 19:37:55.098 INFO    http.log        server running  {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2023/10/08 19:37:55.098 INFO    http    enabling automatic TLS certificate management   {"domains": ["10.0.0.23", "vtt.masterflinter.net"]}
2023/10/08 19:37:55.098 INFO    tls     finished cleaning storage units
2023/10/08 19:37:55.099 WARN    tls     stapling OCSP   {"error": "no OCSP stapling for [10.0.0.23]: no OCSP server specified in certificate", "identifiers": ["10.0.0.23"]}
2023/10/08 19:37:55.099 DEBUG   tls.cache       added certificate to cache      {"subjects": ["10.0.0.23"], "expiration": "2023/10/09 02:21:04.000", "managed": true, "issuer_key": "local", "hash": "fbb3789a5940cf1eb8f6fbf47640ef9b8dcc4e23c8e7d179c37b02c17626c7be", "cache_size": 1, "cache_capacity": 10000}
2023/10/08 19:37:55.099 DEBUG   events  event   {"name": "cached_managed_cert", "id": "efc810d6-42c8-4695-9015-8cf3122c69b6", "origin": "tls", "data": {"sans":["10.0.0.23"]}}
2023/10/08 19:37:55.099 INFO    autosaved config (load with --resume flag)      {"file": "/home/flint/.config/caddy/autosave.json"}
2023/10/08 19:37:55.099 INFO    serving initial configuration
2023/10/08 19:37:55.099 INFO    tls.obtain      acquiring lock  {"identifier": "vtt.masterflinter.net"}
2023/10/08 19:37:55.100 INFO    tls.obtain      lock acquired   {"identifier": "vtt.masterflinter.net"}
2023/10/08 19:37:55.100 INFO    tls.obtain      obtaining certificate   {"identifier": "vtt.masterflinter.net"}
2023/10/08 19:37:55.100 DEBUG   events  event   {"name": "cert_obtaining", "id": "72bea34c-35e5-4f83-9c49-dde66f60c487", "origin": "tls", "data": {"identifier":"vtt.masterflinter.net"}}
2023/10/08 19:37:55.100 DEBUG   tls.obtain      trying issuer 1/2       {"issuer": "acme-v02.api.letsencrypt.org-directory"}
2023/10/08 19:37:55.269 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "GET", "url": "https://acme-v02.api.letsencrypt.org/directory", "headers": {"User-Agent":["Caddy/2.7.4 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["752"],"Content-Type":["application/json"],"Date":["Sun, 08 Oct 2023 19:37:55 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2023/10/08 19:37:55.313 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "HEAD", "url": "https://acme-v02.api.letsencrypt.org/acme/new-nonce", "headers": {"User-Agent":["Caddy/2.7.4 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Date":["Sun, 08 Oct 2023 19:37:55 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["_s_5u1NQ_1R6zAjpagMeeY3lBbwMZsnNUVEqXlYN-B_cNSEVLAE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2023/10/08 19:37:55.376 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/new-acct", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.4 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["1350494416"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["335"],"Content-Type":["application/json"],"Date":["Sun, 08 Oct 2023 19:37:55 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf>;rel=\"terms-of-service\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/acct/1350494416"],"Replay-Nonce":["IAvmRRVVRKsAZIfMhwDXE3o1Hc0s_B9fyv7lQzyR9la8dhmnPvQ"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 201}
2023/10/08 19:37:55.377 INFO    tls.issuance.acme       waiting on internal rate limiter        {"identifiers": ["vtt.masterflinter.net"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "cdflint.caddy@masterflinter.net"}
2023/10/08 19:37:55.377 INFO    tls.issuance.acme       done waiting on internal rate limiter   {"identifiers": ["vtt.masterflinter.net"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "cdflint.caddy@masterflinter.net"}
2023/10/08 19:37:55.448 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/new-order", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.4 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["1350494416"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["347"],"Content-Type":["application/json"],"Date":["Sun, 08 Oct 2023 19:37:55 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/1350494416/213739532386"],"Replay-Nonce":["_s_5u1NQ2FbSV5pkgU4XwPjNczQUp55g85xId4OlYWnRcfL3XQk"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 201}
2023/10/08 19:37:55.495 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz-v3/271985688956", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.4 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["1350494416"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["805"],"Content-Type":["application/json"],"Date":["Sun, 08 Oct 2023 19:37:55 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["IAvmRRVVa2fqHATSlZJ8VVvqJo6gHNJLR2ytoFm1l7ebzm7rSS0"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2023/10/08 19:37:55.495 DEBUG   tls.issuance.acme.acme_client   no solver configured    {"challenge_type": "dns-01"}
2023/10/08 19:37:55.495 INFO    tls.issuance.acme.acme_client   trying to solve challenge       {"identifier": "vtt.masterflinter.net", "challenge_type": "tls-alpn-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2023/10/08 19:37:55.496 DEBUG   tls.issuance.acme.acme_client   waiting for solver before continuing    {"identifier": "vtt.masterflinter.net", "challenge_type": "tls-alpn-01"}
2023/10/08 19:37:55.496 DEBUG   tls.issuance.acme.acme_client   done waiting for solver {"identifier": "vtt.masterflinter.net", "challenge_type": "tls-alpn-01"}
2023/10/08 19:37:55.496 DEBUG   http.stdlib     http: TLS handshake error from 127.0.0.1:43406: EOF
2023/10/08 19:37:55.547 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/271985688956/Xw1Sgw", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.4 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["1350494416"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["191"],"Content-Type":["application/json"],"Date":["Sun, 08 Oct 2023 19:37:55 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/authz-v3/271985688956>;rel=\"up\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/chall-v3/271985688956/Xw1Sgw"],"Replay-Nonce":["IAvmRRVVDTQd5cDy67to-X5LnsrzaNAf2cq1-KuSTtdWQqvj84M"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2023/10/08 19:37:55.547 DEBUG   tls.issuance.acme.acme_client   challenge accepted      {"identifier": "vtt.masterflinter.net", "challenge_type": "tls-alpn-01"}
2023/10/08 19:37:55.845 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz-v3/271985688956", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.4 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["1350494416"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["805"],"Content-Type":["application/json"],"Date":["Sun, 08 Oct 2023 19:37:55 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["IAvmRRVVDbRF_4zwW9qdIDS61vj-09Hi_0yZTbJakeim5KjA6Tg"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2023/10/08 19:37:56.142 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz-v3/271985688956", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.4 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["1350494416"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["805"],"Content-Type":["application/json"],"Date":["Sun, 08 Oct 2023 19:37:56 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["IAvmRRVVl5y2PpFTyZ6r1iabtv8jAEhwxwCfUEDzUqgLtvHxpSc"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2023/10/08 19:37:56.438 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz-v3/271985688956", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.4 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["1350494416"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["805"],"Content-Type":["application/json"],"Date":["Sun, 08 Oct 2023 19:37:56 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["_s_5u1NQDaNJNIQtSMwbQbHtgEfvwJlxfnhv-H2_aMldipjF6zw"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2023/10/08 19:37:56.735 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz-v3/271985688956", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.4 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["1350494416"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["805"],"Content-Type":["application/json"],"Date":["Sun, 08 Oct 2023 19:37:56 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["IAvmRRVVYv3PGY1PsT3x4BHnToNwBP_dQ_MFRUODGloYBYQNSE8"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2023/10/08 19:37:57.031 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz-v3/271985688956", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.4 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["1350494416"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["805"],"Content-Type":["application/json"],"Date":["Sun, 08 Oct 2023 19:37:57 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["IAvmRRVVX0nKaM-bogsTBw4gwf-zLpGzFieZt8wd-WOD4B9oWAQ"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2023/10/08 19:37:57.327 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz-v3/271985688956", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.4 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["1350494416"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["805"],"Content-Type":["application/json"],"Date":["Sun, 08 Oct 2023 19:37:57 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["IAvmRRVVfLIyqKjNeoR5zaTvJG_7X4cbNCFy0KpFwLt3LRqzU8g"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2023/10/08 19:37:57.623 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz-v3/271985688956", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.4 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["1350494416"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["805"],"Content-Type":["application/json"],"Date":["Sun, 08 Oct 2023 19:37:57 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["_s_5u1NQo1hXDY-cz_rV2kdE1TK7b3YYtNcsaOD1_UpbBsdZLV4"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2023/10/08 19:37:57.921 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz-v3/271985688956", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.4 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["1350494416"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["805"],"Content-Type":["application/json"],"Date":["Sun, 08 Oct 2023 19:37:57 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["_s_5u1NQBFTo51HdcmUk4ENYQHt04T-m0NxSs6KQRrVg3yqEryE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2023/10/08 19:37:58.217 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz-v3/271985688956", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.4 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["1350494416"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["805"],"Content-Type":["application/json"],"Date":["Sun, 08 Oct 2023 19:37:58 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["_s_5u1NQqrBtON8ABA8kfoIQ1ujukSSAfEo_5jAAqe_GvNmudMw"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}

3. Caddy version:

v2.7.4 h1:J8nisjdOxnYHXlorUKXY75Gr6iBfudfoGhrJ8t7/flI=

4. How I installed and ran Caddy:

  1. add Caddy repository
  2. install Caddy using apt
  3. apt update+upgrade
  4. Edit Caddyfile in /etc/caddy/

a. System environment:

Running Caddy on Ubuntu Server 22.04

b. Command:

caddy run --config /etc/caddy/Caddyfile

or 

caddy run (if I am already in /etc/caddy)

c. Service/unit/compose file:

d. My complete Caddy config:

# This replaces the existing content in /etc/caddy/Caddyfile

# A CONFIG SECTION FOR YOUR IP AND HOSTNAME
{
        default_sni 10.0.0.23
        debug
        email cdflint.caddy@masterflinter.net
}

10.0.0.23 {
        # PROXY ALL REQUEST TO PORT 30000
        tls internal
        reverse_proxy localhost:30000
        encode zstd gzip
}

vtt.masterflinter.net {
        # PROXY ALL REQUEST TO PORT 30000
        reverse_proxy localhost:30000
        encode zstd gzip
}
# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile

5. Links to relevant resources:

flint@flint-home-server:/etc/caddy$ caddy run --config /etc/caddy/Caddyfile
2023/10/08 19:21:27.351 INFO    using provided configuration    {"config_file": "/etc/caddy/Caddyfile", "config_adapter": ""}
2023/10/08 19:21:27.352 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2023/10/08 19:21:27.352 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc0004dd580"}
2023/10/08 19:21:27.352 INFO    http.auto_https enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2023/10/08 19:21:27.352 INFO    http    enabling HTTP/3 listener        {"addr": ":443"}
2023/10/08 19:21:27.353 INFO    failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
2023/10/08 19:21:27.353 INFO    http.log        server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2023/10/08 19:21:27.353 INFO    http.log        server running  {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2023/10/08 19:21:27.353 INFO    http    enabling automatic TLS certificate management   {"domains": ["vtt.masterflinter.net", "10.0.0.23"]}
2023/10/08 19:21:27.353 WARN    tls     stapling OCSP   {"error": "no OCSP stapling for [10.0.0.23]: no OCSP server specified in certificate", "identifiers": ["10.0.0.23"]}
2023/10/08 19:21:27.353 INFO    tls.obtain      acquiring lock  {"identifier": "vtt.masterflinter.net"}
2023/10/08 19:21:27.354 INFO    tls.obtain      lock acquired   {"identifier": "vtt.masterflinter.net"}
2023/10/08 19:21:27.354 INFO    tls.obtain      obtaining certificate   {"identifier": "vtt.masterflinter.net"}
2023/10/08 19:21:27.354 INFO    tls     waiting on internal rate limiter        {"identifiers": ["vtt.masterflinter.net"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2023/10/08 19:21:27.354 INFO    tls     done waiting on internal rate limiter   {"identifiers": ["vtt.masterflinter.net"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2023/10/08 19:21:27.360 INFO    pki.ca.local    root certificate is already trusted by system   {"path": "storage:pki/authorities/local/root.crt"}
2023/10/08 19:21:27.360 INFO    tls     cleaning storage unit   {"description": "FileStorage:/home/flint/.local/share/caddy"}
2023/10/08 19:21:27.360 INFO    autosaved config (load with --resume flag)      {"file": "/home/flint/.config/caddy/autosave.json"}
2023/10/08 19:21:27.360 INFO    serving initial configuration
2023/10/08 19:21:27.360 INFO    tls     finished cleaning storage units
2023/10/08 19:21:27.684 INFO    tls.acme_client trying to solve challenge       {"identifier": "vtt.masterflinter.net", "challenge_type": "http-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2023/10/08 19:21:37.932 ERROR   tls.acme_client challenge failed        {"identifier": "vtt.masterflinter.net", "challenge_type": "http-01", "problem": {"type": "urn:ietf:params:acme:error:connection", "title": "", "detail": "162.207.102.104: Fetching http://vtt.masterflinter.net/.well-known/acme-challenge/fFNtVFoan7-l-MIoycCgRtgnosau5y-3vTSV__u9Y2c: Timeout during connect (likely firewall problem)", "instance": "", "subproblems": []}}
2023/10/08 19:21:37.932 ERROR   tls.acme_client validating authorization        {"identifier": "vtt.masterflinter.net", "problem": {"type": "urn:ietf:params:acme:error:connection", "title": "", "detail": "162.207.102.104: Fetching http://vtt.masterflinter.net/.well-known/acme-challenge/fFNtVFoan7-l-MIoycCgRtgnosau5y-3vTSV__u9Y2c: Timeout during connect (likely firewall problem)", "instance": "", "subproblems": []}, "order": "https://acme-v02.api.letsencrypt.org/acme/order/1350155356/213737215266", "attempt": 1, "max_attempts": 3}
2023/10/08 19:21:38.990 ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "vtt.masterflinter.net", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/failed-validation-limit/"}
2023/10/08 19:21:38.991 INFO    tls     waiting on internal rate limiter        {"identifiers": ["vtt.masterflinter.net"], "ca": "https://acme.zerossl.com/v2/DV90", "account": "caddy@zerossl.com"}
2023/10/08 19:21:38.991 INFO    tls     done waiting on internal rate limiter   {"identifiers": ["vtt.masterflinter.net"], "ca": "https://acme.zerossl.com/v2/DV90", "account": "caddy@zerossl.com"}
2023/10/08 19:21:40.287 INFO    tls.acme_client trying to solve challenge       {"identifier": "vtt.masterflinter.net", "challenge_type": "http-01", "ca": "https://acme.zerossl.com/v2/DV90"}
2023/10/08 19:21:52.297 ERROR   tls.acme_client challenge failed        {"identifier": "vtt.masterflinter.net", "challenge_type": "http-01", "problem": {"type": "", "title": "", "detail": "", "instance": "", "subproblems": []}}
2023/10/08 19:21:52.298 ERROR   tls.acme_client validating authorization        {"identifier": "vtt.masterflinter.net", "problem": {"type": "", "title": "", "detail": "", "instance": "", "subproblems": []}, "order": "https://acme.zerossl.com/v2/DV90/order/X_tbddXK20D3U-DmuzabiQ", "attempt": 1, "max_attempts": 3}
2023/10/08 19:21:52.298 ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "vtt.masterflinter.net", "issuer": "acme.zerossl.com-v2-DV90", "error": "HTTP 0  - "}
2023/10/08 19:21:52.298 ERROR   tls.obtain      will retry      {"error": "[vtt.masterflinter.net] Obtain: [vtt.masterflinter.net] solving challenge: vtt.masterflinter.net: [vtt.masterflinter.net] authorization failed: HTTP 0  -  (ca=https://acme.zerossl.com/v2/DV90)", "attempt": 1, "retrying_in": 60, "elapsed": 24.943678448, "max_duration": 2592000}
2023/10/08 19:22:52.301 INFO    tls.obtain      obtaining certificate   {"identifier": "vtt.masterflinter.net"}
2023/10/08 19:22:52.729 INFO    tls.acme_client trying to solve challenge       {"identifier": "vtt.masterflinter.net", "challenge_type": "tls-alpn-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2023/10/08 19:23:03.274 ERROR   tls.acme_client challenge failed        {"identifier": "vtt.masterflinter.net", "challenge_type": "tls-alpn-01", "problem": {"type": "urn:ietf:params:acme:error:connection", "title": "", "detail": "162.207.102.104: Timeout during connect (likely firewall problem)", "instance": "", "subproblems": []}}
2023/10/08 19:23:03.274 ERROR   tls.acme_client validating authorization        {"identifier": "vtt.masterflinter.net", "problem": {"type": "urn:ietf:params:acme:error:connection", "title": "", "detail": "162.207.102.104: Timeout during connect (likely firewall problem)", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/121166254/11453412124", "attempt": 1, "max_attempts": 3}
2023/10/08 19:23:04.408 INFO    tls.acme_client trying to solve challenge       {"identifier": "vtt.masterflinter.net", "challenge_type": "http-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2023/10/08 19:23:14.650 ERROR   tls.acme_client challenge failed        {"identifier": "vtt.masterflinter.net", "challenge_type": "http-01", "problem": {"type": "urn:ietf:params:acme:error:connection", "title": "", "detail": "162.207.102.104: Fetching http://vtt.masterflinter.net/.well-known/acme-challenge/xVklkbh4Kde1GZ48sQbLRFyEeG17wvPqnuBWgTua-gM: Timeout during connect (likely firewall problem)", "instance": "", "subproblems": []}}
2023/10/08 19:23:14.650 ERROR   tls.acme_client validating authorization        {"identifier": "vtt.masterflinter.net", "problem": {"type": "urn:ietf:params:acme:error:connection", "title": "", "detail": "162.207.102.104: Fetching http://vtt.masterflinter.net/.well-known/acme-challenge/xVklkbh4Kde1GZ48sQbLRFyEeG17wvPqnuBWgTua-gM: Timeout during connect (likely firewall problem)", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/121166254/11453413924", "attempt": 2, "max_attempts": 3}
2023/10/08 19:23:14.650 ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "vtt.masterflinter.net", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 400 urn:ietf:params:acme:error:connection - 162.207.102.104: Fetching http://vtt.masterflinter.net/.well-known/acme-challenge/xVklkbh4Kde1GZ48sQbLRFyEeG17wvPqnuBWgTua-gM: Timeout during connect (likely firewall problem)"}
2023/10/08 19:23:16.318 INFO    tls.acme_client trying to solve challenge       {"identifier": "vtt.masterflinter.net", "challenge_type": "http-01", "ca": "https://acme.zerossl.com/v2/DV90"}
2023/10/08 19:23:28.155 ERROR   tls.acme_client challenge failed        {"identifier": "vtt.masterflinter.net", "challenge_type": "http-01", "problem": {"type": "", "title": "", "detail": "", "instance": "", "subproblems": []}}
2023/10/08 19:23:28.155 ERROR   tls.acme_client validating authorization        {"identifier": "vtt.masterflinter.net", "problem": {"type": "", "title": "", "detail": "", "instance": "", "subproblems": []}, "order": "https://acme.zerossl.com/v2/DV90/order/fXKrQY8gv2wVjjsVTz-y7g", "attempt": 1, "max_attempts": 3}
2023/10/08 19:23:28.155 ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "vtt.masterflinter.net", "issuer": "acme.zerossl.com-v2-DV90", "error": "HTTP 0  - "}
2023/10/08 19:23:28.155 ERROR   tls.obtain      will retry      {"error": "[vtt.masterflinter.net] Obtain: [vtt.masterflinter.net] solving challenge: vtt.masterflinter.net: [vtt.masterflinter.net] authorization failed: HTTP 0  -  (ca=https://acme.zerossl.com/v2/DV90)", "attempt": 2, "retrying_in": 120, "elapsed": 120.801572335, "max_duration": 2592000}
2023/10/08 19:25:28.156 INFO    tls.obtain      obtaining certificate   {"identifier": "vtt.masterflinter.net"}
2023/10/08 19:25:28.335 INFO    tls.acme_client trying to solve challenge       {"identifier": "vtt.masterflinter.net", "challenge_type": "tls-alpn-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2023/10/08 19:25:38.609 ERROR   tls.acme_client challenge failed        {"identifier": "vtt.masterflinter.net", "challenge_type": "tls-alpn-01", "problem": {"type": "urn:ietf:params:acme:error:connection", "title": "", "detail": "162.207.102.104: Timeout during connect (likely firewall problem)", "instance": "", "subproblems": []}}
2023/10/08 19:25:38.609 ERROR   tls.acme_client validating authorization        {"identifier": "vtt.masterflinter.net", "problem": {"type": "urn:ietf:params:acme:error:connection", "title": "", "detail": "162.207.102.104: Timeout during connect (likely firewall problem)", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/121166254/11453438254", "attempt": 1, "max_attempts": 3}
2023/10/08 19:25:39.734 INFO    tls.acme_client trying to solve challenge       {"identifier": "vtt.masterflinter.net", "challenge_type": "http-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2023/10/08 19:25:49.961 ERROR   tls.acme_client challenge failed        {"identifier": "vtt.masterflinter.net", "challenge_type": "http-01", "problem": {"type": "urn:ietf:params:acme:error:connection", "title": "", "detail": "162.207.102.104: Fetching http://vtt.masterflinter.net/.well-known/acme-challenge/ZsN5Z-bRkQ46yGuRCUY4kFRImGVTOsgSejHwJZWqwt0: Timeout during connect (likely firewall problem)", "instance": "", "subproblems": []}}
2023/10/08 19:25:49.961 ERROR   tls.acme_client validating authorization        {"identifier": "vtt.masterflinter.net", "problem": {"type": "urn:ietf:params:acme:error:connection", "title": "", "detail": "162.207.102.104: Fetching http://vtt.masterflinter.net/.well-known/acme-challenge/ZsN5Z-bRkQ46yGuRCUY4kFRImGVTOsgSejHwJZWqwt0: Timeout during connect (likely firewall problem)", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/121166254/11453439644", "attempt": 2, "max_attempts": 3}
2023/10/08 19:25:49.961 ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "vtt.masterflinter.net", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 400 urn:ietf:params:acme:error:connection - 162.207.102.104: Fetching http://vtt.masterflinter.net/.well-known/acme-challenge/ZsN5Z-bRkQ46yGuRCUY4kFRImGVTOsgSejHwJZWqwt0: Timeout during connect (likely firewall problem)"}
2023/10/08 19:25:51.311 INFO    tls.acme_client trying to solve challenge       {"identifier": "vtt.masterflinter.net", "challenge_type": "http-01", "ca": "https://acme.zerossl.com/v2/DV90"}
2023/10/08 19:26:03.792 ERROR   tls.acme_client challenge failed        {"identifier": "vtt.masterflinter.net", "challenge_type": "http-01", "problem": {"type": "", "title": "", "detail": "", "instance": "", "subproblems": []}}
2023/10/08 19:26:03.792 ERROR   tls.acme_client validating authorization        {"identifier": "vtt.masterflinter.net", "problem": {"type": "", "title": "", "detail": "", "instance": "", "subproblems": []}, "order": "https://acme.zerossl.com/v2/DV90/order/s0WTsxTj7wAXuiPg59CTWQ", "attempt": 1, "max_attempts": 3}
2023/10/08 19:26:03.792 ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "vtt.masterflinter.net", "issuer": "acme.zerossl.com-v2-DV90", "error": "HTTP 0  - "}
2023/10/08 19:26:03.792 ERROR   tls.obtain      will retry      {"error": "[vtt.masterflinter.net] Obtain: [vtt.masterflinter.net] solving challenge: vtt.masterflinter.net: [vtt.masterflinter.net] authorization failed: HTTP 0  -  (ca=https://acme.zerossl.com/v2/DV90)", "attempt": 3, "retrying_in": 120, "elapsed": 276.437968256, "max_duration": 2592000}
2023/10/08 19:28:03.793 INFO    tls.obtain      obtaining certificate   {"identifier": "vtt.masterflinter.net"}
2023/10/08 19:28:03.972 INFO    tls.acme_client trying to solve challenge       {"identifier": "vtt.masterflinter.net", "challenge_type": "tls-alpn-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2023/10/08 19:28:14.286 ERROR   tls.acme_client challenge failed        {"identifier": "vtt.masterflinter.net", "challenge_type": "tls-alpn-01", "problem": {"type": "urn:ietf:params:acme:error:connection", "title": "", "detail": "162.207.102.104: Timeout during connect (likely firewall problem)", "instance": "", "subproblems": []}}
2023/10/08 19:28:14.286 ERROR   tls.acme_client validating authorization        {"identifier": "vtt.masterflinter.net", "problem": {"type": "urn:ietf:params:acme:error:connection", "title": "", "detail": "162.207.102.104: Timeout during connect (likely firewall problem)", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/121166254/11453462784", "attempt": 1, "max_attempts": 3}
2023/10/08 19:28:15.422 INFO    tls.acme_client trying to solve challenge       {"identifier": "vtt.masterflinter.net", "challenge_type": "http-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2023/10/08 19:28:25.706 ERROR   tls.acme_client challenge failed        {"identifier": "vtt.masterflinter.net", "challenge_type": "http-01", "problem": {"type": "urn:ietf:params:acme:error:connection", "title": "", "detail": "162.207.102.104: Fetching http://vtt.masterflinter.net/.well-known/acme-challenge/oJVJwf3dChqPJXIrO2LgckFP-mkGCRwIA_qFExrLuag: Timeout during connect (likely firewall problem)", "instance": "", "subproblems": []}}
2023/10/08 19:28:25.706 ERROR   tls.acme_client validating authorization        {"identifier": "vtt.masterflinter.net", "problem": {"type": "urn:ietf:params:acme:error:connection", "title": "", "detail": "162.207.102.104: Fetching http://vtt.masterflinter.net/.well-known/acme-challenge/oJVJwf3dChqPJXIrO2LgckFP-mkGCRwIA_qFExrLuag: Timeout during connect (likely firewall problem)", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/121166254/11453464954", "attempt": 2, "max_attempts": 3}
2023/10/08 19:28:25.706 ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "vtt.masterflinter.net", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 400 urn:ietf:params:acme:error:connection - 162.207.102.104: Fetching http://vtt.masterflinter.net/.well-known/acme-challenge/oJVJwf3dChqPJXIrO2LgckFP-mkGCRwIA_qFExrLuag: Timeout during connect (likely firewall problem)"}
2023/10/08 19:28:27.942 INFO    tls.acme_client trying to solve challenge       {"identifier": "vtt.masterflinter.net", "challenge_type": "http-01", "ca": "https://acme.zerossl.com/v2/DV90"}
2023/10/08 19:28:39.447 ERROR   tls.acme_client challenge failed        {"identifier": "vtt.masterflinter.net", "challenge_type": "http-01", "problem": {"type": "", "title": "", "detail": "", "instance": "", "subproblems": []}}
2023/10/08 19:28:39.447 ERROR   tls.acme_client validating authorization        {"identifier": "vtt.masterflinter.net", "problem": {"type": "", "title": "", "detail": "", "instance": "", "subproblems": []}, "order": "https://acme.zerossl.com/v2/DV90/order/9wi8NApJ2UlwR8keH9tPVA", "attempt": 1, "max_attempts": 3}
2023/10/08 19:28:39.447 ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "vtt.masterflinter.net", "issuer": "acme.zerossl.com-v2-DV90", "error": "HTTP 0  - "}
2023/10/08 19:28:39.447 ERROR   tls.obtain      will retry      {"error": "[vtt.masterflinter.net] Obtain: [vtt.masterflinter.net] solving challenge: vtt.masterflinter.net: [vtt.masterflinter.net] authorization failed: HTTP 0  -  (ca=https://acme.zerossl.com/v2/DV90)", "attempt": 4, "retrying_in": 300, "elapsed": 432.093219116, "max_duration": 2592000}
2023/10/08 19:33:39.449 INFO    tls.obtain      obtaining certificate   {"identifier": "vtt.masterflinter.net"}
2023/10/08 19:33:39.759 INFO    tls.acme_client trying to solve challenge       {"identifier": "vtt.masterflinter.net", "challenge_type": "tls-alpn-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2023/10/08 19:33:50.246 ERROR   tls.acme_client challenge failed        {"identifier": "vtt.masterflinter.net", "challenge_type": "tls-alpn-01", "problem": {"type": "urn:ietf:params:acme:error:connection", "title": "", "detail": "162.207.102.104: Timeout during connect (likely firewall problem)", "instance": "", "subproblems": []}}
2023/10/08 19:33:50.246 ERROR   tls.acme_client validating authorization        {"identifier": "vtt.masterflinter.net", "problem": {"type": "urn:ietf:params:acme:error:connection", "title": "", "detail": "162.207.102.104: Timeout during connect (likely firewall problem)", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/121166254/11453517664", "attempt": 1, "max_attempts": 3}
2023/10/08 19:33:51.375 INFO    tls.acme_client trying to solve challenge       {"identifier": "vtt.masterflinter.net", "challenge_type": "http-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2023/10/08 19:34:01.558 ERROR   tls.acme_client challenge failed        {"identifier": "vtt.masterflinter.net", "challenge_type": "http-01", "problem": {"type": "urn:ietf:params:acme:error:connection", "title": "", "detail": "162.207.102.104: Fetching http://vtt.masterflinter.net/.well-known/acme-challenge/O3icm81Y8aF4HFY9aRsoa5V8P-PdCx_rWVAyx94WKHg: Timeout during connect (likely firewall problem)", "instance": "", "subproblems": []}}
2023/10/08 19:34:01.559 ERROR   tls.acme_client validating authorization        {"identifier": "vtt.masterflinter.net", "problem": {"type": "urn:ietf:params:acme:error:connection", "title": "", "detail": "162.207.102.104: Fetching http://vtt.masterflinter.net/.well-known/acme-challenge/O3icm81Y8aF4HFY9aRsoa5V8P-PdCx_rWVAyx94WKHg: Timeout during connect (likely firewall problem)", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/121166254/11453519314", "attempt": 2, "max_attempts": 3}
2023/10/08 19:34:01.559 ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "vtt.masterflinter.net", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 400 urn:ietf:params:acme:error:connection - 162.207.102.104: Fetching http://vtt.masterflinter.net/.well-known/acme-challenge/O3icm81Y8aF4HFY9aRsoa5V8P-PdCx_rWVAyx94WKHg: Timeout during connect (likely firewall problem)"}
2023/10/08 19:34:03.425 INFO    tls.acme_client trying to solve challenge       {"identifier": "vtt.masterflinter.net", "challenge_type": "http-01", "ca": "https://acme.zerossl.com/v2/DV90"}
2023/10/08 19:34:15.840 ERROR   tls.acme_client challenge failed        {"identifier": "vtt.masterflinter.net", "challenge_type": "http-01", "problem": {"type": "", "title": "", "detail": "", "instance": "", "subproblems": []}}
2023/10/08 19:34:15.840 ERROR   tls.acme_client validating authorization        {"identifier": "vtt.masterflinter.net", "problem": {"type": "", "title": "", "detail": "", "instance": "", "subproblems": []}, "order": "https://acme.zerossl.com/v2/DV90/order/cfWxTsXvfupfn4tzqDIVuw", "attempt": 1, "max_attempts": 3}
2023/10/08 19:34:15.840 ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "vtt.masterflinter.net", "issuer": "acme.zerossl.com-v2-DV90", "error": "HTTP 0  - "}
2023/10/08 19:34:15.840 ERROR   tls.obtain      will retry      {"error": "[vtt.masterflinter.net] Obtain: [vtt.masterflinter.net] solving challenge: vtt.masterflinter.net: [vtt.masterflinter.net] authorization failed: HTTP 0  -  (ca=https://acme.zerossl.com/v2/DV90)", "attempt": 5, "retrying_in": 600, "elapsed": 768.486235928, "max_duration": 2592000}

2023/10/08 19:38:14.010 DEBUG tls.issuance.acme.acme_client http request {“method”: “POST”, “url”: “https://acme-v02.api.letsencrypt.org/acme/authz-v3/271985729486”, “headers”: {“Content-Type”:[“application/jose+json”],“User-Agent”:[“Caddy/2.7.4 CertMagic acmez (linux; amd64)”]}, “response_headers”: {“Boulder-Requester”:[“1350494416”],“Cache-Control”:[“public, max-age=0, no-cache”],“Content-Length”:[“805”],“Content-Type”:[“application/json”],“Date”:[“Sun, 08 Oct 2023 19:38:13 GMT”],“Link”:[“https://acme-v02.api.letsencrypt.org/directory;rel="index"”],“Replay-Nonce”:[“IAvmRRVVwJRvn2zAQHUS1NU6nxZFOqFguCJqeWecymBtOaHbV5I”],“Server”:[“nginx”],“Strict-Transport-Security”:[“max-age=604800”],“X-Frame-Options”:[“DENY”]}, “status_code”: 200}
2023/10/08 19:38:14.308 DEBUG tls.issuance.acme.acme_client http request {“method”: “POST”, “url”: “https://acme-v02.api.letsencrypt.org/acme/authz-v3/271985729486”, “headers”: {“Content-Type”:[“application/jose+json”],“User-Agent”:[“Caddy/2.7.4 CertMagic acmez (linux; amd64)”]}, “response_headers”: {“Boulder-Requester”:[“1350494416”],“Cache-Control”:[“public, max-age=0, no-cache”],“Content-Length”:[“805”],“Content-Type”:[“application/json”],“Date”:[“Sun, 08 Oct 2023 19:38:14 GMT”],“Link”:[“https://acme-v02.api.letsencrypt.org/directory;rel="index"”],“Replay-Nonce”:[“IAvmRRVVN617HC4m8RZZzYK36OuMeJ-WJJx0YaWzGA8hz7kZFTY”],“Server”:[“nginx”],“Strict-Transport-Security”:[“max-age=604800”],“X-Frame-Options”:[“DENY”]}, “status_code”: 200}
2023/10/08 19:38:14.615 DEBUG tls.issuance.acme.acme_client http request {“method”: “POST”, “url”: “https://acme-v02.api.letsencrypt.org/acme/authz-v3/271985729486”, “headers”: {“Content-Type”:[“application/jose+json”],“User-Agent”:[“Caddy/2.7.4 CertMagic acmez (linux; amd64)”]}, “response_headers”: {“Boulder-Requester”:[“1350494416”],“Cache-Control”:[“public, max-age=0, no-cache”],“Content-Length”:[“805”],“Content-Type”:[“application/json”],“Date”:[“Sun, 08 Oct 2023 19:38:14 GMT”],“Link”:[“https://acme-v02.api.letsencrypt.org/directory;rel="index"”],“Replay-Nonce”:[“_s_5u1NQcroy7zYVxpO_vqm-FjlwaSw46c_4tF22nHlsXV6OoSI”],“Server”:[“nginx”],“Strict-Transport-Security”:[“max-age=604800”],“X-Frame-Options”:[“DENY”]}, “status_code”: 200}

Timeout during connect (likely firewall problem)

I would take this advice and double-check that your server is open to the public Internet on ports 80 and 443 – that seems to be the problem. (Firewall, networking, routing issue somewhere close to your server.)

Thanks for chiming in. I started snooping around my networking and noticed that Docker, which I am not using at all right now, had an interface lingering around. After removing the interface (docker0) and rebooting my box caddy fails to run giving the following error:

2023/10/08 22:40:20.175 INFO    using provided configuration    {"config_file": "/etc/caddy/Caddyfile", "config_adapter": ""}
Error: loading initial config: loading new config: starting caddy administration endpoint: listen tcp 127.0.0.1:2019: bind: address already in use

netstat shows that 2019 is bound to Caddy so I am not sure what the problem is. Also, I am not sure what 2019 is for. Is that a default port that Caddy uses for something? Why did removing Docker0 create this problem?

Caddy starts an administration endpoint at port 2019. So it is already running on that system.

I’m not sure about the Docker stuff, you’ll have to ask the Docker community how to manage ports I guess. I usually recommend running Caddy natively on the host rather than in a container, because Caddy is already a single static binary and it makes things simpler when you’re just starting out.

FWIW I don’t agree, running Caddy in Docker is fine.

Anyway, how did you try to start Caddy? You should follow these instructions: Keep Caddy Running — Caddy Documentation Don’t run caddy run directly, run it as a service instead.

1 Like

I am running Caddy natively on the host.

In order to grab the logs I did caddy run directly, but if things were working correctly I would be letting pm2 run Caddy.

pm2 start "caddy run --config /etc/caddy/Caddyfile" --name caddy

Is this a bad way to use Caddy?

@matt I have verified that no firewall is running on my server. I can also see that caddy is bound to 80 and 443:

tcp        0      0 127.0.0.1:2019          0.0.0.0:*               LISTEN      4735/caddy          
tcp6       0      0 :::443                  :::*                    LISTEN      4735/caddy          
tcp6       0      0 :::80                   :::*                    LISTEN      4735/caddy          
udp6       0      0 :::443                  :::*                                4735/caddy

I noticed it is listening on tcp6 which google tells me is ipv6. Is this a concern? I had ipv6 disabled on my router but re-enabled it to no avail, my connection still times out when trying to hit vtt.masterflinter.net.

Also thank you both for helping out I really appreciate it.

Having it listen on tcp6/udp6 isn’t a problem. Not having it also listen on tcp4/udp4 might be a problem.

You probably don’t have an IPv6 record (an AAAA record) if you haven’t configured one - so even if IPv6 is available, unless DNS resolves to it, it won’t be used to connect.

I’d say, run Caddy, then curl it locally, directly to its LAN IP, to check if Caddy is behaving as expected. Do it like so:

curl -v --resolve vtt.masterflinter.net:443:10.0.0.23 vtt.masterflinter.net

That will essentially “short circuit” the DNS so it’s not trying to reach out through public internet. It’ll just treat 10.0.0.23 as the canonical IP of the website and go straight there. If Caddy’s working, we’ll get a working website, and we then know that the issue is probably somewhere around public internet and the router firewall. If it doesn’t work, we maybe start thinking there’s an issue with Caddy itself.

My other hunch is that maybe your ISP allows port 30000 but denies ports 80/443 as “known hosting ports”; some ISPs claim to do this to curb abusive behaviour. You could try forwarding router port 80 to the Foundry server port 30000 and see if that works; if it doesn’t, but 30000 → 30000 did, you’ve got a pretty good confirmation that’d be the issue.

1 Like

To be clear, tcp6/udp6 mean “tcp4 + tcp6”, it doesn’t mean only 6. It’s misleading unfortunately and I wish it was better indicated in those programs. I think they don’t show tcp4 when it’s only 4 because some old programs may depend on only seeing “tcp” in the output :man_shrugging:

Anyway, if your system is systemd-based, I recommend using Caddy as a systemd service. That’s what we have experience with and what we can support. I don’t know much about pm2 but it’s not something we actively support (i.e. we can’t really help you with it because we don’t use it, and we don’t have documentation for it).

2 Likes

(It is fine, I just wanted to suggest trying without since that requires an extra skill set and for troubleshooting it may be useful without.)

Would be nice if it were something that at least hinted at the dual nature. tcp+6 / udp+6 for example.

Alright glad to know tcp6 isn’t the problem. I tried your curl and something I did not expect happened. The connection timed out but it was trying to connect using my public ip, not going straight to my internal ip.

flint@flint-home-server:~$ curl -v --resolve vtt.masterflinter.net:443:10.0.0.23 vtt.masterflinter.net
* Added vtt.masterflinter.net:443:10.0.0.23 to DNS cache
*   Trying 162.207.102.104:80...
* connect to 162.207.102.104 port 80 failed: Connection timed out
* Failed to connect to vtt.masterflinter.net port 80 after 129317 ms: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to vtt.masterflinter.net port 80 after 129317 ms: Connection timed out

For what it’s worth if I enter 10.0.0.23:443 in a web browser it returns a line of text: “Client sent an HTTP request to an HTTPS server.”. Entering 10.0.0.23:80 does serve up my foundry instance.

So it appears that Caddy is successfully reverse-proxying traffic to my foundry server but is not successfully handling SSL, right? However, your curl was unable to connect on port 80 when using my public ip so maybe my port forwarding is not working correctly and I have two problems at once.

I think my DNS is set up correctly since vtt.masterflinter.net is correctly resolving to my public ip address but I have not proven that port forwarding is working for 80/443 although it is working for 30000.

I will try this soon, thanks for the direction.

Oh, whoops! My bad. I should have given you the command:

curl -v --resolve vtt.masterflinter.net:443:10.0.0.23 https://vtt.masterflinter.net

Note the https:// scheme. Because I didn’t give you that, it tried HTTP on port 80 instead (which we didn’t pin with --resolve vtt.masterflinter.net:80:10.0.0.23) so it went out to DNS.

That’s a good result from 10.0.0.23:80 but, FYI, you should use https://10.0.0.23 instead of 10.0.0.23:443. Browsers might not infer that you’re trying to make a secure connection and might literally try HTTP on port 443.

1 Like

This is the result of the corrected curl:

flint@flint-home-server:~/Documents$ cat curl443 
flint@flint-home-server:~$ curl -v --resolve vtt.masterflinter.net:443:10.0.0.23 https://vtt.masterflinter.net
* Added vtt.masterflinter.net:443:10.0.0.23 to DNS cache
* Hostname vtt.masterflinter.net was found in DNS cache
*   Trying 10.0.0.23:443...
* Connected to vtt.masterflinter.net (10.0.0.23) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Unknown (21):
* TLSv1.3 (IN), TLS alert, internal error (592):
* error:0A000438:SSL routines::tlsv1 alert internal error
* Closing connection 0
curl: (35) error:0A000438:SSL routines::tlsv1 alert internal error

That’s… actually kinda the expected result. Caddy shouldn’t have a valid cert yet so it’ll just shut down an attempt to connect.

Try adding tls internal to your vtt.masterflinter.net site in your Caddyfile and start Caddy again. Repeat the curl command you just ran, but this time, add the k flag:

curl -kv --resolve vtt.masterflinter.net:443:10.0.0.23 https://vtt.masterflinter.net

We expect to see a proper handshake and HTTPS request/response. Assuming you get one, try without the --resolve flag:

curl -kv https://vtt.masterflinter.net

We expect to see the same result. If we get a different result, we can infer that while Caddy is responding as expected within the network, something is going wrong trying to access it via the router.

1 Like

When using ‘tls internal’ I get the same result:

flint@flint-home-server:~$ curl -kv --resolve vtt.masterflinter.net:443:10.0.0.23 https://vtt.masterflinter.net
* Added vtt.masterflinter.net:443:10.0.0.23 to DNS cache
* Hostname vtt.masterflinter.net was found in DNS cache
*   Trying 10.0.0.23:443...
* Connected to vtt.masterflinter.net (10.0.0.23) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: [NONE]
*  start date: Oct 11 17:21:13 2023 GMT
*  expire date: Oct 12 05:21:13 2023 GMT
*  issuer: CN=Caddy Local Authority - ECC Intermediate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x557de2db7e90)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host: vtt.masterflinter.net
> user-agent: curl/7.81.0
> accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 302 
< alt-svc: h3=":443"; ma=2592000
< content-type: text/plain; charset=utf-8
< date: Wed, 11 Oct 2023 18:20:22 GMT
< location: /setup
< server: Caddy
< vary: Accept
< x-powered-by: Express
< content-length: 28
< 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection #0 to host vtt.masterflinter.net left intact
flint@flint-home-server:~$ curl -kv https://vtt.masterflinter.net
*   Trying 162.207.102.104:443...
* Connected to vtt.masterflinter.net (162.207.102.104) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: [NONE]
*  start date: Oct 11 17:21:13 2023 GMT
*  expire date: Oct 12 05:21:13 2023 GMT
*  issuer: CN=Caddy Local Authority - ECC Intermediate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x5595bd63ae90)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host: vtt.masterflinter.net
> user-agent: curl/7.81.0
> accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 302 
< alt-svc: h3=":443"; ma=2592000
< content-type: text/plain; charset=utf-8
< date: Wed, 11 Oct 2023 18:21:22 GMT
< location: /setup
< server: Caddy
< vary: Accept
< x-powered-by: Express
< content-length: 28
< 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection #0 to host vtt.masterflinter.net left intact
Found. Redirecting to /setup

If I do not have ‘tls internal’ then my connection times out when not using the resolve flag

That’s interesting. Both of those outputs look good, but I would have expected that curl: (35) error:0A000438:SSL routines::tlsv1 alert internal error response again once you turned tls internal off for the DNS-based request.

If it works as expected when the certificate is there, we expect it to break in a predictable manner when we take the certificate away. Timing out is not really a predictable failure mode when the only change in configuration is the lack of a certificate. It almost makes me wonder if there’s some other transport or application layer thing happening in the middle, here…

1 Like

FWIW I can connect to your server; I see an HTTP->HTTPS redirect from curl -v http://vtt.masterflinter.net as expected, and curl -vk https://vtt.masterflinter.net/setup returns HTML (using -k to ignore the error from tls internal which you still seem to have on).

So let’s start from the beginning:

Make sure you’re running Caddy as a systemd service (not with caddy run directly from your terminal) so that it stores the certs in the right location. Turn off any other instances of Caddy that may be running first (if you ran caddy start at some point, etc). Then restart the systemd service.

Remove tls internal and restart Caddy. Check your Caddy logs (via journalctl). What do you see? I’m not seeing any firewall/dns/connectivity issues at this point so cert issuance should work just fine.

2 Likes