Caddy won't generate on-demand certificates at all, bug found?

Help
1

1. The problem I’m having:

I am deploying Caddy to Vultr via a Docker image. Using Caddyfiles, no issues with on-demand certificates, moved to JSON via MySQL adapter, issues caused.

2. Error messages and/or full log output:

There are literally zero errors in the caddy logs with debug enabled.

Only errors received are via browser when accessing the domain:

sent an invalid response.

ERR_SSL_PROTOCOL_ERROR

and via curl:

C:\Users\xxx>curl -I -k https://redacted.io
curl: (35) schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.

3. Caddy version:

Docker, latest.

4. How I installed and ran Caddy:

Docker.

b. Command:

[supervisord]
nodaemon=true
user=root
logfile=/dev/stdout
logfile_maxbytes=0
pidfile=/var/run/supervisord.pid

[program:caddy]
command=caddy run --adapter mysql --config /usr/bin/mysql.json
autostart=true
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

[program:optimizer]
command=/usr/local/bin/optimizer
autostart=true
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

c. Service/unit/compose file:

version: '3.8'
services:
  app:
    build: .
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
      - "3001:3001"
      - "2019:2019"
    volumes:
      - ./mysql.json:/usr/bin/mysql.json
    restart: unless-stopped

d. My complete Caddy config:

config.app

{
  "http": {
    "http_port": 80,
    "https_port": 443,
    "servers": {
      "srv0": {
        "listen": [
          ":443",
          ":80"
        ],
        "protocols": [
          "h1",
          "h2",
          "h3"
        ]
      }
    }
  },
  "cache": {
    "allowed_http_verbs": [
      "GET",
      "HEAD"
    ],
    "cache_name": "Redis-Proxy",
    "default_cache_control": "public, max-age=7200",
    "distributed": true,
    "mode": "bypass",
    "redis": {
      "found": true,
      "configuration": {
        "AlwaysPipelining": true,
        "ClientName": "redacted",
        "DisableTCPNoDelay": "true",
        "InitAddress": [
          "redacted"
        ]
      }
    },
    "timeout": {
      "backend": "0s",
      "cache": "0s"
    },
    "ttl": "10s"
  },
  "tls": {
    "automation": {
      "policies": [
        {
          "on_demand": true
        }
      ],
      "on_demand": {
        "permission": {
          "module": "http",
          "endpoint": "https://redacted.io/try/ssl.php"
        }
      }
    }
  }
}

config.apps.http.servers.srv0.routes

{
  "handle": [
    {
      "handler": "subroute",
      "routes": [
        {
          "handle": [
            {
              "handler": "encode",
              "encodings": {
                "zstd": {
                  "level": "fastest"
                },
                "gzip": {
                  "level": 1
                }
              },
              "prefer": [
                "zstd"
              ]
            },
            {
              "handler": "cache",
              "cache_name": "Redis-Proxy"
            },
            {
              "handler": "reverse_proxy",
              "headers": {
                "request": {
                  "set": {
                    "Host": [
                      "redacted.io"
                    ]
                  }
                }
              },
              "upstreams": [
                {
                  "dial": "7x.6x.x1.xxx:80"
                }
              ]
            }
          ]
        }
      ]
    }
  ],
  "match": [
    {
      "host": [
        "redacted.io"
      ]
    }
  ]
}
2

Further console output:

{"level":"info","ts":1741476953.5616205,"msg":"using config from file","file":"/usr/bin/mysql.json"}                                                                               
app-1  | {"level":"info","ts":1741476953.9279468,"msg":"adapted config to JSON","adapter":"mysql"}
app-1  | {"level":"warn","ts":1741476953.9289687,"logger":"admin","msg":"admin endpoint disabled"}
app-1  | {"level":"info","ts":1741476953.9297888,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
app-1  | {"level":"info","ts":1741476953.930279,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00047fc00"}                                 
app-1  | {"level":"info","ts":1741476953.9314852,"logger":"storages.cache.redis","msg":"Cannot parse your redis configuration: json: cannot unmarshal string into Go struct field ClientOption.DisableTCPNoDelay of type bool"}                                                                                                                                                         
app-1  | {"level":"error","ts":1741476954.9803865,"logger":"http.handlers.cache","msg":"Error during Redis init, did you include the Redis storage (--with github.com/darkweak/storages/redis/caddy or github.com/darkweak/storages/go-redis/caddy)? provision storages.cache.redis: dial tcp 10.7.13.249:6379: i/o timeout"}
app-1  | {"level":"warn","ts":1741476954.9804125,"logger":"http.handlers.cache","msg":"You're running Souin with the default storage that is not optimized and for development purpose. We recommend to use at least one of the storages from https://github.com/darkweak/storages"}                                                                                                    
app-1  | {"level":"info","ts":1741476954.9804342,"logger":"http.handlers.cache","msg":"Set backend timeout to 10s"}
app-1  | {"level":"info","ts":1741476954.980436,"logger":"http.handlers.cache","msg":"Set cache timeout to 10ms"}                                                                           
app-1  | {"level":"info","ts":1741476954.9804378,"logger":"http.handlers.cache","msg":"Souin configuration is now loaded."}                                                                 
app-1  | 2025-03-08 23:35:54,980 INFO success: caddy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)                                                          
app-1  | 2025-03-08 23:35:54,980 INFO success: caddy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)                                                          
app-1  | 2025-03-08 23:35:54,980 INFO success: optimizer entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)                                                      
app-1  | 2025-03-08 23:35:54,980 INFO success: optimizer entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
app-1  | {"level":"info","ts":1741476954.9808707,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}                                                                            
app-1  | {"level":"info","ts":1741476954.981464,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}                                                                                                                                                  
app-1  | {"level":"warn","ts":1741476954.982101,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"}
app-1  | {"level":"warn","ts":1741476954.9821455,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"}                                               
app-1  | {"level":"info","ts":1741476954.9821482,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
app-1  | {"level":"info","ts":1741476954.9821513,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["redacted.io"]}                                         
app-1  | {"level":"info","ts":1741476954.9821637,"msg":"serving initial configuration"}                                                                                                     
app-1  | {"level":"info","ts":1741476954.9888117,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/root/.local/share/caddy"}                                             
app-1  | {"level":"info","ts":1741476954.995714,"logger":"tls","msg":"finished cleaning storage units"}
3 
C:\Users\xxx>curl -I http://redacted.io
HTTP/1.1 200 OK
Accept-Ranges: bytes
Alt-Svc: h3=":443"; ma=2592000
Cache-Control: public, max-age=7200
Cache-Status: Redis-Proxy; fwd=uri-miss; detail=UPSTREAM-ERROR-OR-EMPTY-RESPONSE; key=HEAD-http-redacted.io-/
Content-Length: 20767
Content-Type: text/html
Date: Sat, 08 Mar 2025 23:40:39 GMT
Etag: "511f-679a555b-198025e;;;"
Last-Modified: Wed, 29 Jan 2025 16:20:43 GMT
Server: Caddy
Server: LiteSpeed

Website is accessible via http.

4

Likely not a bug, probably a misconfiguration.

Looks like the only logs available here are server startup, but there’s no evidence Caddy emitted any error such as you describe (ERR_SSL_PROTOCOL_ERROR).

Enabling DEBUG-level logging might be helpful, and including all the log output including while trying a request.

Since you’ve redacted the domains – which is against our forum rules as the help template mentions – we cannot help you any further with troubleshooting until we have enough information.