1. The problem I’m having:
Based on my previous post (Dockerize Caddy with existing SSL certificate), I’ve let caddy handle all the necessary steps to issue the certificate for my staging environment. Now, I want to apply it to production as well (it has a different domain name).
Currently, we’re using a TLS configuration that is using email for the production. As you can see in the config, we use on_demand_tls with a specific endpoint there. The issuer of our wildcard subdomain is AWS with the auto-renewal enabled. So, the endpoint is working as expected even if I put an https on on_demand_tls.
With the current configuration that I attached (not the currently working one), we got an issue when getting the certificate, both from letsencrypt or zerossl. If both failed, what else can I do to make automatic-https work as expected? Especially without breaking the current setup (the current one is running on a different server, the new one is being tested as an ECS service).
2. Error messages and/or full log output:
The log is in cloudwatch format.
{"level": "info","ts": 1707713402.5923235,"msg": "using provided configuration","config_file": "/etc/caddy/Caddyfile","config_adapter": ""}
{"level": "info","ts": 1707713402.612486,"logger": "admin","msg": "admin endpoint started","address": "0.0.0.0:2020","enforce_origin": false,"origins": ["//0.0.0.0:2020"]}
{"level": "warn","ts": 1707713402.6125135,"logger": "admin","msg": "admin endpoint on open interface; host checking disabled","address": "0.0.0.0:2020"}
{"level": "info","ts": 1707713402.6157515,"logger": "caddy.storage.redis","msg": "Provision Redis simple storage using address [127.0.0.1:6379]"}
{"level": "info","ts": 1707713402.6163588,"logger": "tls.cache.maintenance","msg": "started background certificate maintenance","cache": "0xc000240080"}
{"level": "info","ts": 1707713402.6166704,"logger": "http.auto_https","msg": "server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name": "srv0","https_port": 443}
{"level": "info","ts": 1707713402.6166928,"logger": "http.auto_https","msg": "enabling automatic HTTP->HTTPS redirects","server_name": "srv0"}
{"level": "info","ts": 1707713402.6190116,"logger": "http","msg": "enabling HTTP/3 listener","addr": ":443"}
{"level": "info","ts": 1707713402.6191123,"msg": "failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
{"level": "info","ts": 1707713402.6196246,"logger": "http.log","msg": "server running","name": "srv0","protocols": ["h1","h2","h3"]}
{"level": "info","ts": 1707713402.6196887,"logger": "http.log","msg": "server running","name": "remaining_auto_https_redirects","protocols": ["h1","h2","h3"]}
{"level": "info","ts": 1707713402.619697,"logger": "http","msg": "enabling automatic TLS certificate management","domains": ["*.company-domain.id","*.ordivo.id"]}
{"level": "info","ts": 1707713402.6308644,"msg": "autosaved config (load with --resume flag)","file": "/config/caddy/autosave.json"}
{"level": "info","ts": 1707713402.6308906,"msg": "serving initial configuration"}
{"level": "warn","ts": 1707713402.6321352,"logger": "tls","msg": "storage cleaning happened too recently; skipping for now","storage": "{\"client_type\":\"simple\",\"address\":[\"127.0.0.1:6379\"],\"host\":[],\"port\":[],\"db\":0,\"timeout\":\"5\",\"username\":\"\",\"password\":\"REDACTED\",\"master_name\":\"\",\"key_prefix\":\"caddy\",\"encryption_key\":\"\",\"compression\":false,\"tls_enabled\":false,\"tls_insecure\":true,\"tls_server_certs_pem\":\"\",\"tls_server_certs_path\":\"\",\"route_by_latency\":false,\"route_randomly\":false}","instance": "ef441035-bdfb-4312-8459-d29bffa0d529","try_again": 1707799802.6321328,"try_again_in": 86399.999999488}
{"level": "info","ts": 1707713402.6326098,"logger": "tls","msg": "finished cleaning storage units"}
{"level": "info","ts": 1707713402.6363957,"logger": "tls.obtain","msg": "acquiring lock","identifier": "*.company-domain.id"}
{"level": "info","ts": 1707713402.6370473,"logger": "tls.obtain","msg": "lock acquired","identifier": "*.company-domain.id"}
{"level": "info","ts": 1707713402.6381674,"logger": "tls.obtain","msg": "obtaining certificate","identifier": "*.company-domain.id"}
{"level": "info","ts": 1707713402.6422555,"logger": "tls","msg": "waiting on internal rate limiter","identifiers": ["*.company-domain.id"],"ca": "https://acme-v02.api.letsencrypt.org/directory","account": ""}
{"level": "info","ts": 1707713402.6422963,"logger": "tls","msg": "done waiting on internal rate limiter","identifiers": ["*.company-domain.id"],"ca": "https://acme-v02.api.letsencrypt.org/directory","account": ""}
{"level": "error","ts": 1707713404.3228283,"logger": "tls.obtain","msg": "could not get certificate from issuer","identifier": "*.company-domain.id","issuer": "acme-v02.api.letsencrypt.org-directory","error": "[*.company-domain.id] solving challenges: *.company-domain.id: no solvers available for remaining challenges (configured=[tls-alpn-01 http-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-v02.api.letsencrypt.org/acme/order/1561474067/243795882457) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
{"level": "info","ts": 1707713404.3278944,"logger": "tls","msg": "waiting on internal rate limiter","identifiers": ["*.company-domain.id"],"ca": "https://acme.zerossl.com/v2/DV90","account": "caddy@zerossl.com"}
{"level": "info","ts": 1707713404.3279233,"logger": "tls","msg": "done waiting on internal rate limiter","identifiers": ["*.company-domain.id"],"ca": "https://acme.zerossl.com/v2/DV90","account": "caddy@zerossl.com"}
{"level": "error","ts": 1707713433.9811187,"logger": "tls.obtain","msg": "could not get certificate from issuer","identifier": "*.company-domain.id","issuer": "acme.zerossl.com-v2-DV90","error": "[*.company-domain.id] solving challenges: *.company-domain.id: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme.zerossl.com/v2/DV90/order/U3vw-dFMXAr0mddxjQP5VQ) (ca=https://acme.zerossl.com/v2/DV90)"}
{"level": "error","ts": 1707713433.9812117,"logger": "tls.obtain","msg": "will retry","error": "[*.company-domain.id] Obtain: [*.company-domain.id] solving challenges: *.company-domain.id: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme.zerossl.com/v2/DV90/order/U3vw-dFMXAr0mddxjQP5VQ) (ca=https://acme.zerossl.com/v2/DV90)","attempt": 1,"retrying_in": 60,"elapsed": 31.344116349,"max_duration": 2592000}
{"level": "info","ts": 1707713493.982795,"logger": "tls.obtain","msg": "obtaining certificate","identifier": "*.company-domain.id"}
{"level": "error","ts": 1707713495.4197855,"logger": "tls.obtain","msg": "could not get certificate from issuer","identifier": "*.company-domain.id","issuer": "acme-v02.api.letsencrypt.org-directory","error": "[*.company-domain.id] solving challenges: *.company-domain.id: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/135734103/14443443283) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
{"level": "error","ts": 1707713514.7285583,"logger": "tls.obtain","msg": "could not get certificate from issuer","identifier": "*.company-domain.id","issuer": "acme.zerossl.com-v2-DV90","error": "[*.company-domain.id] solving challenges: *.company-domain.id: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme.zerossl.com/v2/DV90/order/sdyuYIPWhjCUR01SokucKw) (ca=https://acme.zerossl.com/v2/DV90)"}
{"level": "error","ts": 1707713514.7286437,"logger": "tls.obtain","msg": "will retry","error": "[*.company-domain.id] Obtain: [*.company-domain.id] solving challenges: *.company-domain.id: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme.zerossl.com/v2/DV90/order/sdyuYIPWhjCUR01SokucKw) (ca=https://acme.zerossl.com/v2/DV90)","attempt": 2,"retrying_in": 120,"elapsed": 112.091548146,"max_duration": 2592000}
3. Caddy version:
Caddy 2.7.6
4. How I installed and ran Caddy:
a. System environment:
b. Command:
CMD ["/usr/bin/caddy", "run", "--config", "/etc/caddy/Caddyfile"]
c. Service/unit/compose file:
PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.
d. My complete Caddy config:
{
admin 0.0.0.0:2020
on_demand_tls {
ask https://api.company-domain.id/domain-checker
}
storage redis {
host 127.0.0.1
port 6379
# username
password somerandompassword
db 0
timeout 5
key_prefix "caddy"
tls_enabled false
tls_insecure true
}
}
*.company-domain.id {
encode gzip
reverse_proxy /* {
to 127.0.0.1:2015 # localhost is just a placeholder
lb_policy least_conn
fail_duration 30s
header_up Host {host}
header_up X-Real-IP {header.X-Forwarded-For}
header_up Access-Control-Allow-Origin *
header_up Access-Control-Allow-Methods "GET, POST, PUT, PATCH, OPTIONS, DELETE"
header_down Access-Control-Allow-Origin *
header_down Access-Control-Allow-Methods "GET, POST, PUT, PATCH, OPTIONS, DELETE"
}
@assets path /js* /css* /favicon.ico
header @assets Cache-Control "public, max-age=31536000;"
log {
output file /var/log/caddy/access.log
format console
}
}
:443 {
encode gzip
tls company.email@gmail.com {
on_demand
}
reverse_proxy /* {
to 127.0.0.1:2015 # localhost is just a placeholder
lb_policy least_conn
fail_duration 30s
header_up Host {host}
header_up X-Real-IP {header.X-Forwarded-For}
}
@assets path /js* /css* /favicon.ico
header @assets Cache-Control "public, max-age=31536000;"
log {
output file /var/log/caddy/access.log
format console
}
}
