1. The problem I’m having:
When I do systemctl restart caddy-api it fails with below errors.
But if I run same command almost directly (as seen systemctl status): /usr/bin/caddy run --environ --config /etc/caddy/caddy.json (with env file) it works perfectly.
> curl -vL https://ns1.pvepve.charmain.com
* Host ns1.pvepve.charmain.com:443 was resolved.
* IPv6: 2406:7400:94:2e31:3e6a:d2ff:fed8:120
* IPv4: 192.168.0.120
* Trying [2406:7400:94:2e31:3e6a:d2ff:fed8:120]:443...
* Immediate connect fail for 2406:7400:94:2e31:3e6a:d2ff:fed8:120: Network is unreachable
* Trying 192.168.0.120:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS alert, internal error (592):
* TLS connect error: error:0A000438:SSL routines::tlsv1 alert internal error
* closing connection #0
curl: (35) TLS connect error: error:0A000438:SSL routines::tlsv1 alert internal error
2. Error messages and/or full log output:
Following error message is from journalctl :
Nov 03 16:07:58 caddy caddy[966]: {"level":"info","ts":1762186078.1566324,"msg":"using config from file","file":"/etc/caddy/caddy.json"}
Nov 03 16:07:58 caddy caddy[966]: {"level":"info","ts":1762186078.1575623,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
Nov 03 16:07:58 caddy caddy[966]: {"level":"info","ts":1762186078.158433,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Nov 03 16:07:58 caddy caddy[966]: {"level":"info","ts":1762186078.1585112,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Nov 03 16:07:58 caddy caddy[966]: {"level":"warn","ts":1762186078.1588588,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"}
Nov 03 16:07:58 caddy caddy[966]: {"level":"warn","ts":1762186078.158909,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"}
Nov 03 16:07:58 caddy caddy[966]: {"level":"info","ts":1762186078.1589453,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Nov 03 16:07:58 caddy caddy[966]: {"level":"info","ts":1762186078.1590075,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Nov 03 16:07:58 caddy caddy[966]: {"level":"info","ts":1762186078.1590953,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
Nov 03 16:07:58 caddy caddy[966]: {"level":"info","ts":1762186078.1592019,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Nov 03 16:07:58 caddy caddy[966]: {"level":"info","ts":1762186078.1592412,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["*.pvepve.charmain.com"]}
Nov 03 16:07:58 caddy caddy[966]: {"level":"info","ts":1762186078.159463,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Nov 03 16:07:58 caddy caddy[966]: {"level":"info","ts":1762186078.159541,"msg":"serving initial configuration"}
Nov 03 16:07:58 caddy systemd[1]: Started caddy.service - Caddy.
Nov 03 16:07:58 caddy caddy[966]: {"level":"info","ts":1762186078.1613352,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/var/lib/caddy/.local/share/caddy","instance":"1e3480e2-2de0-4ddc-9af0-137b1d74d0e2","try_again":1762272478.1613343,"try_again_in":86399.999999672}
Nov 03 16:07:58 caddy caddy[966]: {"level":"info","ts":1762186078.1622725,"logger":"tls","msg":"finished cleaning storage units"}
Nov 03 16:07:58 caddy caddy[966]: {"level":"info","ts":1762186078.162568,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0000e8000"}
Nov 03 16:07:58 caddy caddy[966]: {"level":"info","ts":1762186078.163396,"logger":"tls.obtain","msg":"acquiring lock","identifier":"*.pvepve.charmain.com"}
Nov 03 16:07:58 caddy caddy[966]: {"level":"info","ts":1762186078.1638653,"logger":"tls.obtain","msg":"lock acquired","identifier":"*.pvepve.charmain.com"}
Nov 03 16:07:58 caddy caddy[966]: {"level":"info","ts":1762186078.1639633,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"*.pvepve.charmain.com"}
Nov 03 16:07:58 caddy caddy[966]: {"level":"info","ts":1762186078.1646638,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["*.pvepve.charmain.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
Nov 03 16:07:58 caddy caddy[966]: {"level":"info","ts":1762186078.164733,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["*.pvepve.charmain.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
Nov 03 16:07:58 caddy caddy[966]: {"level":"info","ts":1762186078.1647856,"logger":"tls.issuance.acme","msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/2771724901","account_contact":[]}
Nov 03 16:07:59 caddy caddy[966]: {"level":"info","ts":1762186079.7057455,"msg":"trying to solve challenge","identifier":"*.pvepve.charmain.com","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Nov 03 16:08:00 caddy caddy[966]: {"level":"error","ts":1762186080.6301908,"msg":"cleaning up solver","identifier":"*.pvepve.charmain.com","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.pvepve.charmain.com\" (usually OK if presenting also failed)","stacktrace":"github.com/mholt/acmez/v3.(*Client).solveChallenges.func1\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:318\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:363\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:136\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.24.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.24.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.2/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.24.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.24.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.24.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.24.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.24.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.24.0/async.go:73"}
Nov 03 16:08:01 caddy caddy[966]: {"level":"error","ts":1762186081.278485,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"*.pvepve.charmain.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[*.pvepve.charmain.com] solving challenges: presenting for challenge: adding temporary record for zone \"pvepve.charmain.com.\": expected 1 zone, got 0 for pvepve.charmain.com. (order=https://acme-v02.api.letsencrypt.org/acme/order/2771724901/444581164241) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
Nov 03 16:08:01 caddy caddy[966]: {"level":"error","ts":1762186081.278522,"logger":"tls.obtain","msg":"will retry","error":"[*.pvepve.charmain.com] Obtain: [*.pvepve.charmain.com] solving challenges: presenting for challenge: adding temporary record for zone \"pvepve.charmain.com.\": expected 1 zone, got 0 for pvepve.charmain.com. (order=https://acme-v02.api.letsencrypt.org/acme/order/2771724901/444581164241) (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":3.114600301,"max_duration":2592000}
Nov 03 16:09:01 caddy caddy[966]: {"level":"info","ts":1762186141.278829,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"*.pvepve.charmain.com"}
Nov 03 16:09:01 caddy caddy[966]: {"level":"info","ts":1762186141.2793884,"logger":"tls.issuance.acme","msg":"using ACME account","account_id":"https://acme-staging-v02.api.letsencrypt.org/acme/acct/239945983","account_contact":[]}
Nov 03 16:09:02 caddy caddy[966]: {"level":"info","ts":1762186142.7989619,"msg":"trying to solve challenge","identifier":"*.pvepve.charmain.com","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Nov 03 16:09:03 caddy caddy[966]: {"level":"error","ts":1762186143.7097785,"msg":"cleaning up solver","identifier":"*.pvepve.charmain.com","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.pvepve.charmain.com\" (usually OK if presenting also failed)","stacktrace":"github.com/mholt/acmez/v3.(*Client).solveChallenges.func1\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:318\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:363\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:136\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.24.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.24.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.2/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.24.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.24.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.24.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.24.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.24.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.24.0/async.go:73"}
Nov 03 16:09:03 caddy caddy[966]: {"level":"error","ts":1762186143.9527235,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"*.pvepve.charmain.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[*.pvepve.charmain.com] solving challenges: presenting for challenge: adding temporary record for zone \"pvepve.charmain.com.\": expected 1 zone, got 0 for pvepve.charmain.com. (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/239945983/28474330823) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
Nov 03 16:09:03 caddy caddy[966]: {"level":"error","ts":1762186143.952766,"logger":"tls.obtain","msg":"will retry","error":"[*.pvepve.charmain.com] Obtain: [*.pvepve.charmain.com] solving challenges: presenting for challenge: adding temporary record for zone \"pvepve.charmain.com.\": expected 1 zone, got 0 for pvepve.charmain.com. (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/239945983/28474330823) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":2,"retrying_in":120,"elapsed":65.788844099,"max_duration":2592000}
Nov 03 16:11:03 caddy caddy[966]: {"level":"info","ts":1762186263.9529068,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"*.pvepve.charmain.com"}
Nov 03 16:11:03 caddy caddy[966]: {"level":"info","ts":1762186263.9538019,"logger":"tls.issuance.acme","msg":"using ACME account","account_id":"https://acme-staging-v02.api.letsencrypt.org/acme/acct/239945983","account_contact":[]}
Nov 03 16:11:04 caddy caddy[966]: {"level":"info","ts":1762186264.6717188,"msg":"trying to solve challenge","identifier":"*.pvepve.charmain.com","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Nov 03 16:11:05 caddy caddy[966]: {"level":"error","ts":1762186265.7932308,"msg":"cleaning up solver","identifier":"*.pvepve.charmain.com","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.pvepve.charmain.com\" (usually OK if presenting also failed)","stacktrace":"github.com/mholt/acmez/v3.(*Client).solveChallenges.func1\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:318\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:363\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:136\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.24.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.24.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.2/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.24.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.24.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.24.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.24.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.24.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.24.0/async.go:73"}
Nov 03 16:11:06 caddy caddy[966]: {"level":"error","ts":1762186266.0342307,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"*.pvepve.charmain.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[*.pvepve.charmain.com] solving challenges: presenting for challenge: adding temporary record for zone \"pvepve.charmain.com.\": expected 1 zone, got 0 for pvepve.charmain.com. (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/239945983/28474361503) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
Nov 03 16:11:06 caddy caddy[966]: {"level":"error","ts":1762186266.0342684,"logger":"tls.obtain","msg":"will retry","error":"[*.pvepve.charmain.com] Obtain: [*.pvepve.charmain.com] solving challenges: presenting for challenge: adding temporary record for zone \"pvepve.charmain.com.\": expected 1 zone, got 0 for pvepve.charmain.com. (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/239945983/28474361503) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":3,"retrying_in":120,"elapsed":187.870346675,"max_duration":2592000}
3. Caddy version:
v2.10.2 h1:g/gTYjGMD0dec+UgMw8SnfmJ3I9+M2TdvoRL/Ovu6U8=
4. How I installed and ran Caddy:
Created Debian LXC on Proxmox and followed Debian steps at Install — Caddy Documentation . Ran it 2 ways.
a. System environment:
Debian 13 LXC on Proxmox 9
b. Command:
Direct run with command works perfectly:
/usr/bin/caddy run --environ --config /etc/caddy/caddy.json --envfile /etc/caddy/Caddy.env
c. Service/unit/compose file:
Running via systemd service doesn’t work and I did it as suggested in docs at Keep Caddy Running — Caddy Documentation
> cat /etc/systemd/system/caddy.service.d/override.conf
[Service]
EnvironmentFile=/etc/caddy/Caddy.env
ExecStart=
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/caddy.json
ExecReload=
ExecReload=/usr/bin/caddy reload --config /etc/caddy/caddy.json
d. My complete Caddy config:
{
"apps": {
"http": {
"servers": {
"srv0": {
"listen": [
":443"
],
"routes": [
{
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"handler": "reverse_proxy",
"upstreams": [
{
"dial": "ns1.pvepve.charmain.lan:5380"
}
]
}
]
}
]
}
],
"match": [
{
"host": [
"ns1.pvepve.charmain.com"
]
}
]
}
]
}
],
"match": [
{
"host": [
"*.pvepve.charmain.com"
]
}
],
"terminal": true
}
]
}
}
},
"tls": {
"automation": {
"policies": [
{
"issuers": [
{
"challenges": {
"dns": {
"provider": {
"api_token": "{env.CF_API_TOKEN}",
"name": "cloudflare"
}
}
},
"module": "acme"
}
],
"subjects": [
"*.pvepve.charmain.com"
]
}
]
}
}
}
}