I have an internal Bind9 DNS server as well as Cloudfare for DNS challenges. I’m trying to setup Caddy as a reverse proxy to handle TLS certificates for all our internal servers.
I have the DNS entries for a few of our servers changed to the ip of our Caddy instance (10.0.0.13) and have our Caddyfile setup, but I clearly don’t have it set correctly as curl returns nothing for our subdomains. What am I missing/doing wrong?
2. Error messages and/or full log output:
curl 10.0.0.10: Welcome to nginx!
curl nginx:
curl smx:
curl -vL smx:
Trying 10.0.0.13:80...
* Connected to smx (10.0.0.13) port 80 (#0)
> GET / HTTP/1.1
> Host: smx
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://smx/
< Server: Caddy
< Date: Tue, 30 Apr 2024 20:03:05 GMT
< Content-Length: 0
<
* Closing connection 0
* Clear auth, redirects to port from 80 to 443
* Issue another request to this URL: 'https://smx/'
* Trying 10.0.0.13:443...
* Connected to smx (10.0.0.13) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Unknown (21):
* TLSv1.3 (IN), TLS alert, internal error (592):
* error:0A000438:SSL routines::tlsv1 alert internal error
* Closing connection 1
curl: (35) error:0A000438:SSL routines::tlsv1 alert internal error
3. Caddy version: 2.7.6
4. How I installed and ran Caddy: systemd service from apt repository
a. System environment: Ubuntu 22.04, systemd
b. Command: Systemctl start caddy
d. My complete Caddy config:
{ debug
acme_dns cloudflare api-token
email ******
)
smx.ecfinternal.net {
reverse_proxy 10.0.0.7:3443 {
}
}
nginx.ecfinternal.net {
reverse_proxy 10.0.0.10:80 {
}
}
Logs:
: Apr 30 19:51:43 auth systemd[1]: Started Caddy.
Apr 30 19:51:43 auth caddy[1858]: {"level":"info","ts":1714506703.7118087,"msg":"serving initial configuration"}
Apr 30 19:51:43 auth caddy[1858]: {"level":"info","ts":1714506703.7135735,"logger":"tls.obtain","msg":"acquiring lock","identifier":"nginx.ecfinternal.net"}
Apr 30 19:51:43 auth caddy[1858]: {"level":"warn","ts":1714506703.7145426,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/var/lib/caddy/.local/share/caddy","instance":"531e7759-e154-46fd-a653-18750e0d82ee","try_again":1714593103.7145417,"try_again_in":86399.99999977}
Apr 30 19:51:43 auth caddy[1858]: {"level":"info","ts":1714506703.71458,"logger":"tls","msg":"finished cleaning storage units"}
Apr 30 19:51:43 auth caddy[1858]: {"level":"info","ts":1714506703.7157898,"logger":"tls.obtain","msg":"acquiring lock","identifier":"smx.ecfinternal.net"}
Apr 30 19:51:43 auth caddy[1858]: {"level":"info","ts":1714506703.7165918,"logger":"tls.obtain","msg":"lock acquired","identifier":"nginx.ecfinternal.net"}
Apr 30 19:51:43 auth caddy[1858]: {"level":"info","ts":1714506703.7167416,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"nginx.ecfinternal.net"}
Apr 30 19:51:43 auth caddy[1858]: {"level":"debug","ts":1714506703.7168398,"logger":"events","msg":"event","name":"cert_obtaining","id":"38f29ce1-371a-4c47-8637-c70436825be8","origin":"tls","data":{"identifier":"nginx.ecfinternal.net"}}
Apr 30 19:51:43 auth caddy[1858]: {"level":"debug","ts":1714506703.7171643,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
Apr 30 19:51:43 auth caddy[1858]: {"level":"info","ts":1714506703.7175317,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["nginx.ecfinternal.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"myemail@example.com"}
Apr 30 19:51:43 auth caddy[1858]: {"level":"info","ts":1714506703.7176096,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["nginx.ecfinternal.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"myemail@example.com"}
Apr 30 19:51:43 auth caddy[1858]: {"level":"info","ts":1714506703.7190003,"logger":"tls.obtain","msg":"lock acquired","identifier":"smx.ecfinternal.net"}
Apr 30 19:51:43 auth caddy[1858]: {"level":"info","ts":1714506703.7191534,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"smx.ecfinternal.net"}
Apr 30 19:51:43 auth caddy[1858]: {"level":"debug","ts":1714506703.719242,"logger":"events","msg":"event","name":"cert_obtaining","id":"0050e9d2-6070-4d35-ad3a-e24cf949d294","origin":"tls","data":{"identifier":"smx.ecfinternal.net"}}
Apr 30 19:51:43 auth caddy[1858]: {"level":"debug","ts":1714506703.719447,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
Apr 30 19:51:43 auth caddy[1858]: {"level":"info","ts":1714506703.719613,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["smx.ecfinternal.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"myemail@example.com"}
Apr 30 19:51:43 auth caddy[1858]: {"level":"info","ts":1714506703.7197278,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["smx.ecfinternal.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"myemail@example.com"}
Apr 30 19:51:43 auth caddy[1858]: {"level":"debug","ts":1714506703.9026036,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"GET","url":"https://acme-v02.api.letsencrypt.org/directory","headers":{"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["747"],"Content-Type":["application/json"],"Date":["Tue, 30 Apr 2024 19:51:43 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Apr 30 19:51:43 auth caddy[1858]: {"level":"debug","ts":1714506703.9491193,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Tue, 30 Apr 2024 19:51:43 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["O0afatDIw8HAhqcwsdD6zsdvtNBxg67T5tlbWZ0v7-QQ9IpY3EU"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Apr 30 19:51:43 auth caddy[1858]: {"level":"debug","ts":1714506703.9499054,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Tue, 30 Apr 2024 19:51:43 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["Y_7AIQuU7LXlha7ggUNqlPklCno0nKn6KIkUqvE8IZ7tv50kpiE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Apr 30 19:51:44 auth caddy[1858]: {"level":"debug","ts":1714506704.1833467,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1663302997"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["345"],"Content-Type":["application/json"],"Date":["Tue, 30 Apr 2024 19:51:44 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/1663302997/265477647987"],"Replay-Nonce":["O0afatDIMLN3LPMu4KHxE3M1HVoXfVMnWsBgc-iJH4a9_pXUAJk"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
Apr 30 19:51:44 auth caddy[1858]: {"level":"debug","ts":1714506704.235609,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/345060138147","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1663302997"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["803"],"Content-Type":["application/json"],"Date":["Tue, 30 Apr 2024 19:51:44 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["O0afatDI7z2a_msMg3p6tXgF8a11Q00bOnH_J2zk0sFZYf2uJQY"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Apr 30 19:51:44 auth caddy[1858]: {"level":"info","ts":1714506704.2357879,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"smx.ecfinternal.net","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Apr 30 19:51:44 auth caddy[1858]: {"level":"debug","ts":1714506704.2626612,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1663302997"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["347"],"Content-Type":["application/json"],"Date":["Tue, 30 Apr 2024 19:51:44 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/1663302997/265477648047"],"Replay-Nonce":["O0afatDI0kcDzfz9px5OQ924FdBuxJI9eEPDKVJ8ycoSb-uFBws"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
Apr 30 19:51:44 auth caddy[1858]: {"level":"debug","ts":1714506704.3129365,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/345060138247","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1663302997"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["805"],"Content-Type":["application/json"],"Date":["Tue, 30 Apr 2024 19:51:44 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["O0afatDIjOoUBvaHrm_dnyzEJ6KMZRmz8Uoz4i5wTIEJS7jSM4U"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Apr 30 19:51:44 auth caddy[1858]: {"level":"info","ts":1714506704.312996,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"nginx.ecfinternal.net","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Apr 30 19:51:44 auth caddy[1858]: {"level":"debug","ts":1714506704.7217104,"logger":"tls.issuance.acme.acme_client","msg":"waiting for solver before continuing","identifier":"smx.ecfinternal.net","challenge_type":"dns-01"}
Apr 30 19:51:44 auth caddy[1858]: {"level":"debug","ts":1714506704.84069,"logger":"tls.issuance.acme.acme_client","msg":"waiting for solver before continuing","identifier":"nginx.ecfinternal.net","challenge_type":"dns-01"}
Apr 30 19:53:44 auth caddy[1858]: {"level":"debug","ts":1714506824.9076054,"logger":"tls.issuance.acme.acme_client","msg":"done waiting for solver","identifier":"smx.ecfinternal.net","challenge_type":"dns-01"}
Apr 30 19:53:44 auth caddy[1858]: {"level":"debug","ts":1714506824.9812305,"logger":"tls.issuance.acme.acme_client","msg":"done waiting for solver","identifier":"nginx.ecfinternal.net","challenge_type":"dns-01"}
Apr 30 19:53:45 auth caddy[1858]: {"level":"debug","ts":1714506825.437513,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/345060138247","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1663302997"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["809"],"Content-Type":["application/json"],"Date":["Tue, 30 Apr 2024 19:53:45 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["O0afatDIhtne-vwY3i04s4D8geuQn1bioslSEVw6I-WZ75Dgan4"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Apr 30 19:53:45 auth caddy[1858]: {"level":"error","ts":1714506825.4376564,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"nginx.ecfinternal.net","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[nginx.ecfinternal.net] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-v02.api.letsencrypt.org/acme/order/1663302997/265477648047) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
Apr 30 19:53:45 auth caddy[1858]: {"level":"debug","ts":1714506825.4376724,"logger":"tls.obtain","msg":"trying issuer 2/2","issuer":"acme.zerossl.com-v2-DV90"}
Apr 30 19:53:45 auth caddy[1858]: {"level":"info","ts":1714506825.4378846,"logger":"tls.issuance.zerossl","msg":"waiting on internal rate limiter","identifiers":["nginx.ecfinternal.net"],"ca":"https://acme.zerossl.com/v2/DV90","account":"amendenhall@staff.gwi.net"}
Apr 30 19:53:45 auth caddy[1858]: {"level":"info","ts":1714506825.4378927,"logger":"tls.issuance.zerossl","msg":"done waiting on internal rate limiter","identifiers":["nginx.ecfinternal.net"],"ca":"https://acme.zerossl.com/v2/DV90","account":"amendenhall@staff.gwi.net"}
Apr 30 19:53:45 auth caddy[1858]: {"level":"debug","ts":1714506825.5379317,"logger":"tls.issuance.zerossl.acme_client","msg":"http request","method":"GET","url":"https://acme.zerossl.com/v2/DV90","headers":{"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Content-Length":["645"],"Content-Type":["application/json"],"Date":["Tue, 30 Apr 2024 19:53:45 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]},"status_code":200}
Apr 30 19:53:45 auth caddy[1858]: {"level":"debug","ts":1714506825.563805,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/345060138147","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1663302997"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["807"],"Content-Type":["application/json"],"Date":["Tue, 30 Apr 2024 19:53:45 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["O0afatDIymCFYruCCD6FWHktT4KqTO5HmPTD1rjib9BMQHS2buc"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Apr 30 19:53:45 auth caddy[1858]: {"level":"error","ts":1714506825.5638955,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"smx.ecfinternal.net","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[smx.ecfinternal.net] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-v02.api.letsencrypt.org/acme/order/1663302997/265477647987) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
Apr 30 19:53:45 auth caddy[1858]: {"level":"debug","ts":1714506825.5639098,"logger":"tls.obtain","msg":"trying issuer 2/2","issuer":"acme.zerossl.com-v2-DV90"}