Local HTTPS -- Using local ACME endpoint

Help
1

1. The problem I’m having:

I am trying to use Caddy for local HTTPS between my reverse proxy (frontend) and LAN server (backend). I am following this guide: Use Caddy for local HTTPS (TLS) between front-end reverse proxy and LAN hosts. The frontend is running Caddy’s internal ACME server. I want the backend to obtain a certificate from the frontend’s ACME server, but based on the logs that is not happening. It appears that the backend is using ZeroSSL instead for the certificate? When the frontend forwards the request to the backend I get the error: certificate signed by unknown authority.
(Note that I hardcoded all of the IP addresses for testing/development, but they will eventually be replaced with environment variables.)

2. Error messages and/or full log output:

Frontend Log File:

2023/05/30 20:13:38.543	info	http	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2023/05/30 20:13:38.543	info	http	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2023/05/30 20:13:38.544	info	tls.cache.maintenance	started background certificate maintenance	{"cache": "0x400028ed20"}
2023/05/30 20:13:38.611	warn	pki.ca.local	installing root certificate (you might be prompted for password)	{"path": "storage:pki/authorities/local/root.crt"}
2023/05/30 20:13:39.930	info	http	enabling HTTP/3 listener	{"addr": ":443"}
2023/05/30 20:13:39.930	info	tls	cleaning storage unit	{"description": "FileStorage:/var/lib/caddy"}
2023/05/30 20:13:39.931	info	tls	finished cleaning storage units
2023/05/30 20:13:39.931	debug	http	starting server loop	{"address": "[::]:443", "tls": true, "http3": true}
2023/05/30 20:13:39.931	info	http.log	server running	{"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2023/05/30 20:13:39.931	debug	http	starting server loop	{"address": "[::]:80", "tls": false, "http3": false}
2023/05/30 20:13:39.931	info	http.log	server running	{"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2023/05/30 20:13:39.931	info	http	enabling automatic TLS certificate management	{"domains": ["10.1.0.168", "*.inductor-test-jkunzler.link"]}
2023/05/30 20:13:39.932	info	tls.obtain	acquiring lock	{"identifier": "*.inductor-test-jkunzler.link"}
2023/05/30 20:13:39.932	info	tls.obtain	acquiring lock	{"identifier": "10.1.0.168"}
2023/05/30 20:13:39.935	info	tls.obtain	lock acquired	{"identifier": "10.1.0.168"}
2023/05/30 20:13:39.935	info	tls.obtain	obtaining certificate	{"identifier": "10.1.0.168"}
2023/05/30 20:13:39.935	debug	events	event	{"name": "cert_obtaining", "id": "247c3147-394f-47a9-9263-4e0e44a5dd68", "origin": "tls", "data": {"identifier":"10.1.0.168"}}
2023/05/30 20:13:39.935	info	tls.obtain	lock acquired	{"identifier": "*.inductor-test-jkunzler.link"}
2023/05/30 20:13:39.936	info	tls.obtain	obtaining certificate	{"identifier": "*.inductor-test-jkunzler.link"}
2023/05/30 20:13:39.936	debug	events	event	{"name": "cert_obtaining", "id": "228fd8a0-159a-43dc-b639-89fd256d4479", "origin": "tls", "data": {"identifier":"*.inductor-test-jkunzler.link"}}
2023/05/30 20:13:39.936	debug	tls.obtain	trying issuer 1/2	{"issuer": "acme-v02.api.letsencrypt.org-directory"}
2023/05/30 20:13:39.937	debug	tls.obtain	trying issuer 1/1	{"issuer": "local"}
2023/05/30 20:13:39.938	debug	pki.ca.local	using intermediate signer	{"serial": "155439818784745081847555145964792104664", "not_before": "2023-05-30 20:13:38 +0000 UTC", "not_after": "2023-06-06 20:13:38 +0000 UTC"}
2023/05/30 20:13:39.939	info	tls.obtain	certificate obtained successfully	{"identifier": "10.1.0.168"}
2023/05/30 20:13:39.939	debug	events	event	{"name": "cert_obtained", "id": "6943334d-e12d-4c8f-bb2d-c5e32d801592", "origin": "tls", "data": {"identifier":"10.1.0.168","issuers":"local","renewal":false,"storage_key":"10.1.0.168"}}
2023/05/30 20:13:39.939	info	tls.obtain	releasing lock	{"identifier": "10.1.0.168"}
2023/05/30 20:13:39.939	warn	tls	stapling OCSP	{"error": "no OCSP stapling for [10.1.0.168]: no OCSP server specified in certificate", "identifiers": ["10.1.0.168"]}
2023/05/30 20:13:39.939	debug	tls.cache	added certificate to cache	{"subjects": ["10.1.0.168"], "expiration": "2023/05/31 08:13:40.000", "managed": true, "issuer_key": "local", "hash": "fe77c781a2d615c6fe92161a47ab07d4dae3677f54c37735d9008b5681eb1d31", "cache_size": 1, "cache_capacity": 10000}
2023/05/30 20:13:39.939	debug	events	event	{"name": "cached_managed_cert", "id": "0c883074-1c22-4c87-a0d7-3e690bd7c846", "origin": "tls", "data": {"sans":["10.1.0.168"]}}
2023/05/30 20:13:40.079	debug	http.acme_client	http request	{"method": "GET", "url": "https://acme-v02.api.letsencrypt.org/directory", "headers": {"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; arm64)"]}, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["752"],"Content-Type":["application/json"],"Date":["Tue, 30 May 2023 20:13:40 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2023/05/30 20:13:40.125	debug	http.acme_client	http request	{"method": "HEAD", "url": "https://acme-v02.api.letsencrypt.org/acme/new-nonce", "headers": {"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; arm64)"]}, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Date":["Tue, 30 May 2023 20:13:40 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["15C9ecNwoXKB7ID5mrB0nJN09kukJ6IJh62jmKC1NbcSaF8"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2023/05/30 20:13:40.226	debug	http.acme_client	http request	{"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/new-acct", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; arm64)"]}, "response_headers": {"Boulder-Requester":["1135429557"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["266"],"Content-Type":["application/json"],"Date":["Tue, 30 May 2023 20:13:40 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf>;rel=\"terms-of-service\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/acct/1135429557"],"Replay-Nonce":["1AADfkoR__8bqmlhD1ruE1nKP7i8r7MQiaJ6xkvzjfRkyLU"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 201}
2023/05/30 20:13:40.227	info	http	waiting on internal rate limiter	{"identifiers": ["*.inductor-test-jkunzler.link"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2023/05/30 20:13:40.227	info	http	done waiting on internal rate limiter	{"identifiers": ["*.inductor-test-jkunzler.link"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2023/05/30 20:13:40.292	debug	http.acme_client	http request	{"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/new-order", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; arm64)"]}, "response_headers": {"Boulder-Requester":["1135429557"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["335"],"Content-Type":["application/problem+json"],"Date":["Tue, 30 May 2023 20:13:40 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://letsencrypt.org/docs/rate-limits>;rel=\"help\""],"Replay-Nonce":["1AADfbZ6yVrw4_RHzrrICSk33J8V1odLhVY5aJYro8nRM90"],"Retry-After":["117212"],"Server":["nginx"]}, "status_code": 429}
2023/05/30 20:13:40.292	error	tls.obtain	could not get certificate from issuer	{"identifier": "*.inductor-test-jkunzler.link", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: *.inductor-test-jkunzler.link, retry after 2023-06-01T04:47:12Z: see https://letsencrypt.org/docs/duplicate-certificate-limit/"}
2023/05/30 20:13:40.292	debug	tls.obtain	trying issuer 2/2	{"issuer": "acme.zerossl.com-v2-DV90"}
2023/05/30 20:13:40.293	warn	tls.issuance.zerossl	missing email address for ZeroSSL; it is strongly recommended to set one for next time
2023/05/30 20:13:40.678	info	tls.issuance.zerossl	generated EAB credentials	{"key_id": "zL2FxpZiYLukn-Qnt5_kMw"}
2023/05/30 20:13:40.722	debug	http.acme_client	http request	{"method": "GET", "url": "https://acme.zerossl.com/v2/DV90", "headers": {"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; arm64)"]}, "response_headers": {"Access-Control-Allow-Origin":["*"],"Content-Length":["645"],"Content-Type":["application/json"],"Date":["Tue, 30 May 2023 20:13:40 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]}, "status_code": 200}
2023/05/30 20:13:41.284	debug	http.acme_client	http request	{"method": "HEAD", "url": "https://acme.zerossl.com/v2/DV90/newNonce", "headers": {"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; arm64)"]}, "response_headers": {"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Type":["application/octet-stream"],"Date":["Tue, 30 May 2023 20:13:41 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["DFYdl-D93IJCYOEoyAttZYlY1Y12vSi8d1IIFPdPg7M"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]}, "status_code": 200}
2023/05/30 20:13:41.695	debug	http.acme_client	http request	{"method": "POST", "url": "https://acme.zerossl.com/v2/DV90/newAccount", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; arm64)"]}, "response_headers": {"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["579"],"Content-Type":["application/json"],"Date":["Tue, 30 May 2023 20:13:41 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Location":["https://acme.zerossl.com/v2/DV90/account/zL2FxpZiYLukn-Qnt5_kMw"],"Replay-Nonce":["7Sj787J1KCXciJgrslWLbRfYfamP6SCPxwBz9QYUlcM"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]}, "status_code": 201}
2023/05/30 20:13:41.695	info	http	waiting on internal rate limiter	{"identifiers": ["*.inductor-test-jkunzler.link"], "ca": "https://acme.zerossl.com/v2/DV90", "account": ""}
2023/05/30 20:13:41.695	info	http	done waiting on internal rate limiter	{"identifiers": ["*.inductor-test-jkunzler.link"], "ca": "https://acme.zerossl.com/v2/DV90", "account": ""}
2023/05/30 20:13:42.101	debug	http.acme_client	http request	{"method": "POST", "url": "https://acme.zerossl.com/v2/DV90/newOrder", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; arm64)"]}, "response_headers": {"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["291"],"Content-Type":["application/json"],"Date":["Tue, 30 May 2023 20:13:42 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/W5t3x_6YdjJr98_FsJivBQ"],"Replay-Nonce":["YW4Rlc0Pce30k3pHtffusvxWJ0hmAYEJKoH84h0AHKs"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]}, "status_code": 201}
2023/05/30 20:13:42.459	debug	http.acme_client	http request	{"method": "POST", "url": "https://acme.zerossl.com/v2/DV90/authz/QfEXcRIlIwcXs4bmrC3EuA", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; arm64)"]}, "response_headers": {"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["311"],"Content-Type":["application/json"],"Date":["Tue, 30 May 2023 20:13:42 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["PcC-f0ZQU_Dsep3ACNAC6rrEjKLJk-EeMFsgAbbBO4M"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]}, "status_code": 200}
2023/05/30 20:13:42.460	info	http.acme_client	trying to solve challenge	{"identifier": "*.inductor-test-jkunzler.link", "challenge_type": "dns-01", "ca": "https://acme.zerossl.com/v2/DV90"}
2023/05/30 20:13:42.886	debug	http.acme_client	waiting for solver before continuing	{"identifier": "*.inductor-test-jkunzler.link", "challenge_type": "dns-01"}
2023/05/30 20:13:57.055	debug	http.acme_client	done waiting for solver	{"identifier": "*.inductor-test-jkunzler.link", "challenge_type": "dns-01"}
2023/05/30 20:13:57.551	debug	http.acme_client	http request	{"method": "POST", "url": "https://acme.zerossl.com/v2/DV90/chall/bGkD7Qb60GuWqH4gvk3Jaw", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; arm64)"]}, "response_headers": {"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["163"],"Content-Type":["application/json"],"Date":["Tue, 30 May 2023 20:13:57 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90/authz/QfEXcRIlIwcXs4bmrC3EuA>;rel=\"up\""],"Replay-Nonce":["8EXjKcGh1ghffHue7vl_vQAuRELunUv-KP_V4xFKGXI"],"Retry-After":["10"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]}, "status_code": 200}
2023/05/30 20:13:57.551	debug	http.acme_client	challenge accepted	{"identifier": "*.inductor-test-jkunzler.link", "challenge_type": "dns-01"}
2023/05/30 20:13:58.196	debug	http.acme_client	http request	{"method": "POST", "url": "https://acme.zerossl.com/v2/DV90/authz/QfEXcRIlIwcXs4bmrC3EuA", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; arm64)"]}, "response_headers": {"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["314"],"Content-Type":["application/json"],"Date":["Tue, 30 May 2023 20:13:58 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["AVbWdAiK2Q8NNmp1zvNlVVmwaw90bwehHvAiJkJhqKQ"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]}, "status_code": 200}
2023/05/30 20:14:04.051	debug	http.acme_client	http request	{"method": "POST", "url": "https://acme.zerossl.com/v2/DV90/authz/QfEXcRIlIwcXs4bmrC3EuA", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; arm64)"]}, "response_headers": {"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["342"],"Content-Type":["application/json"],"Date":["Tue, 30 May 2023 20:14:04 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["23hkm6ZNEJledkzatECUpW-2shIjLgQLNfW_wvpphNg"],"Retry-After":["300"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]}, "status_code": 200}
2023/05/30 20:14:04.479	info	http.acme_client	authorization finalized	{"identifier": "*.inductor-test-jkunzler.link", "authz_status": "valid"}
2023/05/30 20:14:04.479	info	http.acme_client	validations succeeded; finalizing order	{"order": "https://acme.zerossl.com/v2/DV90/order/W5t3x_6YdjJr98_FsJivBQ"}
2023/05/30 20:14:04.915	debug	http.acme_client	http request	{"method": "POST", "url": "https://acme.zerossl.com/v2/DV90/order/W5t3x_6YdjJr98_FsJivBQ/finalize", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; arm64)"]}, "response_headers": {"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["294"],"Content-Type":["application/json"],"Date":["Tue, 30 May 2023 20:14:04 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/W5t3x_6YdjJr98_FsJivBQ"],"Replay-Nonce":["TC6COUYWzLyzRghZmCELbFt062AOm3BDeA-NrIGoeY0"],"Retry-After":["15"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]}, "status_code": 200}
2023/05/30 20:14:20.310	debug	http.acme_client	http request	{"method": "POST", "url": "https://acme.zerossl.com/v2/DV90/order/W5t3x_6YdjJr98_FsJivBQ", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; arm64)"]}, "response_headers": {"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["366"],"Content-Type":["application/json"],"Date":["Tue, 30 May 2023 20:14:20 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/W5t3x_6YdjJr98_FsJivBQ"],"Replay-Nonce":["RMLuyCO23BQpXuQW2fmvPHGVSvUW4t0tMj9IPHGMsPA"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]}, "status_code": 200}
2023/05/30 20:14:20.693	debug	http.acme_client	http request	{"method": "POST", "url": "https://acme.zerossl.com/v2/DV90/cert/e9_2mqDYrudsgRQSGhzMNw", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; arm64)"]}, "response_headers": {"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["4152"],"Content-Type":["application/pem-certificate-chain"],"Date":["Tue, 30 May 2023 20:14:20 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["KntWrvTcdHaKWt66R9TiNxRGWGZAVT4GJ0VRvicE3v8"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]}, "status_code": 200}
2023/05/30 20:14:20.693	info	http.acme_client	successfully downloaded available certificate chains	{"count": 1, "first_url": "https://acme.zerossl.com/v2/DV90/cert/e9_2mqDYrudsgRQSGhzMNw"}
2023/05/30 20:14:20.693	info	tls.obtain	certificate obtained successfully	{"identifier": "*.inductor-test-jkunzler.link"}
2023/05/30 20:14:20.694	debug	events	event	{"name": "cert_obtained", "id": "3137eddc-b4f9-4920-8cd9-198ad70b3dae", "origin": "tls", "data": {"identifier":"*.inductor-test-jkunzler.link","issuers":"acme.zerossl.com-v2-DV90","renewal":false,"storage_key":"*.inductor-test-jkunzler.link"}}
2023/05/30 20:14:20.694	info	tls.obtain	releasing lock	{"identifier": "*.inductor-test-jkunzler.link"}
2023/05/30 20:14:20.694	debug	tls	loading managed certificate	{"domain": "*.inductor-test-jkunzler.link", "expiration": "2023/08/29 00:00:00.000", "issuer_key": "acme.zerossl.com-v2-DV90", "storage": "FileStorage:/var/lib/caddy"}
2023/05/30 20:14:20.727	debug	tls.cache	added certificate to cache	{"subjects": ["*.inductor-test-jkunzler.link"], "expiration": "2023/08/29 00:00:00.000", "managed": true, "issuer_key": "acme.zerossl.com-v2-DV90", "hash": "f9d530855114f9a24c4fa23ab23ed0e182dee54b91e0eadaff4ed24233f68887", "cache_size": 2, "cache_capacity": 10000}
2023/05/30 20:14:20.727	debug	events	event	{"name": "cached_managed_cert", "id": "1e4617a3-bd59-4ee7-b7b1-1a07cbee818c", "origin": "tls", "data": {"sans":["*.inductor-test-jkunzler.link"]}}
2023/05/30 20:19:18.302	debug	events	event	{"name": "tls_get_certificate", "id": "22c4f60a-4fe8-42c0-ad7b-6d3b4496e8ce", "origin": "tls", "data": {"client_hello":{"CipherSuites":[14906,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"a.inductor-test-jkunzler.link","SupportedCurves":[39578,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[51914,772,771],"Conn":{}}}}
2023/05/30 20:19:18.302	debug	tls.handshake	no matching certificates and no custom selection logic	{"identifier": "a.inductor-test-jkunzler.link"}
2023/05/30 20:19:18.302	debug	tls.handshake	choosing certificate	{"identifier": "*.inductor-test-jkunzler.link", "num_choices": 1}
2023/05/30 20:19:18.302	debug	tls.handshake	default certificate selection results	{"identifier": "*.inductor-test-jkunzler.link", "subjects": ["*.inductor-test-jkunzler.link"], "managed": true, "issuer_key": "acme.zerossl.com-v2-DV90", "hash": "f9d530855114f9a24c4fa23ab23ed0e182dee54b91e0eadaff4ed24233f68887"}
2023/05/30 20:19:18.302	debug	tls.handshake	matched certificate in cache	{"remote_ip": "71.212.33.44", "remote_port": "61360", "subjects": ["*.inductor-test-jkunzler.link"], "managed": true, "expiration": "2023/08/29 00:00:00.000", "hash": "f9d530855114f9a24c4fa23ab23ed0e182dee54b91e0eadaff4ed24233f68887"}
2023/05/30 20:19:18.384	debug	http.handlers.reverse_proxy	selected upstream	{"dial": "10.1.0.185:443", "total_upstreams": 1}
2023/05/30 20:19:18.388	debug	http.handlers.reverse_proxy	upstream roundtrip	{"upstream": "{http.vars.backend_hostport}", "duration": 0.00308678, "request": {"remote_ip": "71.212.33.44", "remote_port": "61360", "proto": "HTTP/2.0", "method": "GET", "host": "10.1.0.185:443", "uri": "/", "headers": {"Accept-Language": ["en-US,en;q=0.5"], "Accept-Encoding": ["gzip, deflate, br"], "Sec-Fetch-Mode": ["navigate"], "Sec-Fetch-Dest": ["document"], "Sec-Ch-Ua": ["\"Chromium\";v=\"112\", \"Brave\";v=\"112\", \"Not:A-Brand\";v=\"99\""], "Sec-Ch-Ua-Mobile": ["?0"], "Sec-Fetch-User": ["?1"], "X-Forwarded-Host": ["a.inductor-test-jkunzler.link"], "Sec-Fetch-Site": ["none"], "Upgrade-Insecure-Requests": ["1"], "Sec-Ch-Ua-Platform": ["\"macOS\""], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"], "Cache-Control": ["max-age=0"], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36"], "Sec-Gpc": ["1"], "X-Forwarded-For": ["71.212.33.44"], "X-Forwarded-Proto": ["https"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "a.inductor-test-jkunzler.link"}}, "error": "x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"Caddy Local Authority - 2023 ECC Root\")"}
2023/05/30 20:19:18.388	debug	http.log.error	x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "Caddy Local Authority - 2023 ECC Root")	{"request": {"remote_ip": "71.212.33.44", "remote_port": "61360", "proto": "HTTP/2.0", "method": "GET", "host": "a.inductor-test-jkunzler.link", "uri": "/", "headers": {"Cache-Control": ["max-age=0"], "Sec-Ch-Ua-Platform": ["\"macOS\""], "Upgrade-Insecure-Requests": ["1"], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36"], "Sec-Ch-Ua": ["\"Chromium\";v=\"112\", \"Brave\";v=\"112\", \"Not:A-Brand\";v=\"99\""], "Sec-Ch-Ua-Mobile": ["?0"], "Sec-Gpc": ["1"], "Sec-Fetch-Mode": ["navigate"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"], "Accept-Language": ["en-US,en;q=0.5"], "Sec-Fetch-Site": ["none"], "Sec-Fetch-User": ["?1"], "Sec-Fetch-Dest": ["document"], "Accept-Encoding": ["gzip, deflate, br"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "a.inductor-test-jkunzler.link"}}, "duration": 0.003466311, "status": 502, "err_id": "e30z6t8gp", "err_trace": "reverseproxy.statusError (reverseproxy.go:1299)"}

Backend Log File:

2023/05/30 20:18:01.153	info	http	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2023/05/30 20:18:01.154	info	http	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2023/05/30 20:18:01.158	debug	http	starting server loop	{"address": "[::]:80", "tls": false, "http3": false}
2023/05/30 20:18:01.154	info	tls.cache.maintenance	started background certificate maintenance	{"cache": "0x40002fb490"}
2023/05/30 20:18:01.158	info	http.log	server running	{"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2023/05/30 20:18:01.158	info	http	enabling HTTP/3 listener	{"addr": ":443"}
2023/05/30 20:18:01.159	debug	http	starting server loop	{"address": "[::]:443", "tls": true, "http3": true}
2023/05/30 20:18:01.159	info	http.log	server running	{"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2023/05/30 20:18:01.159	info	http	enabling automatic TLS certificate management	{"domains": ["10.1.0.185"]}
2023/05/30 20:18:01.164	info	tls	cleaning storage unit	{"description": "FileStorage:/var/lib/caddy"}
2023/05/30 20:18:01.165	info	tls	finished cleaning storage units
2023/05/30 20:18:01.165	info	tls.obtain	acquiring lock	{"identifier": "10.1.0.185"}
2023/05/30 20:18:01.172	info	tls.obtain	lock acquired	{"identifier": "10.1.0.185"}
2023/05/30 20:18:01.175	info	tls.obtain	obtaining certificate	{"identifier": "10.1.0.185"}
2023/05/30 20:18:01.179	debug	events	event	{"name": "cert_obtaining", "id": "36d97985-3164-4603-9d73-ba7f13afc44c", "origin": "tls", "data": {"identifier":"10.1.0.185"}}
2023/05/30 20:18:01.179	debug	tls.obtain	trying issuer 1/1	{"issuer": "local"}
2023/05/30 20:18:01.180	debug	pki.ca.local	using intermediate signer	{"serial": "66231171514396345981548353251188727485", "not_before": "2023-05-30 20:18:01 +0000 UTC", "not_after": "2023-06-06 20:18:01 +0000 UTC"}
2023/05/30 20:18:01.181	info	tls.obtain	certificate obtained successfully	{"identifier": "10.1.0.185"}
2023/05/30 20:18:01.181	debug	events	event	{"name": "cert_obtained", "id": "4634c8f2-e0bb-4d5e-a097-79dc0b263237", "origin": "tls", "data": {"identifier":"10.1.0.185","issuers":"local","renewal":false,"storage_key":"10.1.0.185"}}
2023/05/30 20:18:01.181	info	tls.obtain	releasing lock	{"identifier": "10.1.0.185"}
2023/05/30 20:18:01.182	warn	tls	stapling OCSP	{"error": "no OCSP stapling for [10.1.0.185]: no OCSP server specified in certificate", "identifiers": ["10.1.0.185"]}
2023/05/30 20:18:01.182	debug	tls.cache	added certificate to cache	{"subjects": ["10.1.0.185"], "expiration": "2023/05/31 08:18:02.000", "managed": true, "issuer_key": "local", "hash": "1867886ed5f0f0b8f73724f0c56266b8a3a8d2301d642c51c26465125bc4b3a2", "cache_size": 1, "cache_capacity": 10000}
2023/05/30 20:18:01.182	debug	events	event	{"name": "cached_managed_cert", "id": "e60a7dbd-40a7-4602-b081-0a37bc9b0a2d", "origin": "tls", "data": {"sans":["10.1.0.185"]}}
2023/05/30 20:18:01.198	warn	pki.ca.local	installing root certificate (you might be prompted for password)	{"path": "storage:pki/authorities/local/root.crt"}
2023/05/30 20:19:18.384	debug	events	event	{"name": "tls_get_certificate", "id": "2b74c1c5-ec91-44c8-86c3-00fb75d9889c", "origin": "tls", "data": {"client_hello":{"CipherSuites":[49195,49199,49196,49200,52393,52392,49161,49171,49162,49172,156,157,47,53,49170,10,4865,4866,4867],"ServerName":"","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2052,1027,2055,2053,2054,1025,1281,1537,1283,1539,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"Conn":{}}}}
2023/05/30 20:19:18.384	debug	tls.handshake	choosing certificate	{"identifier": "10.1.0.185", "num_choices": 1}
2023/05/30 20:19:18.384	debug	tls.handshake	default certificate selection results	{"identifier": "10.1.0.185", "subjects": ["10.1.0.185"], "managed": true, "issuer_key": "local", "hash": "1867886ed5f0f0b8f73724f0c56266b8a3a8d2301d642c51c26465125bc4b3a2"}
2023/05/30 20:19:18.384	debug	tls.handshake	matched certificate in cache	{"remote_ip": "10.1.0.168", "remote_port": "47898", "subjects": ["10.1.0.185"], "managed": true, "expiration": "2023/05/31 08:18:02.000", "hash": "1867886ed5f0f0b8f73724f0c56266b8a3a8d2301d642c51c26465125bc4b3a2"}
2023/05/30 20:19:18.386	debug	http.stdlib	http: TLS handshake error from 10.1.0.168:47898: remote error: tls: bad certificate

3. Caddy version:

Frontend Caddy Version:
v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=
Backend Caddy Version:
v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=

4. How I installed and ran Caddy:

a. System environment:

Amazon Linux 2023

b. Command:

Frontend:

go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest
xcaddy build --with github.com/caddy-dns/route53

Backend:

dnf -y install 'dnf-command(copr)'
dnf -y copr enable @caddy/caddy epel-9-x86_64
dnf -y install caddy

I additionally copied the contents of /var/lib/caddy/pki/authorities/local/root.crt on the frontend to /etc/ssl/certs/root.crt on the backend.

d. My complete Caddy config:

Frontend Caddy File:

{
	debug
	admin :2019
	storage file_system {
		root /var/lib/caddy
	}
	log {
		output stdout
	}
	log file_logger {
		output file caddy.log
	}
}

# ACME Server
10.1.0.168 {
	# defining FQDN for ACME server
	acme_server # defining the ACME server
	tls internal
}

*.inductor-test-jkunzler.link {
	map {host} {backend_host} {
		a.inductor-test-jkunzler.link 10.1.0.185
		default unknown
	}
	@unknown expression `{backend_host} == "unknown"`
	handle @unknown {
		respond "App not found." 404
	}
	vars backend_hostport {backend_host}:443

	handle_errors {
		respond "Internal Error." 404
	}

	reverse_proxy {
		to {vars.backend_hostport}
		header_up Host {upstream_hostport}
		transport http {
			tls
		}
	}

	tls {
		dns route53
	}
}

Backend Caddy File:

{
	# General Option
	debug

	# TLS Options
	acme_ca https://10.1.0.168/acme/local/directory # point to ACME server
	acme_ca_root /etc/ssl/certs/root.crt  # define root certificate

    storage file_system {
		root /var/lib/caddy
	}

	log {
		output stdout
	}
	log file_logger {
		output file caddy.log
	}
}

10.1.0.185 {
    reverse_proxy :8000
}

5. Links to relevant resources:

I am following this guide: Use Caddy for local HTTPS (TLS) between front-end reverse proxy and LAN hosts

2

After removing the global option acme_ca and instead using the tls directive, the backend’s acme client is correctly pointed to the frontend’s acme server.

10.1.0.185 {
	reverse_proxy :8000

	tls {
		ca  https://10.1.0.168/acme/local/directory
		ca_root /home/app_agent/root.crt
	}
}

I am not sure why the global directive did not work.

3

Hmm, weird. If you run caddy adapt --pretty --config <your-Caddyfile> you’ll see the problem – your first config doesn’t have any tls config in the JSON output at all, but the second does. If you use an actual domain instead of an IP address with acme_ca it does have tls config in the JSON output.

I think we have some logic in the Caddyfile adapter that skips IP address sites as not being eligible for managed TLS, so it ends up with no config. I’m not sure if there’s a valid fix for that though. It’s correct that we ignore IP addresses because they aren’t eligible for public domains, but it is a confusing config result. /cc @matt if you have thoughts

1 Like
4

Global options are hard and I still sometimes regret implementing them :frowning:

I’ll try to figure out a fix soon.

5

Okie, so this is tricky, because we can’t know if the CA specified in the global options is an internal one or a publicly-trusted one.

The Caddyfile adapter assumes it’s a public one since internal CAs aren’t very common. So yes, @francislavoie is right: the Caddyfile adapter sees the IP 10.1.0.185 and treats it as an internal hostname (which it is), and does not set up the ACME issuer because it probably can only be handled by the internal authority, right? In other words, the acme_* global options only apply to public hostnames.

I would like for it to work for internal ones too but unfortunately it’s ambiguous as-is. Is the ACME CA capable of providing a cert for your IP address? Maybe. Probably not. What about your localhost site (hypothetical but very real)? Almost certainly not.

So anyway, to sum up:

  • We don’t know whether the CA in the global options is intended for internal hostnames too. Most aren’t.
  • I’m inclined to leave this be for now, and add a note to our docs that the global acme_* options are for public hostnames, because internally-managed ACME servers are uncommon. (This allows you to specify the global option for all your public sites and still specify localhost and other internal names/IPs without having to override it.)
  • In hindsight, I probably should have had public_acme_ca and private_acme_ca options… and maybe we can still do that, where acme_ca is applied to all sites regardless of whether they are public or private identifiers.

I want to hold out on changing the docs until we decide whether current behavior is correct or a bug. :thinking:

2 Likes
6

I guess auto-https could warn if it skips a hostname for the purposes of hostname eligibility

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.