1. The problem I’m having:
Setup:
I am running a Caddy Server on an EC2 instance to function as a reverse proxy. The mappings for the reverse proxy (which map subdomains to backend services) are set by another service using the Caddy REST API. The Caddy Server also functions as an ACME server for local HTTPS between itself (the reverse proxy) and the backend services.
Issue:
When I initially start my Caddy Server and send a request to the server (to be forwarded), I get the error: x509: certificate signed by unknown authority. However, just rebooting the server (with caddy stop and then caddy start --config caddy.json) resolves the issue and I am able to send a request without error.
This is a consistent problem when starting my Caddy Server on a new EC2 instance. The temporary workaround for me is just to adjust my EC2 start script to include:
./caddy start --config caddy.json
./caddy stop
./caddy start --config caddy.json
However, I would like to figure out the root cause of this issue.
2. Error messages and/or full log output:
Logs (truncated due to character limits) after running ./caddy start --config caddy.json and making a request to the server (that fails).
2023/07/14 18:55:48.342 info tls.cache.maintenance started background certificate maintenance {"cache": "0x400045f490"}
2023/07/14 18:55:48.345 info http server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2023/07/14 18:55:48.345 info http enabling automatic HTTP->HTTPS redirects {"server_name": "srv0"}
2023/07/14 18:55:48.382 info http enabling HTTP/3 listener {"addr": ":443"}
2023/07/14 18:55:48.383 debug http starting server loop {"address": "[::]:443", "tls": true, "http3": true}
2023/07/14 18:55:48.383 info http.log server running {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2023/07/14 18:55:48.383 debug http starting server loop {"address": "[::]:80", "tls": false, "http3": false}
2023/07/14 18:55:48.383 info http.log server running {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2023/07/14 18:55:48.383 info http enabling automatic TLS certificate management {"domains": ["10.1.0.10", "*.inductor-test-jkunzler.link"]}
2023/07/14 18:55:48.384 info tls.obtain acquiring lock {"identifier": "10.1.0.10"}
2023/07/14 18:55:48.385 info tls.obtain acquiring lock {"identifier": "*.inductor-test-jkunzler.link"}
2023/07/14 18:55:48.386 info tls.obtain lock acquired {"identifier": "10.1.0.10"}
2023/07/14 18:55:48.386 info tls.obtain obtaining certificate {"identifier": "10.1.0.10"}
2023/07/14 18:55:48.386 debug events event {"name": "cert_obtaining", "id": "5ef3f0e7-7d38-4209-a038-0728b1d079e3", "origin": "tls", "data": {"identifier":"10.1.0.10"}}
2023/07/14 18:55:48.386 debug tls.obtain trying issuer 1/1 {"issuer": "local"}
2023/07/14 18:55:48.387 debug pki.ca.local using intermediate signer {"serial": "133945864164094359504464780601471048988", "not_before": "2023-07-14 18:55:48 +0000 UTC", "not_after": "2023-07-21 18:55:48 +0000 UTC"}
2023/07/14 18:55:48.388 info tls.obtain certificate obtained successfully {"identifier": "10.1.0.10"}
2023/07/14 18:55:48.389 debug events event {"name": "cert_obtained", "id": "3b7a3f4d-21c5-461e-bc72-f2a71c0a5fa9", "origin": "tls", "data": {"identifier":"10.1.0.10","issuers":"local","renewal":false,"storage_key":"10.1.0.10"}}
2023/07/14 18:55:48.389 info tls.obtain releasing lock {"identifier": "10.1.0.10"}
2023/07/14 18:55:48.390 warn tls stapling OCSP {"error": "no OCSP stapling for [10.1.0.10]: no OCSP server specified in certificate", "identifiers": ["10.1.0.10"]}
2023/07/14 18:55:48.390 debug tls.cache added certificate to cache {"subjects": ["10.1.0.10"], "expiration": "2023/07/15 06:55:49.000", "managed": true, "issuer_key": "local", "hash": "e12ed862c8b9ccb0dd988efedf741e1a5fd117f46c73b7d0b7a02dfb4af9dfd2", "cache_size": 1, "cache_capacity": 10000}
2023/07/14 18:55:48.390 debug events event {"name": "cached_managed_cert", "id": "be94d3e6-c791-4e6c-88ee-c4613fdaedd3", "origin": "tls", "data": {"sans":["10.1.0.10"]}}
2023/07/14 18:55:48.391 info tls.obtain lock acquired {"identifier": "*.inductor-test-jkunzler.link"}
2023/07/14 18:55:48.391 info tls.obtain obtaining certificate {"identifier": "*.inductor-test-jkunzler.link"}
2023/07/14 18:55:48.391 debug events event {"name": "cert_obtaining", "id": "60f1d7df-f0ee-424b-8a83-9e0417edb1c9", "origin": "tls", "data": {"identifier":"*.inductor-test-jkunzler.link"}}
2023/07/14 18:55:48.391 debug tls.obtain trying issuer 1/2 {"issuer": "acme-staging-v02.api.letsencrypt.org-directory"}
2023/07/14 18:55:48.411 warn pki.ca.local installing root certificate (you might be prompted for password) {"path": "storage:pki/authorities/local/root.crt"}
2023/07/14 18:55:48.565 debug http.acme_client http request {"method": "GET", "url": "https://acme-staging-v02.api.letsencrypt.org/directory", "headers": {"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; arm64)"]}, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["826"],"Content-Type":["application/json"],"Date":["Fri, 14 Jul 2023 18:55:48 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2023/07/14 18:55:48.620 debug http.acme_client http request {"method": "HEAD", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce", "headers": {"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; arm64)"]}, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Date":["Fri, 14 Jul 2023 18:55:48 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["BEB90u28Lt8NxeflW0H023D2bb6v5m5MarpjeIu9RTOXIGU"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2023/07/14 18:55:48.689 debug http.acme_client http request {"method": "POST", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/new-acct", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; arm64)"]}, "response_headers": {"Boulder-Requester":["110940734"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["267"],"Content-Type":["application/json"],"Date":["Fri, 14 Jul 2023 18:55:48 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf>;rel=\"terms-of-service\""],"Location":["https://acme-staging-v02.api.letsencrypt.org/acme/acct/110940734"],"Replay-Nonce":["BEB9JYe21Q4NohErKSuU4kMS-1ICmv1eAfidRnZ8AiNCEzw"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 201}
2023/07/14 18:55:48.690 info http waiting on internal rate limiter {"identifiers": ["*.inductor-test-jkunzler.link"], "ca": "https://acme-staging-v02.api.letsencrypt.org/directory", "account": ""}
2023/07/14 18:55:48.690 info http done waiting on internal rate limiter {"identifiers": ["*.inductor-test-jkunzler.link"], "ca": "https://acme-staging-v02.api.letsencrypt.org/directory", "account": ""}
…
2023/07/14 18:57:44.457 debug http.log.error x509: certificate signed by unknown authority {"request": {"remote_ip": "174.21.143.134", "remote_port": "60278", "proto": "HTTP/2.0", "method": "GET", "host": "app-1-8.inductor-test-jkunzler.link", "uri": "/favicon.ico", "headers": {"Sec-Ch-Ua-Platform": ["\"macOS\""], "Sec-Fetch-Site": ["same-origin"], "Sec-Fetch-Mode": ["no-cors"], "Accept-Encoding": ["gzip, deflate, br"], "Sec-Ch-Ua": ["\"Not.A/Brand\";v=\"8\", \"Chromium\";v=\"114\", \"Brave\";v=\"114\""], "Sec-Ch-Ua-Mobile": ["?0"], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"], "Sec-Fetch-Dest": ["image"], "Referer": ["https://app-1-8.inductor-test-jkunzler.link/"], "Accept": ["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"], "Sec-Gpc": ["1"], "Accept-Language": ["en-US,en;q=0.6"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "app-1-8.inductor-test-jkunzler.link"}}, "duration": 0.016727965, "status": 502, "err_id": "tajubzwc4", "err_trace": "reverseproxy.statusError (reverseproxy.go:1299)"}
A continuation of the logs after running caddy stop, a second ./caddy start --config caddy.json, and a second request to the server (that is successful):
2023/07/14 18:58:37.338 debug events event {"name": "tls_get_certificate", "id": "627eb7fa-8e9b-4e08-a070-0176f011a42b", "origin": "tls", "data": {"client_hello":{"CipherSuites":[49195,49199,49196,49200,52393,52392,49161,49171,49162,49172,156,157,47,53,49170,10,4865,4866,4867],"ServerName":"","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2052,1027,2055,2053,2054,1025,1281,1537,1283,1539,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"Conn":{}}}}
2023/07/14 18:58:37.338 debug tls.handshake choosing certificate {"identifier": "10.1.0.10", "num_choices": 1}
2023/07/14 18:58:37.338 debug tls.handshake default certificate selection results {"identifier": "10.1.0.10", "subjects": ["10.1.0.10"], "managed": true, "issuer_key": "local", "hash": "e12ed862c8b9ccb0dd988efedf741e1a5fd117f46c73b7d0b7a02dfb4af9dfd2"}
2023/07/14 18:58:37.338 debug tls.handshake matched certificate in cache {"remote_ip": "10.1.0.51", "remote_port": "57794", "subjects": ["10.1.0.10"], "managed": true, "expiration": "2023/07/15 06:55:49.000", "hash": "e12ed862c8b9ccb0dd988efedf741e1a5fd117f46c73b7d0b7a02dfb4af9dfd2"}
2023/07/14 18:58:37.644 debug pki.ca.local using intermediate signer {"serial": "133945864164094359504464780601471048988", "not_before": "2023-07-14 18:55:48 +0000 UTC", "not_after": "2023-07-21 18:55:48 +0000 UTC"}
2023/07/14 18:59:16.548 info tls.cache.maintenance started background certificate maintenance {"cache": "0x40001fb490"}
2023/07/14 18:59:16.548 info http server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2023/07/14 18:59:16.548 info http enabling automatic HTTP->HTTPS redirects {"server_name": "srv0"}
2023/07/14 18:59:16.549 debug http.handlers.acme_server loaded preexisting CA database {"db_key": "local"}
2023/07/14 18:59:16.567 info http enabling HTTP/3 listener {"addr": ":443"}
2023/07/14 18:59:16.567 debug http starting server loop {"address": "[::]:443", "tls": true, "http3": true}
2023/07/14 18:59:16.567 info http.log server running {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2023/07/14 18:59:16.567 debug http starting server loop {"address": "[::]:80", "tls": false, "http3": false}
2023/07/14 18:59:16.567 info http.log server running {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2023/07/14 18:59:16.567 info http enabling automatic TLS certificate management {"domains": ["10.1.0.10", "*.inductor-test-jkunzler.link"]}
2023/07/14 18:59:16.568 warn tls stapling OCSP {"error": "no OCSP stapling for [10.1.0.10]: no OCSP server specified in certificate", "identifiers": ["10.1.0.10"]}
2023/07/14 18:59:16.568 debug tls.cache added certificate to cache {"subjects": ["10.1.0.10"], "expiration": "2023/07/15 06:55:49.000", "managed": true, "issuer_key": "local", "hash": "e12ed862c8b9ccb0dd988efedf741e1a5fd117f46c73b7d0b7a02dfb4af9dfd2", "cache_size": 1, "cache_capacity": 10000}
2023/07/14 18:59:16.568 debug events event {"name": "cached_managed_cert", "id": "10ec1a96-0a83-4420-bb69-8da6b0a8235a", "origin": "tls", "data": {"sans":["10.1.0.10"]}}
2023/07/14 18:59:16.568 debug tls loading managed certificate {"domain": "*.inductor-test-jkunzler.link", "expiration": "2023/10/12 17:56:18.000", "issuer_key": "acme-staging-v02.api.letsencrypt.org-directory", "storage": "FileStorage:/var/lib/caddy"}
2023/07/14 18:59:16.568 debug tls.cache added certificate to cache {"subjects": ["*.inductor-test-jkunzler.link"], "expiration": "2023/10/12 17:56:18.000", "managed": true, "issuer_key": "acme-staging-v02.api.letsencrypt.org-directory", "hash": "d3c557fdda8c785a264983438420ea3774084e3ee7b5095b37a5f092b5f929a4", "cache_size": 2, "cache_capacity": 10000}
2023/07/14 18:59:16.568 debug events event {"name": "cached_managed_cert", "id": "01eb083b-329f-4ac5-ae3e-61f7e8555249", "origin": "tls", "data": {"sans":["*.inductor-test-jkunzler.link"]}}
2023/07/14 18:59:16.569 warn pki.ca.local installing root certificate (you might be prompted for password) {"path": "storage:pki/authorities/local/root.crt"}
2023/07/14 18:59:17.868 debug http servers shutting down with eternal grace period
2023/07/14 18:59:17.869 info tls.cache.maintenance stopped background certificate maintenance {"cache": "0x400057b880"}
2023/07/14 18:59:34.613 debug http servers shutting down with eternal grace period
2023/07/14 18:59:34.613 info tls.cache.maintenance stopped background certificate maintenance {"cache": "0x40001fb490"}
2023/07/14 18:59:34.613 debug http.handlers.acme_server unloading unused CA database {"db_key": "local"}
2023/07/14 18:59:48.483 info tls.cache.maintenance started background certificate maintenance {"cache": "0x400056ce00"}
2023/07/14 18:59:48.483 info http server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2023/07/14 18:59:48.483 info http enabling automatic HTTP->HTTPS redirects {"server_name": "srv0"}
2023/07/14 18:59:48.542 info pki.ca.local root certificate is already trusted by system {"path": "storage:pki/authorities/local/root.crt"}
2023/07/14 18:59:48.543 debug http starting server loop {"address": "[::]:80", "tls": false, "http3": false}
2023/07/14 18:59:48.543 info http.log server running {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2023/07/14 18:59:48.543 info http enabling HTTP/3 listener {"addr": ":443"}
2023/07/14 18:59:48.543 info tls cleaning storage unit {"description": "FileStorage:/var/lib/caddy"}
2023/07/14 18:59:48.544 debug http starting server loop {"address": "[::]:443", "tls": true, "http3": true}
2023/07/14 18:59:48.544 info http.log server running {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2023/07/14 18:59:48.544 info http enabling automatic TLS certificate management {"domains": ["10.1.0.10", "*.inductor-test-jkunzler.link"]}
2023/07/14 18:59:48.545 info tls finished cleaning storage units
2023/07/14 18:59:48.545 warn tls stapling OCSP {"error": "no OCSP stapling for [10.1.0.10]: no OCSP server specified in certificate", "identifiers": ["10.1.0.10"]}
2023/07/14 18:59:48.545 debug tls.cache added certificate to cache {"subjects": ["10.1.0.10"], "expiration": "2023/07/15 06:55:49.000", "managed": true, "issuer_key": "local", "hash": "e12ed862c8b9ccb0dd988efedf741e1a5fd117f46c73b7d0b7a02dfb4af9dfd2", "cache_size": 1, "cache_capacity": 10000}
2023/07/14 18:59:48.545 debug events event {"name": "cached_managed_cert", "id": "e98830e8-10b7-4bff-9d89-454e2ee8a8f6", "origin": "tls", "data": {"sans":["10.1.0.10"]}}
2023/07/14 18:59:48.546 debug tls loading managed certificate {"domain": "*.inductor-test-jkunzler.link", "expiration": "2023/10/12 17:56:18.000", "issuer_key": "acme-staging-v02.api.letsencrypt.org-directory", "storage": "FileStorage:/var/lib/caddy"}
…
2023/07/14 19:00:34.242 debug http.handlers.reverse_proxy selected upstream {"dial": "10.1.0.146:443", "total_upstreams": 1}
2023/07/14 19:00:34.263 debug http.handlers.reverse_proxy upstream roundtrip {"upstream": "{http.vars.backend_hostport}", "duration": 0.020763756, "request": {"remote_ip": "174.21.143.134", "remote_port": "60296", "proto": "HTTP/2.0", "method": "GET", "host": "10.1.0.146:443", "uri": "/inductor_internal/frontend/static/js/main.742afa3c.js", "headers": {"X-Forwarded-For": ["174.21.143.134"], "Sec-Gpc": ["1"], "Accept-Encoding": ["gzip, deflate, br"], "Accept": ["*/*"], "Referer": ["https://app-1-10.inductor-test-jkunzler.link/"], "Accept-Language": ["en-US,en;q=0.6"], "Sec-Ch-Ua-Platform": ["\"macOS\""], "Sec-Fetch-Dest": ["script"], "X-Forwarded-Host": ["app-1-10.inductor-test-jkunzler.link"], "Sec-Fetch-Site": ["same-origin"], "Sec-Fetch-Mode": ["no-cors"], "Sec-Ch-Ua-Mobile": ["?0"], "X-Forwarded-Proto": ["https"], "Sec-Ch-Ua": ["\"Not.A/Brand\";v=\"8\", \"Chromium\";v=\"114\", \"Brave\";v=\"114\""], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "app-1-10.inductor-test-jkunzler.link"}}, "headers": {"Content-Type": ["application/javascript"], "Date": ["Fri, 14 Jul 2023 19:00:34 GMT"], "Etag": ["061f62e762093e361813d38adcf4b997"], "Last-Modified": ["Fri, 28 Apr 2023 21:39:01 GMT"], "Server": ["Caddy", "uvicorn"], "Content-Length": ["1360865"], "Alt-Svc": ["h3=\":443\"; ma=2592000"]}, "status": 200}
2023/07/14 19:00:34.834 debug http.handlers.reverse_proxy selected upstream {"dial": "10.1.0.146:443", "total_upstreams": 1}
2023/07/14 19:00:34.845 debug http.handlers.reverse_proxy upstream roundtrip {"upstream": "{http.vars.backend_hostport}", "duration": 0.011135525, "request": {"remote_ip": "174.21.143.134", "remote_port": "60296", "proto": "HTTP/2.0", "method": "POST", "host": "10.1.0.146:443", "uri": "/inductor_internal/ui/", "headers": {"X-Forwarded-For": ["174.21.143.134"], "Origin": ["https://app-1-10.inductor-test-jkunzler.link"], "Accept-Language": ["en-US,en;q=0.6"], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"], "Sec-Fetch-Dest": ["empty"], "Sec-Ch-Ua-Platform": ["\"macOS\""], "Sec-Fetch-Mode": ["cors"], "X-Forwarded-Proto": ["https"], "Sec-Ch-Ua": ["\"Not.A/Brand\";v=\"8\", \"Chromium\";v=\"114\", \"Brave\";v=\"114\""], "X-Forwarded-Host": ["app-1-10.inductor-test-jkunzler.link"], "Referer": ["https://app-1-10.inductor-test-jkunzler.link/"], "Sec-Gpc": ["1"], "Content-Length": ["2"], "Sec-Ch-Ua-Mobile": ["?0"], "Content-Type": ["application/json"], "Accept": ["*/*"], "Sec-Fetch-Site": ["same-origin"], "Accept-Encoding": ["gzip, deflate, br"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "app-1-10.inductor-test-jkunzler.link"}}, "headers": {"Content-Type": ["application/json"], "Date": ["Fri, 14 Jul 2023 19:00:34 GMT"], "Server": ["Caddy", "uvicorn"], "Content-Length": ["454"], "Alt-Svc": ["h3=\":443\"; ma=2592000"]}, "status": 200}
2023/07/14 19:00:34.907 debug http.handlers.reverse_proxy selected upstream {"dial": "10.1.0.146:443", "total_upstreams": 1}
2023/07/14 19:00:34.910 debug http.handlers.reverse_proxy upstream roundtrip {"upstream": "{http.vars.backend_hostport}", "duration": 0.002987035, "request": {"remote_ip": "174.21.143.134", "remote_port": "60296", "proto": "HTTP/2.0", "method": "GET", "host": "10.1.0.146:443", "uri": "/inductor_internal/frontend/favicon.png", "headers": {"Sec-Ch-Ua-Mobile": ["?0"], "Sec-Ch-Ua": ["\"Not.A/Brand\";v=\"8\", \"Chromium\";v=\"114\", \"Brave\";v=\"114\""], "Sec-Fetch-Mode": ["no-cors"], "X-Forwarded-Proto": ["https"], "Sec-Fetch-Site": ["same-origin"], "Sec-Fetch-Dest": ["image"], "X-Forwarded-For": ["174.21.143.134"], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"], "Sec-Ch-Ua-Platform": ["\"macOS\""], "Accept-Language": ["en-US,en;q=0.6"], "Sec-Gpc": ["1"], "Accept-Encoding": ["gzip, deflate, br"], "Referer": ["https://app-1-10.inductor-test-jkunzler.link/"], "Accept": ["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"], "X-Forwarded-Host": ["app-1-10.inductor-test-jkunzler.link"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "app-1-10.inductor-test-jkunzler.link"}}, "headers": {"Content-Type": ["image/png"], "Date": ["Fri, 14 Jul 2023 19:00:34 GMT"], "Etag": ["45a19c829ca52d3a6677c7277829e1ce"], "Last-Modified": ["Fri, 28 Apr 2023 21:38:18 GMT"], "Server": ["Caddy", "uvicorn"], "Content-Length": ["153"], "Alt-Svc": ["h3=\":443\"; ma=2592000"]}, "status": 200}
3. Caddy version:
v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=
4. How I installed and ran Caddy:
a. System environment:
Amazon Linux 2023
b. Command:
#!/bin/bash
mkdir /home/reverse_proxy
cd /home/reverse_proxy
dnf -y update
aws s3 cp s3://inductor-cloud/reverse_proxy/ . --recursive --include "*"
mkdir -p /var/lib/caddy/pki/authorities/local/
cp root.crt root.key /var/lib/caddy/pki/authorities/local/
echo "export PRIVATE_IP=$(ifconfig ens5 | grep 'inet ' | awk '{print $2}')" >> /etc/environment
echo "export XDG_CONFIG_HOME=/config" >> /etc/environment
echo "export XDG_DATA_HOME=/data" >> /etc/environment
source /etc/environment
tar -xzvf caddy.tar.gz
./caddy start --config caddy.json
d. My complete Caddy config:
Reverse Proxy:
{
"admin": {
"listen": ":2019"
},
"logging": {
"logs": {
"default": {
"writer": {
"output": "stdout"
},
"level": "DEBUG"
},
"file_logger": {
"writer": {
"filename": "caddy.log",
"output": "file"
},
"level": "DEBUG"
}
}
},
"storage": {
"module": "file_system",
"root": "/var/lib/caddy"
},
"apps": {
"http": {
"servers": {
"srv0": {
"listen": [
":443"
],
"routes": [
{
"match": [
{
"host": [
"{env.PRIVATE_IP}"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"handler": "acme_server"
}
]
}
]
}
],
"terminal": true
},
{
"match": [
{
"host": [
"*.inductor-test-jkunzler.link"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"@id": "map",
"defaults": [
"unknown"
],
"destinations": [
"{backend_host}"
],
"handler": "map",
"mappings": [],
"source": "{http.request.host}"
},
{
"backend_hostport": "{backend_host}:443",
"handler": "vars"
}
]
},
{
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"body": "App not found.",
"handler": "static_response",
"status_code": 404
}
]
}
]
}
],
"match": [
{
"expression": "{backend_host} == \"unknown\""
}
]
},
{
"handle": [
{
"handler": "reverse_proxy",
"headers": {
"request": {
"set": {
"Host": [
"{http.reverse_proxy.upstream.hostport}"
]
}
}
},
"transport": {
"protocol": "http",
"tls": {}
},
"upstreams": [
{
"dial": "{http.vars.backend_hostport}"
}
]
}
]
}
]
}
],
"terminal": true
}
],
"errors": {
"routes": [
{
"match": [
{
"host": [
"*.inductor-test-jkunzler.link"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"body": "Reverse Proxy Internal Error.",
"handler": "static_response",
"status_code": 500
}
]
}
]
}
],
"terminal": true
}
]
}
}
}
},
"tls": {
"automation": {
"policies": [
{
"subjects": [
"{env.PRIVATE_IP}"
],
"issuers": [
{
"module": "internal"
}
]
},
{
"subjects": [
"*.inductor-test-jkunzler.link"
],
"issuers": [
{
"ca": "https://acme-staging-v02.api.letsencrypt.org/directory",
"challenges": {
"dns": {
"provider": {
"name": "route53"
}
}
},
"module": "acme"
},
{
"ca": "https://acme-staging-v02.api.letsencrypt.org/directory",
"challenges": {
"dns": {
"provider": {
"name": "route53"
}
}
},
"module": "zerossl"
}
]
}
]
}
}
}
}
Backend Services that the Reverse Proxy forwards request to:
{
# Global Options.
debug
storage file_system {
root /var/lib/caddy
}
log {
output stdout
}
log file_logger {
output file caddy.log
}
# Note that the CA is not specified in the global options due to the
# following behavior:
# https://caddy.community/t/local-https-using-local-acme-endpoint/20062/5
}
# Forward to App
{$PRIVATE_IP} {
reverse_proxy :8000
# Use the Agent ACME Server.
tls {
ca https://{$REVERSE_PROXY_PRIVATE_IP}/acme/local/directory
ca_root /home/app_agent/root.crt
}
}
# Forward to Agent
{$PRIVATE_IP}:{$AGENT_ACCESS_PORT} {
reverse_proxy :{$AGENT_INTERNAL_PORT}
# Use the Agent ACME Server.
tls {
ca https://{$REVERSE_PROXY_PRIVATE_IP}/acme/local/directory
ca_root /home/app_agent/root.crt
}
}