1. The problem I’m having:
Hello, I’m setting up a Caddy server for some AWS EC2 instances that will be behind a load balancer. Since these instances have different local IP addresses when invoked, I need to configure the internal creation and management of certificates (given that there is trust within the load balancer’s network, the certificate isn’t verified by the load balancer). The issue is that, even though the certificate is created in the directory /root/.local/share/caddy/certificates/local/{local_ip}, there is an error when making a curl request from another instance on the local network (an error that the load balancer is also supposed to encounter, as it fails to establish the connection). The error is as follows:
curl --insecure https://172.31.38.163/ --verbose
* Trying 172.31.38.163:443...
* Connected to 172.31.38.163 (172.31.38.163) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, internal error (592):
* OpenSSL/3.0.8: error:0A000438:SSL routines::tlsv1 alert internal error
* Closing connection 0
curl: (35) OpenSSL/3.0.8: error:0A000438:SSL routines::tlsv1 alert internal error
2. Error messages and/or full log output:
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal systemd[1]: Started Caddy web server.
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414180.948598,"msg":"using provided configuration","config_file":"/home/ec2-user/caddy/caddy2.json","config_adapter":""}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414180.9495652,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414180.9519787,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414180.9520009,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"debug","ts":1692414180.952023,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{"subjects":["sub.domain.org"]},{"on_demand":true}]}},"http":{"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"vars","root":"/var/www/html"},{"encodings":{"gzip":{}},"handler":"encode","prefer":["gzip"]},{"handler":"file_server","hide":["./Caddyfile"]}]}]}],"terminal":true},{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"vars","root":"/var/www/html"},{"encodings":{"gzip":{}},"handler":"encode","prefer":["gzip"]},{"handler":"file_server","hide":["./Caddyfile"]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414180.958097,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0005a1a00"}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"warn","ts":1692414180.9690626,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414180.9691734,"msg":"not NSS security databases found"}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414180.9691825,"msg":"define JAVA_HOME environment variable to use the Java trust"}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal sudo[10121]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/tee /etc/pki/ca-trust/source/anchors/Caddy_Local_Authority_-_2023_ECC_Root_163894343757556010333559042737653032436.pem
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"error","ts":1692414180.9833474,"logger":"pki.ca.local","msg":"failed to install root certificate","error":"failed to execute sudo: exit status 1","certificate_file":"storage:pki/authorities/local/root.crt"}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"warn","ts":1692414180.9833913,"logger":"tls","msg":"YOUR SERVER MAY BE VULNERABLE TO ABUSE: on-demand TLS is enabled, but no protections are in place","docs":"https://caddyserver.com/docs/automatic-https#on-demand-tls"}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414180.983495,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"debug","ts":1692414180.9836934,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414180.9837077,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"debug","ts":1692414180.983743,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414180.9837508,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414180.9837568,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["sub.domain.org"]}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414180.9839091,"msg":"autosaved config (load with --resume flag)","file":"/root/.config/caddy/autosave.json"}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414180.9839175,"msg":"serving initial configuration"}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414180.984139,"logger":"tls.obtain","msg":"acquiring lock","identifier":"sub.domain.org"}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414180.9843268,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/root/.local/share/caddy"}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414180.9843507,"logger":"tls","msg":"finished cleaning storage units"}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414180.9869928,"logger":"tls.obtain","msg":"lock acquired","identifier":"sub.domain.org"}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414180.9870942,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"sub.domain.org"}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"debug","ts":1692414180.987117,"logger":"events","msg":"event","name":"cert_obtaining","id":"b6da5b8e-ba8b-4a1b-b61c-a765ff8ed99d","origin":"tls","data":{"identifier":"sub.domain.org"}}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"debug","ts":1692414180.9874213,"logger":"tls.obtain","msg":"trying issuer 1/1","issuer":"local"}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"debug","ts":1692414180.987883,"logger":"pki.ca.local","msg":"using intermediate signer","serial":"226801510284420687048247566856494442785","not_before":"2023-08-19 03:03:00 +0000 UTC","not_after":"2023-08-26 03:03:00 +0000 UTC"}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414180.988391,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"sub.domain.org"}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"debug","ts":1692414180.98847,"logger":"events","msg":"event","name":"cert_obtained","id":"191c6783-8fd8-48ea-abab-1b143c6ea915","origin":"tls","data":{"certificate_path":"certificates/local/sub.domain.org/sub.domain.org.crt","identifier":"sub.domain.org","issuer":"local","metadata_path":"certificates/local/sub.domain.org/sub.domain.org.json","private_key_path":"certificates/local/sub.domain.org/sub.domain.org.key","renewal":false,"storage_path":"certificates/local/sub.domain.org"}}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414180.9884913,"logger":"tls.obtain","msg":"releasing lock","identifier":"sub.domain.org"}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"warn","ts":1692414180.9888752,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [sub.domain.org]: no OCSP server specified in certificate","identifiers":["sub.domain.org"]}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"debug","ts":1692414180.9888954,"logger":"tls.cache","msg":"added certificate to cache","subjects":["sub.domain.org"],"expiration":1692457381,"managed":true,"issuer_key":"local","hash":"d06c2e27d57534928efb0bb40a7133824307edcfc8ad984b37d244a67889e820","cache_size":1,"cache_capacity":10000}
ago 18 22:03:00 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"debug","ts":1692414180.988916,"logger":"events","msg":"event","name":"cached_managed_cert","id":"de21175a-2b72-4b99-a738-ae6de1144cfd","origin":"tls","data":{"sans":["sub.domain.org"]}}
ago 18 22:03:03 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"debug","ts":1692414183.908001,"logger":"events","msg":"event","name":"tls_get_certificate","id":"3652e7ca-14a6-49c4-833c-74fc9e924c06","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,4868,49196,49200,52393,52392,49325,49195,49199,49324,49187,49191,49162,49172,49161,49171,157,49309,156,49308,61,60,53,47,159,52394,49311,158,49310,107,103,57,51,255],"ServerName":"","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"Conn":{}}}}
ago 18 22:03:03 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"debug","ts":1692414183.908147,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"172.31.38.163"}
ago 18 22:03:03 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"debug","ts":1692414183.9081573,"logger":"tls.handshake","msg":"all external certificate managers yielded no certificates and no errors","remote_ip":"172.31.13.16","remote_port":"47166","sni":""}
ago 18 22:03:03 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"debug","ts":1692414183.9082532,"logger":"tls.handshake","msg":"did not load cert from storage","remote_ip":"172.31.13.16","remote_port":"47166","server_name":"","error":"no matching certificate to load for : open /root/.local/share/caddy/certificates/local/wildcard_/wildcard_.key: no such file or directory"}
ago 18 22:03:03 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414183.9082682,"logger":"tls.on_demand","msg":"obtaining new certificate","remote_ip":"172.31.13.16","remote_port":"47166","server_name":"172.31.38.163"}
ago 18 22:03:03 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414183.9084954,"logger":"tls.obtain","msg":"acquiring lock","identifier":"172.31.38.163"}
ago 18 22:03:03 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414183.9110374,"logger":"tls.obtain","msg":"lock acquired","identifier":"172.31.38.163"}
ago 18 22:03:03 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414183.9111583,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"172.31.38.163"}
ago 18 22:03:03 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"debug","ts":1692414183.911182,"logger":"events","msg":"event","name":"cert_obtaining","id":"3ad3e61e-8db3-4d0c-ae1e-90105461536a","origin":"tls","data":{"identifier":"172.31.38.163"}}
ago 18 22:03:03 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"debug","ts":1692414183.9115074,"logger":"tls.obtain","msg":"trying issuer 1/1","issuer":"local"}
ago 18 22:03:03 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"debug","ts":1692414183.911844,"logger":"pki.ca.local","msg":"using intermediate signer","serial":"226801510284420687048247566856494442785","not_before":"2023-08-19 03:03:00 +0000 UTC","not_after":"2023-08-26 03:03:00 +0000 UTC"}
ago 18 22:03:03 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414183.912278,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"172.31.38.163"}
ago 18 22:03:03 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"debug","ts":1692414183.9123573,"logger":"events","msg":"event","name":"cert_obtained","id":"b1d455a2-58ef-4ac0-b3cc-bb1318d495d0","origin":"tls","data":{"certificate_path":"certificates/local/172.31.38.163/172.31.38.163.crt","identifier":"172.31.38.163","issuer":"local","metadata_path":"certificates/local/172.31.38.163/172.31.38.163.json","private_key_path":"certificates/local/172.31.38.163/172.31.38.163.key","renewal":false,"storage_path":"certificates/local/172.31.38.163"}}
ago 18 22:03:03 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"info","ts":1692414183.9123795,"logger":"tls.obtain","msg":"releasing lock","identifier":"172.31.38.163"}
ago 18 22:03:03 ip-172-31-38-163.us-east-2.compute.internal caddy[10112]: {"level":"debug","ts":1692414183.9125311,"logger":"http.stdlib","msg":"http: TLS handshake error from 172.31.13.16:47166: no matching certificate to load for : open /root/.local/share/caddy/certificates/local/wildcard_/wildcard_.key: no such file or directory"}
3. Caddy version:
v2.7.3 h1:eMCNjOyMgB5A1KgOzT2dXKR4I0Va+YHCJYC8HHu+DP0=
4. How I installed and ran Caddy:
Binary downloaded from Download Caddy with GitHub - mholt/caddy-ratelimit: HTTP rate limiting module for Caddy 2 plugin
a. System environment:
Linux 4.14.320-243.544.amzn2.x86_64 #1 SMP Tue Aug 1 21:03:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
b. Command:
sudo systemctl start caddy
c. Service/unit/compose file:
[Unit]
Description=Caddy web server
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
User=root
Group=root
ExecStart=/home/ec2-user/caddy/caddy run --config /home/ec2-user/caddy/caddy2.json
ExecReload=/home/ec2-user/caddy/caddy reload --config /home/ec2-user/caddy/caddy2.json
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
d. My complete Caddy config:
{
"logging": {
"logs": {
"default": {
"level": "DEBUG"
}
}
},
"apps": {
"http": {
"servers": {
"srv0": {
"listen": [
":443"
],
"routes": [
{
"match": [
{
"host": [
"sub.domain.org"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"handler": "vars",
"root": "/var/www/html"
},
{
"encodings": {
"gzip": {}
},
"handler": "encode",
"prefer": [
"gzip"
]
},
{
"handler": "file_server",
"hide": [
"./Caddyfile"
]
}
]
}
]
}
],
"terminal": true
},
{
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"handler": "vars",
"root": "/var/www/html"
},
{
"encodings": {
"gzip": {}
},
"handler": "encode",
"prefer": [
"gzip"
]
},
{
"handler": "file_server",
"hide": [
"./Caddyfile"
]
}
]
}
]
}
],
"terminal": true
}
]
}
}
},
"tls": {
"automation": {
"policies": [
{
"subjects": [
"sub.domain.org"
],
"issuers": [
{
"module": "internal"
}
]
},
{
"issuers": [
{
"module": "internal"
}
],
"on_demand": true
}
]
}
}
}
}