1. The problem I’m having:
We run an internal ACME CA and internal web services available to employees on our network (i.e. not exposed to the public internet, and not using globally resolvable FQDNs).
The internal ACME server has existed for a while and many internal apps are using it using other ACME clients (win-acme, cert bot etc).
We have a new app deployment that has been in testing/development for a while using either http or self-signed certs, and we are in the process of sorting out certs and final config management before going live.
Attempting to get Caddy integrated with our internal ACME infra has led to us getting a crash-bug.
Expected behavior:
Caddy on start would attempt to get a certificate from the ACME server, and if successful allow the website to be accessed while presenting the certificate. If unsuccessful, it should continue running (not crash), and log an error.
Observed Behavior:
When Caddy is launched with a config that would cause it to request a cert from the internal CA, it immediately crashes.
Troubleshooting step 1
Adding “tls internal” to the website config
Expected Behaviour:
Caddy would generate a self signed cert, run and allow the website to be accessed with an untrusted cert warning.
Observed Behavior:
Caddy operates as expected and website can be accessed, with an untrusted cert warning.
Troubleshooting step 2
Removing custom ACME directives from Caddyfile
Expected Behavior:
Caddy on start would attempt to get a certificate from a public ACME server (i.e. lets encrypt), fail as there is no public DNS record for the server website name and the server is not publicly exposed. Caddy would log this process, handle the error and continue running, however the website won’t be available.
Observed Behavior:
Caddy operates as expected, logs show the attempt and failure to get the cert.
Troubleshooting step 3
Replacing acme_ca with https://fake.com/Directory
Expected Behavior:
Caddy on start would attempt to get a certificate from the configured ACME server, fail as no such CA exists. Caddy would log this process, handle the error and continue running, however the website won’t be available.
Observed Behavior:
Caddy operates as expected, logs show the attempt and failure to get the cert.
2. Error messages and/or full log output:
Yes, our lab/test network uses Contoso as a domain because we’re a mostly Windows/Microsoft shop. This gets swapped to a real domain using Hierra in prod.
Result of systemctl status caddy
Docs: https://caddyserver.com/docs/
Process: 252398 ExecStartPre=/usr/bin/caddy validate --config /etc/caddy/Caddyfile (code=exited, status=0/SUCCESS)
Process: 252403 ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile (code=exited, status=1/FAILURE)
Main PID: 252403 (code=exited, status=1/FAILURE)
Status: "loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [mbcinstweblab04.managed.contoso.com.au]: automate: manage [mbcinstweblab04.managed.contoso.com.au]: mbcinstweblab04.managed.contoso.com.au: caching certificate: certificate has no names
CPU: 93ms
Logs after restarting the systemd service.
tail -n 20 /var/log/caddy/caddy.log
{"level":"info","ts":1744324945.9093757,"msg":"serving initial configuration"}
{"level":"info","ts":1744324979.7846282,"msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":1744324979.784707,"msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":1744324979.7849,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0000293b0"}
{"level":"info","ts":1744324979.7849376,"logger":"admin","msg":"stopped previous server","address":":2019"}
{"level":"info","ts":1744324979.7849438,"msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1744324979.844852,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1744324979.8449085,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1744324979.8451698,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002e0cb0"}
{"level":"info","ts":1744324979.845179,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0002e0cb0"}
{"level":"info","ts":1744324979.898288,"logger":"admin","msg":"admin endpoint started","address":":2019","enforce_origin":true,"origins":["//localhost"]}
{"level":"warn","ts":1744324979.8983526,"logger":"admin","msg":"admin endpoint on open interface; host checking disabled","address":":2019"}
{"level":"info","ts":1744324979.898535,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1744324979.8985467,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1744324979.89882,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1744324979.8990006,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1744324979.8990371,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1744324979.899042,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["mbcinstweblab04.managed.contoso.com.au"]}
{"level":"info","ts":1744324979.8994277,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002893b0"}
{"level":"info","ts":1744324979.8994398,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0002893b0"}
3. Caddy version:
4. How I installed and ran Caddy:
Custom Puppet module that pulls from EPEL repo.
Result of dnf info caddy:
Installed Packages
Name : caddy
Version : 2.6.4
Release : 2.el9
Architecture : x86_64
Size : 44 M
Source : caddy-2.6.4-2.el9.src.rpm
Repository : @System
From repo : epel
Summary : Web server with automatic HTTPS
URL : https://caddyserver.com
License : Apache-2.0 AND BSD-2-Clause AND BSD-3-Clause AND MIT AND BSD-2-Clause-Views AND ISC AND CC0-1.0 AND MPL-2.0
Description : Caddy is the web server with automatic HTTPS.
Run using SystemD
a. System environment:
Alma 9
b. Command:
systemctl restart caddy
c. Service/unit/compose file:
systemctl cat caddy
# /usr/lib/systemd/system/caddy.service
# caddy.service
#
# For using Caddy with a config file.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.
[Unit]
Description=Caddy web server
Documentation=https://caddyserver.com/docs/
After=network.target
[Service]
Type=notify
User=caddy
Group=caddy
ExecStartPre=/usr/bin/caddy validate --config /etc/caddy/Caddyfile
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectHome=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
d. My complete Caddy config:
#
# THIS FILE IS MANAGED BY PUPPET
#
{
http_port 80
https_port 443
admin :2019 {
origins localhost
enforce_origin
}
persist_config off
log default {
output file /var/log/caddy/caddy.log {
roll_size 100MiB
roll_keep 10
roll_local_time
}
}
acme_ca https://contosoca04.contoso.com.au/acme/contoso-acme-internal/directory
}
import Caddyfile.d/*.conf
https://mbcinstweblab04.managed.contoso.com.au {
# tls internal
root * /opt/redmine/public
file_server
@notStatic {
not file
}
reverse_proxy @notStatic localhost:3000
}
5. Links to relevant resources:
N/A