Unable to reload and restart Caddy

1. The problem I’m having:

I am using Systemd. Unable to reload Caddy. It was running okay, but I updated the caddyfile and I needed to reload it, so I ran sudo systemctl reload caddy. However, it timed out silently (Timeout in journalctl but it did not hang up).

Afterwards, I ran sudo systemctl stop caddy. However, there were some dangling processes when I ran ps aux | grep caddy. I kill -9 them.

100032 660950 0.0 0.1 1269804 47680 ? Sl 10:42 0:00 /usr/bin/caddy run --config /Caddyfile
root 661661 0.0 0.0 239916 9984 pts/0 T 10:49 0:00 sudo systemctl reload caddy
root 661663 0.0 0.0 237768 8704 pts/0 T 10:49 0:00 systemctl reload caddy
vestati+ 663504 0.0 0.0 221664 2304 pts/0 S+ 10:58 0:00 grep --color=auto caddy

Even after this, when I ran sudo systemctl start, I was unable to start Caddy. Journalctl output is listed below. I have already run lsof -i :8443 and it is not in use by any other reverse proxy (I do not even have nginx or apache enabled)

2. Error messages and/or full log output:

This is when it was working, and hung on a reload, then I ran systemctl stop.

Jan 20 09:10:14 etune caddy[400091]: {"level":"info","ts":1737335414.257538,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/
caddy/autosave.json"}
Jan 20 09:10:14 etune caddy[400091]: {"level":"info","ts":1737335414.2575674,"msg":"serving initial configuration"}
Jan 20 09:10:14 etune systemd[1]: Started Caddy.
Jan 20 09:10:14 etune caddy[400091]: {"level":"info","ts":1737335414.2974813,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/var/lib/cadd
y/.local/share/caddy"}
Jan 20 09:10:14 etune caddy[400091]: {"level":"info","ts":1737335414.405895,"logger":"tls","msg":"finished cleaning storage units"}
Jan 20 09:12:37 etune caddy[400091]: {"level":"info","ts":1737335557.9376636,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2
019","uri":"/stop","remote_ip":"127.0.0.1","remote_port":"45662","headers":{"Accept-Encoding":["gzip"],"Content-Length":["0"],"Origin":["http://localhost:2019
"],"User-Agent":["Go-http-client/1.1"]}}
Jan 20 09:12:37 etune caddy[400091]: {"level":"warn","ts":1737335557.9379663,"logger":"admin.api","msg":"exiting; byeee!! 👋"}
Jan 20 09:12:37 etune caddy[400091]: {"level":"info","ts":1737335557.9379766,"logger":"http","msg":"servers shutting down with eternal grace period"}
Jan 20 09:12:37 etune caddy[400091]: {"level":"info","ts":1737335557.9382372,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
Jan 20 09:12:37 etune caddy[400091]: {"level":"info","ts":1737335557.9382453,"logger":"admin.api","msg":"shutdown complete","exit_code":0}
Jan 20 09:12:37 etune systemd[1]: caddy.service: Deactivated successfully.
Jan 20 09:51:23 etune systemd[1]: Starting Caddy...
Jan 20 09:51:23 etune caddy[597523]: caddy.HomeDir=/var/lib/caddy

This is when I try and start again. (I tried for 2 hours).

Jan 20 11:04:52 etune caddy[664101]: > {"level":"info","ts":1737342292.138957,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Jan 20 11:04:52 etune caddy[664101]: {"level":"info","ts":1737342292.1390014,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Jan 20 11:04:52 etune caddy[664101]: {"level":"info","ts":1737342292.139086,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
Jan 20 11:04:52 etune caddy[664101]: {"level":"info","ts":1737342292.1391923,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Jan 20 11:04:52 etune caddy[664101]: {"level":"info","ts":1737342292.1392403,"logger":"http","msg":"enabling HTTP/3 listener","addr":":8443"}
Jan 20 11:04:52 etune caddy[664101]: Error: loading initial config: loading new config: http app module: start: starting HTTP/3 QUIC listener: listen udp :8443: bind: permission denied
Jan 20 11:04:52 etune systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
Jan 20 11:04:52 etune systemd[1]: caddy.service: Failed with result 'exit-code'.
Jan 20 11:04:52 etune systemd[1]: Failed to start Caddy.

3. Caddy version:

v2.9.1 h1:OEYiZ7DbCzAWVb6TNEkjRcSCRGHVoZsJinoDR/n9oaY=

4. How I installed and ran Caddy:

a. System environment:

systemd, Red Hat Enterprise Linux release 9.5 (Plow)

b. Command:

dnf install 'dnf-command(copr)'
dnf copr enable @caddy/caddy
dnf install caddy
sudo systemctl daemon-reload
sudo systemctl enable --now caddy

c. Service/unit/compose file:

caddy.service

d. My complete Caddy config:

nextcloud.etuneworks.com {
    reverse_proxy localhost:11000
}


https://nextcloud.etuneworks.com:8443 {
    reverse_proxy localhost:11001 {
        transport http {
            tls_insecure_skip_verify
        }
    }
}

5. Links to relevant resources:

I read this and a few other posts already

This is very strange. It’s behaving as if the requested port is privileged. Can you check the existing value for ip_unprivileged_port_start? I think the command is:

sysctl -n net.ipv4.ip_unprivileged_port_start
1 Like

Hi! I ran the same thing twice because I’m not familiar with sysctl

[vestatian@etune ~]$ cat /proc/sys/net/ipv4/ip_unprivileged_port_start
1024
[vestatian@etune ~]$ sysctl -n net.ipv4.ip_unprivileged_port_start
1024

However, I am pretty sure that having the reserved ports (1-1024) as privileged is normal, is it not? Furthermore, from the Caddy docs, it says if I downloaded the COPR package (I did), then

Your systemd file and caddy binary will already be created and labelled correctly (so you may ignore this section).

[vestatian@etune ~]$ sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      33

In addition, in line with your suggestion, I ran the commands listed here, but it still failed, with the same error I think.

If the HTTP and HTTPS ports (80 and 443, respectively) are privileged ports on your OS, you will either need to run with elevated privileges or use higher ports. To gain permission, run as root with sudo -E or use sudo setcap cap_net_bind_service=+ep $(which caddy). Alternatively, to use higher ports, just change the address to something like localhost:2080 and change the HTTP port using the http_port Caddyfile option.

[vestatian@etune ~]$ sudo setcap cap_net_bind_service=+ep $(which caddy)
[vestatian@etune ~]$ sudo -E systemctl restart caddy
Job for caddy.service failed because the control process exited with error code.
See "systemctl status caddy.service" and "journalctl -xeu caddy.service" for details.
[vestatian@etune ~]$ sudo systemctl status caddy
× caddy.service - Caddy
     Loaded: loaded (/usr/lib/systemd/system/caddy.service; enabled; preset: disabled)
     Active: failed (Result: exit-code) since Tue 2025-01-21 04:30:55 +08; 7s ago
   Duration: 23min 31.966s
       Docs: https://caddyserver.com/docs/
    Process: 712227 ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile (code=exited, status=1/FAILURE)
   Main PID: 712227 (code=exited, status=1/FAILURE)
     Status: "loading new config: http app module: start: starting HTTP/3 QUIC listener: listen udp :8443: bind: permission denied"
        CPU: 47ms

Jan 21 04:30:55 etune caddy[712227]: {"level":"info","ts":1737405055.6642761,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","ser>
Jan 21 04:30:55 etune caddy[712227]: {"level":"info","ts":1737405055.6643074,"logger":"tls.cache.maintenance","msg":"started background certificate maintenan>
Jan 21 04:30:55 etune caddy[712227]: {"level":"info","ts":1737405055.66453,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Jan 21 04:30:55 etune caddy[712227]: {"level":"info","ts":1737405055.6645918,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted>
Jan 21 04:30:55 etune caddy[712227]: {"level":"info","ts":1737405055.6646495,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2",>
Jan 21 04:30:55 etune caddy[712227]: {"level":"info","ts":1737405055.664677,"logger":"http","msg":"enabling HTTP/3 listener","addr":":8443"}
Jan 21 04:30:55 etune caddy[712227]: Error: loading initial config: loading new config: http app module: start: starting HTTP/3 QUIC listener: listen udp :84>
Jan 21 04:30:55 etune systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
Jan 21 04:30:55 etune systemd[1]: caddy.service: Failed with result 'exit-code'.
Jan 21 04:30:55 etune systemd[1]: Failed to start Caddy.

Hi, I think I forgot to tag you in my previous reply, do you mind helping me take a look at the above I posted? Thank you :slight_smile:

That’s ok. I saw the reply. I’m consulting with someone who’s more familiar with RHEL than I am.

Can you share your unit file at /usr/lib/systemd/system/caddy.service?

[vestatian@etune ~]$ cat /usr/lib/systemd/system/caddy.service
# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

Strange. I thought you might’ve had a restart directive in the file, but no. Something in your system is blocking Caddy, and it isn’t clear what it is. Does the issue persist after reboot?