Caddy Systemctl Service | Resource temporarily unavailable

[Hendrik_R]

1. Caddy version (caddy version): v2.4.6

2. How I run Caddy:

systemd service + Caddyfile

a. System environment:

Proxmox 7.1-5, debian 11.1 container, kernel: 5.13.19-1-pve

b. Command:

/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

c. Service/unit/compose file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

3. The problem I’m having:

Caddy can’t be start/reloaded from systemctl anymore, see status attached.
Occured after kenel update, wondering if this is an authorization problem.

When running the ExecStart command directly, everything works fine, so Caddyfile is okay.

4. Error messages and/or full log output:

systemctl status caddy outputs me:

root@cCaddy:/etc/caddy# systemctl status caddy
* caddy.service - Caddy
     Loaded: loaded (/lib/systemd/system/caddy.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sun 2021-11-21 22:45:01 CET; 13min ago
       Docs: https://caddyserver.com/docs/
   Main PID: 1510 (code=exited, status=203/EXEC)
        CPU: 12ms

Nov 21 22:45:01 cCaddy systemd[1]: Starting Caddy...
Nov 21 22:45:01 cCaddy systemd[1510]: caddy.service: Failed to execute /usr/bin/caddy: Resource temporarily unavailable
Nov 21 22:45:01 cCaddy systemd[1510]: caddy.service: Failed at step EXEC spawning /usr/bin/caddy: Resource temporarily unavailable
Nov 21 22:45:01 cCaddy systemd[1]: caddy.service: Main process exited, code=exited, status=203/EXEC
Nov 21 22:45:01 cCaddy systemd[1]: caddy.service: Failed with result 'exit-code'.
Nov 21 22:45:01 cCaddy systemd[1]: Failed to start Caddy.

5. What I already tried:

sudo in ExecStart/ExecReload
running service as root

6. Links to relevant resources:

[francislavoie]

It’s probably because there’s some error with the configuration. The systemctl status command only shows a truncated view of the logs, it’s not the best way to see what’s actually going on.

You can find the best command to run to see your logs here in the docs:

Keep Caddy Running - Caddy Documentation

[Hendrik_R]

Thanks for your reply,

i had a look on that doc page before submitting.
Unfortunally the journalctl command does not give many more information:

Nov 22 00:03:46 cCaddy systemd[1]: Starting Caddy...
Nov 22 00:03:46 cCaddy systemd[143]: caddy.service: Failed to execute /usr/bin/caddy: Resource temporarily unavailable
Nov 22 00:03:46 cCaddy systemd[143]: caddy.service: Failed at step EXEC spawning /usr/bin/caddy: Resource temporarily unavailable
Nov 22 00:03:46 cCaddy systemd[1]: caddy.service: Main process exited, code=exited, status=203/EXEC
Nov 22 00:03:46 cCaddy systemd[1]: caddy.service: Failed with result 'exit-code'.
Nov 22 00:03:46 cCaddy systemd[1]: Failed to start Caddy.

[francislavoie]

In that case, it must be something broken with your system, if Caddy doesn’t even get to the point of starting up. I don’t really have anything to suggest unfortunately. Maybe try a reinstall?

[Hendrik_R]

Probably yes.
Tried it out with a brand new container with debian 11.1 and directly after apt install caddy, even within the apt install output, the system outputs the same error.
Seems to be a problem regarding the 5.13.19-1-pve kernel, error is occurring since the update.
Gonna have a look into it, maybe I can find a workaround etc.
May be a problem caddy might facing with upcoming kernel updates.

Regards

[agrajag (Todd Grayson)]

@Hendrik_R

cCaddy

Where is cCaddy emerging from? Do you have two conflicting unit files in your /lib/systemd/system/ path or something?

Check out the troubleshooting points given in this older post as well…

Run caddy as a systemd service on a desktop computer (fedora)

Here’s what I do to run it manually in the shell: $ cd /etc/caddy/ [caddy]$ caddy Activating privacy features... done. http://localhost WARNING: File descriptor limit 1024 is too low for production servers. At least 8192 is recommended. Fix with "ulimit -n 8192". Even the command in the caddy.service file works fine if launched manually: $ /usr/local/bin/caddy -log stdout -agree=true -conf=/etc/caddy/Caddyfile -root=/home/fede/web Activating privacy features... done. http://localhost 2017/04/…

[francislavoie]

@agrajag

Where is cCaddy emerging from? Do you have two conflicting unit files in your /lib/systemd/system/ path or something?

That’s just the name of their machine I think

[agrajag (Todd Grayson)]

@francislavoie

That’s just the name of their machine I think

Ah, I should know that… just not used to seeing UPPERCASE in host names.

[Hendrik_R]

Yes, it is just the containers hostname ^^.
Gonna try a few things out and post them here.

Thanks for your replies

[Hendrik_R]

I tested everything today and booted the system with the 5.11.22-7(-pve) kernel instead of the ?5.13.19-1(-pve) version. Everything is starting up there with no problems.
So it seems to be a problem occuring with the newest linux kernel versions, and it seems to be either a problem with go not being able to create new procresses within the service, or the caddy.service configuration not applicable with 5.13.19-1(-pve) or further.

I hope that might be interesting for the dev team.

We use the current proxmox release 7.1-6 with a debian 11.1 container.

Hopefully i could help you a bit and this might be interesting.