Unable to run Caddy as a daemon

I’m trying to run Caddy as a service with systemd, following https://github.com/mholt/caddy/tree/master/dist/init/linux-systemd but I’m having a lot of trouble trying to make it work.

Now if I run the service I get this error and I don’t know what is causing it.

Started Caddy HTTP/2 web server.
caddy[28100]: Activating privacy features...2017/07/18 10:24:22 [INFO][plorenzo.es] acme: Obtaining bundled     SAN certificate
caddy[28100]: 2017/07/18 10:24:22 [INFO][plorenzo.es] AuthURL: https://acme-  v01.api.letsencrypt.org/acme/authz/v_cIP91xoxxL-EdKYYwHLgdHKPTnatGeTj
caddy[28100]: 2017/07/18 10:24:22 [INFO][plorenzo.es] acme: Trying to solve HTTP-01
caddy[28100]: 2017/07/18 10:24:23 [plorenzo.es] failed to get certificate: [plorenzo.es] error presenting token: Could not start HTTP server 
systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: caddy.service: Unit entered failed state.
systemd[1]: caddy.service: Failed with result 'exit-code'.
systemd[1]: caddy.service: Service hold-off time over, scheduling restart.
systemd[1]: Stopped Caddy HTTP/2 web server.
systemd[1]: caddy.service: Start request repeated too quickly.
systemd[1]: Failed to start Caddy HTTP/2 web server.
systemd[1]: caddy.service: Unit entered failed state.
systemd[1]: caddy.service: Failed with result 'exit-code'.

The command:

ExecStart=/usr/local/bin/caddy -log stdout -agree=true -email=email@email.com -conf=/etc/caddy/Caddyfile -root=/var/tmp

If I run it from the console works fine. So I’m guessing is a problem with the permissions of www-data

My Caddyfile:

example.com {
    root /home/user/project

    log /var/log/caddy/access.log
    errors /var/log/caddy/errors.log

    proxy / unix:/home/user/project/project.sock {
            transparent
            except /static
    }
}

PD: I’m spending a lot of time and effort trying to make this work using different approaches and is getting a little bit frustrating. I know some people don’t agree, but I think it will be super useful and convenient to have an official supported way of running caddy in the background (being a server is what it will do most of the time ) on the documentation.

The log indicates that there are issues fetching certificates.

Is ping plorenzo.es == your_vm_ip?

1 Like

This part says that the certificate requisition failed because Caddy couldn’t start a server to serve the challenge response.

Missing setcap perhaps?

I’m used to seeing log output like failed to get certificate: acme: Error 4xx - urn:acme:error:* with the requester and resolved DNS in cases where public DNS doesn’t match up. Usually Error 403 if the DNS is pointing at the wrong target (because it won’t give the right challenge response), or Error 400 if the DNS isn’t set at all. This seems to be failing earlier than that.

1 Like

If I run the command as my user, caddy works fine and I can access the website under https so I think is not the ip or dns.

I may have found the problem, I ran:

sudo -u www-data /usr/local/bin/caddy -log stdout -agree=true -email=email@email.com -conf=/etc/caddy/Caddyfile -root=/var/tmp 

to see if there was any problem with the permissions of the user www-data tu run caddy and I get the following error:

Activating privacy features...2017/07/19 11:28:52 [INFO] acme: Registering account for email@email.com
could not save user: making user directory: mkdir /var/www/.caddy: permission denied

I don’t understand why caddy is trying to make a folder in /var/www , as you can see in the Caddyfile the project I’m trying to serve is in the /home directory

Any idea where this folder (.caddy) can be coming from?

I also tried to change permissions of /var/www

sudo chown www-data:www-data /var/www
sudo chmod 555 /var/www

with the same result.

Caddy will put the .caddy folder in either $HOME or $CADDYPATH, if it’s trying to manage certificates.

You’re running into that particular problem running as sudo -u www-data because the example unit file sets a $CADDYPATH to override this, but you aren’t.

1 Like

Because journalctl does not have line breaks, I was missing info on the log, the complete line is:

caddy[1356]: 2017/07/19 12:48:44 [plorenzo.es] failed to get certificate: [plorenzo.es] error presenting token: 
Could not start HTTP server for challenge -> listen tcp :80: bind: permission denied

So now it’s clear whats is the problem, any idea how to give it permission? I already use setcap

sudo setcap 'cap_net_bind_service=+ep' /usr/local/bin/caddy

What was the output from running setcap, exactly?

It definitely looks like it didn’t take. That error is almost certainly a privileged port capability issue.

You could also try un-commenting

;CapabilityBoundingSet=CAP_NET_BIND_SERVICE
;AmbientCapabilities=CAP_NET_BIND_SERVICE
;NoNewPrivileges=true

in the service file if you’re on v229 or higher.

1 Like

it didn’t have any output, it should have?

I also tried that (version 232), but I got the following error and I didn’t know what it was so I commented it again

systemd[1]: Network Service is not active.
systemd[1]: Dependency failed for Wait for Network to be Configured.
systemd[1]: systemd-networkd-wait-online.service: Job systemd-networkd-wait-online.service/start failed with result 'dependency'.
systemd[1]: Started Caddy HTTP/2 web server.
systemd[1888]: caddy.service: Failed at step CAPABILITIES spawning /usr/local/bin/caddy: Invalid argument
systemd[1]: caddy.service: Main process exited, code=exited, status=218/CAPABILITIES
systemd[1]: caddy.service: Unit entered failed state.
systemd[1]: caddy.service: Failed with result 'exit-code'.

Ahh, I was hoping that uncommenting those would resolve any capability issues… Unfortunately I’m not that great with systemd.

All I know for sure is that it can’t get :80. If you can run it from the CLI fine and serve a site on :80, that means it’s not being taken by something else, so that basically leaves capability issues.

Any advice from someone more experienced with systemd would be appreciated.

Re: that capabilities failure: nobody has been able to solve it. I have traced the exact error message with strace, but unfortunately the message is not particularly verbose. Here is where I talk about it: Systemd and AmbientCapabilities: Invalid argument / Networking, Server, and Protection / Arch Linux Forums
This is really unfortunate because with systemd 232 I cannot run it even with setcap stuff set and the capability parameters commented out like thus far. I’ve been meaning to post to StackOverflow, but have not had the time and energy.

2 Likes

Thanks for that info… good to know that’s a problem with 232.

Out of curiosity, @plorenzo, which caddy matches the ExecStart from the service unit file, right?

It does

$ which caddy
/usr/local/bin/caddy

I think I will stop trying this for now and go with nohup for the moment…

That’s what I do. :wink: And then I use /etc/rc.local to start Caddy on reboots. (something like nohup sudo -b -i -u myusername caddy -log process.log works great.) Much simpler.

1 Like

Today I finally discovered the cause of my problem: I am running an OVH dedicated server and the kernel was version 3.14. I upgraded it to 4.11 and this solved the systemd problem! Works with the latest systemd 234.
The only thing is that I still have keep the setcap flags for the Caddy binary (sudo setcap cap_net_bind_service=+ep). If the flags are not set, it refuses to bind the ports…

3 Likes

Nice!

@plorenzo does this sound relevant to your situation?

Hey thanks for suggesting this. I just spent several hours going crazy trying to figure out why the damn systemd service couldn’t bind to port 80, and this was the fix. I’m on debian 9 with systemd version 232 and we need to warn users of this, it’s bound to be super common.

I just checked my kernel version and indeed is 3.16 so maybe that’s it.

I don’t have time now to update it but I will come back here when I do to tell you if went well.

(Don’t use setcap, like ever. Doesn’t matter if with systemd or not. It’s like a mild form of setuid.)

You don’t need setcap with systemd because the three lines, to be uncommented as suggested above, will already grant the process the p̶e̶r̶m̶i̶s̶s̶i̶o̶n capability to listen on port 80.

systemd should throw an error if any requested capabilities are not recognized. You would need a senior dev to champion an issue report with systemd’s devs, though. Anyway, good to hear this resolves one of the Exec issues. I wonder if there’s anything more detailed in the kernel logs if this happens, though.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.