Guide to fix back the permission for Caddy in systemd

steadfast · February 17, 2022, 8:20am

It’s better if I write in point form

Running Caddy 2.1.1 for sometime which was install thru apt install caddy and it work fine.
Recently, I discovered several sites are not secure and it shown certs were “revoked”.
In a panic and confuse where to find the solutions, Google is unhelpful, I’ve end up download new Caddy binary in Ubuntu VPS thru curl -sS Caddy | webinstall.dev | bash
Copy the new binary to /usr/bin and version shown:
v2.4.6 h1:HGkGICFGvyrodcqOOclHKfvJC0qTU7vny/7FhYp9hNw=
Tried to set
sudo setcap CAP_NET_BIND_SERVICE=+eip /usr/bin/caddy
Tried to follow the exact permission as old Caddy version
sudo chown steadfast:steadfast /usr/bin/caddy
Clients site are still appear as not secure
sudo systemctl restart caddy
In caddy systemd service config, I had made change from “caddy” to “root” and it work
[Service]
User=root
Group=root

This experience is quite stressful and fragile as there is not easier way to validate where did I done wrong with my lack or forgotten knowledge how to make it work. Caddy should be easier to at least validate folder permissions that is very common issue to millions of developers and error prone and we always have to rely Google “Help” me, same for issue with certbot as well.

I’m curious how do I fix back the caddy binary in the present state to work using “Caddy” permission like before?

francislavoie · February 17, 2022, 4:06pm

Not sure where you got this, but you should use our official instructions to install Caddy:

Binaries in /usr/bin should be owned by root:root with permissions 755 (i.e. rwxr-xr-x)

You should also not do this, Caddy shouldn’t run as root. It should run as the caddy user.

What did you see in Caddy’s logs? See this page in the docs to understand how to use Caddy when running as a service, including the command to run to see your logs:

steadfast · February 18, 2022, 4:37am

In the log, I noticed there is a typo “Caddu”, not sure if it was already fix in Caddy repo.

I have removed domain name from the log for privacy reason.
When using “caddy” in caddy.service, I got the output log:

Feb 18 04:19:55 ubuntu-1cpu-1gb-sg-sin1 systemd[1]: Stopping Caddu...
Feb 18 04:19:55 ubuntu-1cpu-1gb-sg-sin1 caddy[863471]: {"level":"info","ts":1645157995.9117746,"msg":"shutting down apps, then terminating","signal":"SIGTERM"}
Feb 18 04:19:55 ubuntu-1cpu-1gb-sg-sin1 caddy[863471]: {"level":"warn","ts":1645157995.9123497,"msg":"exiting; byeee!! 👋","signal":"SIGTERM"}

Feb 18 04:19:55 ubuntu-1cpu-1gb-sg-sin1 caddy[863471]: {"level":"info","ts":1645157995.933974,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00053e690"}
Feb 18 04:19:55 ubuntu-1cpu-1gb-sg-sin1 caddy[863471]: {"level":"info","ts":1645157995.9368618,"logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
Feb 18 04:19:55 ubuntu-1cpu-1gb-sg-sin1 caddy[863471]: {"level":"info","ts":1645157995.9368834,"msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
Feb 18 04:19:55 ubuntu-1cpu-1gb-sg-sin1 systemd[1]: caddy.service: Succeeded.
Feb 18 04:19:55 ubuntu-1cpu-1gb-sg-sin1 systemd[1]: Stopped Caddu.
Feb 18 04:19:55 ubuntu-1cpu-1gb-sg-sin1 systemd[1]: Starting Caddu...
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: caddy.HomeDir=/var/lib/caddy
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: caddy.Version=v2.4.6 h1:HGkGICFGvyrodcqOOclHKfvJC0qTU7vny/7FhYp9hNw=
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: runtime.GOOS=linux
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: runtime.GOARCH=amd64
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: runtime.Compiler=gc
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: runtime.NumCPU=1
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: runtime.GOMAXPROCS=1
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: runtime.Version=go1.17.2
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: os.Getwd=/
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: LANG=en_US.UTF-8
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: NOTIFY_SOCKET=/run/systemd/notify
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: HOME=/var/lib/caddy
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: LOGNAME=caddy
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: USER=caddy
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: INVOCATION_ID=990dac76ce0c423b8aa9f45e112d6232
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: JOURNAL_STREAM=9:198151816
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: {"level":"info","ts":1645157996.0381103,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: {"level":"warn","ts":1645157996.0594053,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":14}
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: {"level":"info","ts":1645157996.0665991,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["[::1]:2019","127.0.0.1:2019","localhost:2019"]}
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: {"level":"info","ts":1645157996.067354,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: {"level":"info","ts":1645157996.0675013,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: {"level":"info","ts":1645157996.0676966,"logger":"http","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: {"level":"info","ts":1645157996.0768266,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00027cb60"}
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: {"level":"info","ts":1645157996.1161795,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: {"level":"info","ts":1645157996.1263115,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["--removed domains--"]}
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: {"level":"info","ts":1645157996.1319578,"logger":"tls","msg":"finished cleaning storage units"}
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: {"level":"info","ts":1645157996.8839152,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 systemd[1]: Started Caddu.
Feb 18 04:19:56 ubuntu-1cpu-1gb-sg-sin1 caddy[863873]: {"level":"info","ts":1645157996.8870811,"msg":"serving initial configuration"}
(END)

francislavoie · February 18, 2022, 6:02am

That typo has never existed in the Caddy repo. I don’t know where you got that from.

github.com

caddyserver/dist/blob/master/init/caddy.service

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

This file has been truncated. show original

Otherwise, those logs look fine. There’s no problems there, that’s just Caddy’s startup logs. Nothing special.

steadfast · February 18, 2022, 7:14am

Sorry, I saw it was a typo in my service file Description=Caddu could be from my old config. Before posting the log in Caddy forum, I’ve ran apt install caddy.

Compare and is the exact configuration file as the sample you provided, caddy.service, testing on web browser in private mode appear non-secure, back to root user appear as secure HTTPS.

Still thinking if there are other way to check the log.
Systemd version is 245.
Caddy → WordPress

steadfast · February 18, 2022, 8:57am

I get 2 out of other site on the same VPS shown:
Error code: SEC_ERROR_REVOKED_CERTIFICATE

This seem I need to renew?

francislavoie · February 18, 2022, 9:23am

On January 28th, Let’s Encrypt did revoke a ton of certificates. If that’s the error you’re seeing, then yes, you need to renew the certificates.

Caddy should have done it automatically, but it may have not worked for a few different reasons. You said at the top post you were on Caddy v2.1.1, which was quite an old version, which didn’t support automatically renewing revoked certificates.

You can clear out Caddy’s storage location (noted in your logs, caddy.AppDataDir=/var/lib/caddy/.local/share/caddy) then restart Caddy, and it will re-issue all the certificates it needs.

steadfast · February 18, 2022, 9:57am

That solve it! Thought this issue could have been pinned in the Caddy forum for a while if the same issue happen for others.

francislavoie · February 18, 2022, 10:10am

There’s been multiple threads about it, and it should have already been resolved by everyone effected, as long as they pay attention to their sites and keep Caddy up to date.

system · March 19, 2022, 8:21am

This topic was automatically closed after 30 days. New replies are no longer allowed.