1. Caddy version (caddy version
):
v2.3.0 h1:fnrqJLa3G5vfxcxmOH/+kJOcunPLhSBnjgIvjXV/QTA=
2. How I run Caddy:
sudo systemctl start caddy
a. System environment:
Red Hat Enterprise Linux release 8.3 (Ootpa)
b. Command:
sudo systemctl start caddy
c. Service/unit/compose file:
systemd
service configuration file /etc/systemd/system/caddy.service
:
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
# PrivateTmp=true # Temporary switched off for debugging purposes
# ProtectSystem=full # Temporary switched off for debugging purposes
AmbientCapabilities=CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
d. My complete Caddyfile or JSON config:
dandeliion.soton.ac.uk, 192.168.174.219 {
root * /var/www
file_server
@paths {
path /admin/*
path /queue/*
path /simulation/*
path /tos/*
}
reverse_proxy @paths 127.0.0.1:8000
tls internal
log {
output file /tmp/caddy_access.log
format single_field common_log
}
}
3. The problem I’m having:
As you can see from Caddyfile, I set up a file server (for static files) and a reverse proxy (for django project). When I start Caddy using caddy start
or even /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
, it works great! I see my django webpages and static files in the web browser.
Now I want to start Caddy via systemd
. I created caddy.service
file (see above) and now starting Caddy with exactly the same Caddyfile using sudo systemctl start caddy
. The service starts and becomes green (active). But the web browser now only shows django webpages without any static files. Caddy access log file shows 403 error code for all static files (see below).
4. Error messages and/or full log output:
What Caddy access log shows. Django pages have status 200, they were shown properly. But all static files give 403 error code:
152.78.118.247 - - [29/Jan/2021:13:38:18 +0000] "GET /queue/?page=8 HTTP/1.1" 200 26016
152.78.118.247 - - [29/Jan/2021:13:38:18 +0000] "GET /static/style.css HTTP/1.1" 403 0
152.78.118.247 - - [29/Jan/2021:13:38:18 +0000] "GET /static/dl_logo.png HTTP/1.1" 403 0
152.78.118.247 - - [29/Jan/2021:13:38:18 +0000] "GET /static/description.js HTTP/1.1" 403 0
152.78.118.247 - - [29/Jan/2021:13:38:21 +0000] "GET /static/1.txt HTTP/1.1" 403 0
152.78.118.247 - - [29/Jan/2021:13:38:32 +0000] "GET /1.txt HTTP/1.1" 403 0
sudo journalctl -xe
gives the following output:
Jan 29 19:03:28 srv03119 sudo[63009]: ik3g18 : TTY=pts/0 ; PWD=/etc/caddy ; USER=root ; COMMAND=/bin/systemctl start caddy
Jan 29 19:03:28 srv03119 sudo[63009]: pam_unix(sudo:session): session opened for user root by ik3g18(uid=0)
Jan 29 19:03:28 srv03119 systemd[1]: Started Caddy.
-- Subject: Unit caddy.service has finished start-up
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit caddy.service has finished starting up.
--
-- The start-up result is done.
Jan 29 19:03:28 srv03119 sudo[63009]: pam_unix(sudo:session): session closed for user root
Jan 29 19:03:28 srv03119 caddy[63012]: caddy.HomeDir=/var/lib/caddy
Jan 29 19:03:28 srv03119 caddy[63012]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Jan 29 19:03:28 srv03119 caddy[63012]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Jan 29 19:03:28 srv03119 caddy[63012]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Jan 29 19:03:28 srv03119 caddy[63012]: caddy.Version=v2.3.0
Jan 29 19:03:28 srv03119 caddy[63012]: runtime.GOOS=linux
Jan 29 19:03:28 srv03119 caddy[63012]: runtime.GOARCH=amd64
Jan 29 19:03:28 srv03119 caddy[63012]: runtime.Compiler=gc
Jan 29 19:03:28 srv03119 caddy[63012]: runtime.NumCPU=12
Jan 29 19:03:28 srv03119 caddy[63012]: runtime.GOMAXPROCS=12
Jan 29 19:03:28 srv03119 caddy[63012]: runtime.Version=go1.14.12
Jan 29 19:03:28 srv03119 caddy[63012]: os.Getwd=/
Jan 29 19:03:28 srv03119 caddy[63012]: LANG=en_GB.UTF-8
Jan 29 19:03:28 srv03119 caddy[63012]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Jan 29 19:03:28 srv03119 caddy[63012]: HOME=/var/lib/caddy
Jan 29 19:03:28 srv03119 caddy[63012]: LOGNAME=caddy
Jan 29 19:03:28 srv03119 caddy[63012]: USER=caddy
Jan 29 19:03:28 srv03119 caddy[63012]: INVOCATION_ID=67feef6e5e3a48f895839c0d35f082dc
Jan 29 19:03:28 srv03119 caddy[63012]: JOURNAL_STREAM=9:1841204
Jan 29 19:03:28 srv03119 caddy[63012]: {"level":"info","ts":1611947008.121328,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Jan 29 19:03:28 srv03119 caddy[63012]: {"level":"info","ts":1611947008.1235092,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Jan 29 19:03:28 srv03119 caddy[63012]: {"level":"info","ts":1611947008.1237803,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002192d0"}
Jan 29 19:03:28 srv03119 caddy[63012]: {"level":"info","ts":1611947008.136686,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Jan 29 19:03:28 srv03119 caddy[63012]: {"level":"info","ts":1611947008.1367385,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Jan 29 19:03:28 srv03119 caddy[63012]: {"level":"warn","ts":1611947008.1596482,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
Jan 29 19:03:28 srv03119 caddy[63012]: 2021/01/29 19:03:28 not NSS security databases found
Jan 29 19:03:28 srv03119 caddy[63012]: 2021/01/29 19:03:28 define JAVA_HOME environment variable to use the Java trust
Jan 29 19:03:28 srv03119 sudo[63029]: caddy : unable to open /run/sudo/ts/caddy : Permission denied ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/tee /etc/pki/ca-trust/source/anchors/Caddy_Local_Authority_-_2021_ECC_Root_90585801314882041225458515839130920035.pem
Jan 29 19:03:28 srv03119 unix_chkpwd[63033]: check pass; user unknown
Jan 29 19:03:28 srv03119 sudo[63029]: pam_unix(sudo:auth): conversation failed
Jan 29 19:03:28 srv03119 sudo[63029]: pam_unix(sudo:auth): auth could not identify password for [caddy]
Jan 29 19:03:28 srv03119 sudo[63029]: pam_sss(sudo:auth): authentication failure; logname= uid=100999 euid=0 tty= ruser=caddy rhost= user=caddy
Jan 29 19:03:28 srv03119 sudo[63029]: pam_sss(sudo:auth): received for user caddy: 9 (Authentication service cannot retrieve authentication info)
Jan 29 19:03:28 srv03119 sudo[63029]: PAM audit_open() failed: Permission denied
Jan 29 19:03:28 srv03119 sudo[63029]: PAM audit_open() failed: Permission denied
Jan 29 19:03:28 srv03119 postfix/sendmail[63032]: fatal: open /etc/postfix/main.cf: Permission denied
Jan 29 19:03:28 srv03119 sudo[63029]: caddy : user NOT in sudoers ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/tee /etc/pki/ca-trust/source/anchors/Caddy_Local_Authority_-_2021_ECC_Root_90585801314882041225458515839130920035.pem
Jan 29 19:03:28 srv03119 caddy[63012]: {"level":"error","ts":1611947008.1809275,"logger":"pki.ca.local","msg":"failed to install root certificate","error":"failed to execute sudo: exit status 1","certificate_file":"storage:pki/authorities/local/root.crt"}
Jan 29 19:03:28 srv03119 caddy[63012]: {"level":"info","ts":1611947008.1812618,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["dandeliion.soton.ac.uk","192.168.174.219"]}
Jan 29 19:03:28 srv03119 caddy[63012]: {"level":"warn","ts":1611947008.181994,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [dandeliion.soton.ac.uk]: no OCSP server specified in certificate"}
Jan 29 19:03:28 srv03119 caddy[63012]: {"level":"warn","ts":1611947008.1824648,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [192.168.174.219]: no OCSP server specified in certificate"}
Jan 29 19:03:28 srv03119 caddy[63012]: {"level":"info","ts":1611947008.1826508,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Jan 29 19:03:28 srv03119 caddy[63012]: {"level":"info","ts":1611947008.1826704,"msg":"serving initial configuration"}
Jan 29 19:03:28 srv03119 caddy[63012]: {"level":"info","ts":1611947008.1827505,"logger":"tls","msg":"cleaned up storage units"}
Jan 29 19:03:28 srv03119 postfix/sendmail[63036]: fatal: open /etc/postfix/main.cf: Permission denied
5. What I already tried:
Of course I double checked the owners and permissions in /var/www
folder. I even changed the owner/group for all files and directories to caddy:caddy:
$ ls -la /var/www/
total 8
drwxr-xr-x. 3 caddy caddy 33 Jan 27 23:57 .
drwxr-xr-x. 23 root root 4096 Jan 27 23:31 ..
-rw-r--r--. 1 root root 5 Jan 27 23:57 1.txt
drwxr-xr-x. 5 caddy caddy 199 Jan 27 23:58 static
and
$ ls -la /var/www/static
total 136
drwxr-xr-x. 5 caddy caddy 199 Jan 27 23:58 .
drwxr-xr-x. 3 caddy caddy 33 Jan 27 23:57 ..
-rw-r--r--. 1 caddy caddy 5 Jan 27 23:58 1.txt
drwxr-xr-x. 6 caddy caddy 51 Jan 27 23:34 admin
-rw-r--r--. 1 caddy caddy 1753 Jan 27 23:34 description.js
-rw-r--r--. 1 caddy caddy 19050 Jan 27 23:34 dl_logo.png
-rw-r--r--. 1 caddy caddy 26579 Jan 27 23:34 double_part_size_s.png
-rw-r--r--. 1 caddy caddy 630 Jan 27 23:34 favicon.png
drwxr-xr-x. 2 caddy caddy 4096 Jan 27 23:34 queue
drwxr-xr-x. 218 caddy caddy 12288 Jan 27 23:34 results
-rw-r--r--. 1 caddy caddy 45247 Jan 27 23:34 style.css
Here you can see file 1.txt
created for testing purposes. But Caddy still doesn’t show it in the browser and returns 403 error (see Caddy access log above).
But everything works perfect when I do caddy start
, it shows all in the browser, including 1.txt
without any errors. Something doesn’t work when I start it as a service via systemctl
.
6. Links to relevant resources:
N/A