403 Error with SELinux on file_server but not in reverse_proxy

Help
1

1. The problem I’m having:

When I launch Caddy as a service, the browser (and everything’s else) shows me 403 Forbidden. I know it’s caused by SELinux since I saw some posts talking about an issue similar to this, and I can confirm since when I set the enforce mode to disabled, it works. The solutions in the thread didn’t work for me though.

2. Error messages and/or full log output:

juin 02 15:48:10 sewt-server systemd[1]: Starting caddy.service - Caddy web server...
juin 02 15:48:11 sewt-server caddy[5503]: {"level":"info","ts":1748872091.0248208,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
juin 02 15:48:11 sewt-server caddy[5503]: {"level":"info","ts":1748872091.026407,"msg":"adapted config to JSON","adapter":"caddyfile"}
juin 02 15:48:11 sewt-server caddy[5503]: {"level":"info","ts":1748872091.0268722,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
juin 02 15:48:11 sewt-server caddy[5503]: {"level":"info","ts":1748872091.0268927,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
juin 02 15:48:11 sewt-server caddy[5503]: {"level":"warn","ts":1748872091.0269105,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
juin 02 15:48:11 sewt-server caddy[5503]: {"level":"info","ts":1748872091.026945,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0001c1200"}
juin 02 15:48:11 sewt-server caddy[5503]: {"level":"info","ts":1748872091.0274484,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0001c1200"}
juin 02 15:48:11 sewt-server caddy[5503]: Valid configuration
juin 02 15:48:11 sewt-server caddy[5512]: {"level":"info","ts":1748872091.1330614,"msg":"maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined"}
juin 02 15:48:11 sewt-server caddy[5512]: {"level":"info","ts":1748872091.1332448,"msg":"GOMEMLIMIT is updated","package":"github.com/KimMachineGun/automemlimit/memlimit","GOMEMLIMIT":14970238156,"previous":9223372036854775807}
juin 02 15:48:11 sewt-server caddy[5512]: caddy.HomeDir=/var/lib/caddy
juin 02 15:48:11 sewt-server caddy[5512]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
juin 02 15:48:11 sewt-server caddy[5512]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
juin 02 15:48:11 sewt-server caddy[5512]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
juin 02 15:48:11 sewt-server caddy[5512]: caddy.Version=v2.10.0
juin 02 15:48:11 sewt-server caddy[5512]: runtime.GOOS=linux
juin 02 15:48:11 sewt-server caddy[5512]: runtime.GOARCH=amd64
juin 02 15:48:11 sewt-server caddy[5512]: runtime.Compiler=gc
juin 02 15:48:11 sewt-server caddy[5512]: runtime.NumCPU=4
juin 02 15:48:11 sewt-server caddy[5512]: runtime.GOMAXPROCS=4
juin 02 15:48:11 sewt-server caddy[5512]: runtime.Version=go1.24.2
juin 02 15:48:11 sewt-server caddy[5512]: os.Getwd=/
juin 02 15:48:11 sewt-server caddy[5512]: LANG=fr_FR.UTF-8
juin 02 15:48:11 sewt-server caddy[5512]: PATH=/usr/local/bin:/usr/bin
juin 02 15:48:11 sewt-server caddy[5512]: NOTIFY_SOCKET=/run/systemd/notify
juin 02 15:48:11 sewt-server caddy[5512]: USER=caddy
juin 02 15:48:11 sewt-server caddy[5512]: LOGNAME=caddy
juin 02 15:48:11 sewt-server caddy[5512]: HOME=/var/lib/caddy
juin 02 15:48:11 sewt-server caddy[5512]: INVOCATION_ID=12fef9d26afd4ed39d52b7f551392d1c
juin 02 15:48:11 sewt-server caddy[5512]: JOURNAL_STREAM=9:37407
juin 02 15:48:11 sewt-server caddy[5512]: SYSTEMD_EXEC_PID=5512
juin 02 15:48:11 sewt-server caddy[5512]: MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/system.slice/caddy.service/memory.pressure
juin 02 15:48:11 sewt-server caddy[5512]: MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA=
juin 02 15:48:11 sewt-server caddy[5512]: GOTRACEBACK=none
juin 02 15:48:11 sewt-server caddy[5512]: {"level":"info","ts":1748872091.1333652,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
juin 02 15:48:11 sewt-server caddy[5512]: {"level":"info","ts":1748872091.134709,"msg":"adapted config to JSON","adapter":"caddyfile"}
juin 02 15:48:11 sewt-server caddy[5512]: {"level":"info","ts":1748872091.1360605,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
juin 02 15:48:11 sewt-server caddy[5512]: {"level":"info","ts":1748872091.1363387,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
juin 02 15:48:11 sewt-server caddy[5512]: {"level":"info","ts":1748872091.1363566,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
juin 02 15:48:11 sewt-server caddy[5512]: {"level":"info","ts":1748872091.1363797,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000516480"}
juin 02 15:48:11 sewt-server caddy[5512]: {"level":"warn","ts":1748872091.1363838,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
juin 02 15:48:11 sewt-server caddy[5512]: {"level":"info","ts":1748872091.1369312,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
juin 02 15:48:11 sewt-server caddy[5512]: {"level":"info","ts":1748872091.1370285,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
juin 02 15:48:11 sewt-server caddy[5512]: {"level":"info","ts":1748872091.1371171,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
juin 02 15:48:11 sewt-server caddy[5512]: {"level":"warn","ts":1748872091.1371758,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"}
juin 02 15:48:11 sewt-server caddy[5512]: {"level":"warn","ts":1748872091.1371834,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"}
juin 02 15:48:11 sewt-server caddy[5512]: {"level":"info","ts":1748872091.1371894,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
juin 02 15:48:11 sewt-server caddy[5512]: {"level":"info","ts":1748872091.1371958,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["heatblock.skyexplorewt.xyz","admin.skyexplorewt.xyz","skyexplorewt.xyz"]}
juin 02 15:48:11 sewt-server caddy[5512]: {"level":"warn","ts":1748872091.137718,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [heatblock.skyexplorewt.xyz]: no OCSP server specified in certificate","identifiers":["heatblock.skyexplorewt.xyz"]}
juin 02 15:48:11 sewt-server caddy[5512]: {"level":"warn","ts":1748872091.1381803,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [admin.skyexplorewt.xyz]: no OCSP server specified in certificate","identifiers":["admin.skyexplorewt.xyz"]}
juin 02 15:48:11 sewt-server caddy[5512]: {"level":"warn","ts":1748872091.1385694,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [skyexplorewt.xyz]: no OCSP server specified in certificate","identifiers":["skyexplorewt.xyz"]}
juin 02 15:48:11 sewt-server caddy[5512]: {"level":"info","ts":1748872091.1388013,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
juin 02 15:48:11 sewt-server caddy[5512]: {"level":"info","ts":1748872091.138912,"msg":"serving initial configuration"}
juin 02 15:48:11 sewt-server systemd[1]: Started caddy.service - Caddy web server.
juin 02 15:48:11 sewt-server caddy[5512]: {"level":"info","ts":1748872091.1406598,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/var/lib/caddy/.local/share/caddy","instance":"74a0a85c-c9e3-46f5-b569-ece7ba2791c5","try_again":1748958491.1406472,"try_>
juin 02 15:48:11 sewt-server caddy[5512]: {"level":"info","ts":1748872091.1407382,"logger":"tls","msg":"finished cleaning storage units"}
juin 02 15:48:21 sewt-server systemd[1]: Stopping caddy.service - Caddy web server...
juin 02 15:48:21 sewt-server caddy[5512]: {"level":"info","ts":1748872101.6960783,"msg":"shutting down apps, then terminating","signal":"SIGTERM"}
juin 02 15:48:21 sewt-server caddy[5512]: {"level":"warn","ts":1748872101.696193,"msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
juin 02 15:48:21 sewt-server caddy[5512]: {"level":"info","ts":1748872101.6962464,"logger":"http","msg":"servers shutting down with eternal grace period"}
juin 02 15:48:21 sewt-server caddy[5512]: {"level":"info","ts":1748872101.7054718,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
juin 02 15:48:21 sewt-server caddy[5512]: {"level":"info","ts":1748872101.7054968,"msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
juin 02 15:48:21 sewt-server systemd[1]: caddy.service: Deactivated successfully.
juin 02 15:48:21 sewt-server systemd[1]: Stopped caddy.service - Caddy web server.

3. Caddy version:

v2.10.0

4. How I installed and ran Caddy:

a. System environment:

OS: Fedora Linux 42 (Server Edition) x86_64
Host: HP ProDesk 600 G2 DM
Kernel: 6.14.8-300.fc42.x86_64
Uptime: 9 hours, 37 mins
Packages: 1 (npm), 839 (rpm), 11 (brew)
Shell: bash 5.2.37
Editor: VIM - Vi IMproved 9.1
Terminal: /dev/pts/0
CPU: Intel Core i3-6100T (4) @ 3.2GHz
GPU: Intel HD Graphics 530
Memory: 0.61 GiB / 15.49 GiB (3%)
Network: 1 Gbps
Bluetooth: Intel Corp. Bluetooth wireless interface
BIOS: HP 2.58 (07/29/2022)

b. Command:

sudo systemctl start caddy.service

c. Service/unit/compose file:

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

{
        email skyexplorewastaken@gmail.com
}
skyexplorewt.xyz {
        root * /web/
        file_server
}
admin.skyexplorewt.xyz {
        reverse_proxy https://localhost:9090 {
                transport http {
                        tls_insecure_skip_verify
                }
        }
}
heatblock.skyexplorewt.xyz {
        reverse_proxy localhost:3000
}
http://192.168.1.194 {
        root * /web/
        file_server
}
:80, :443 {
        respond "Nah, fuck you." 403
}

5. Links to relevant resources:

2

PS: I already checked permissions for every file, and the file_server works when started with caddy start

3

Move the dir to be under /var

4

Hi ! Thanks for your reply. Is it needed to be under /var specifically or I can put it in a home domain like a /home/caddy, or in a subdomain like /folder/web ? Like does it only needs to not be directly in / ?

5

From what I remember, systemd doesn’t like it when certain directories outside the allowed ones are accessed, and /var is kinda a blessed one (and common standard). I don’t remember the details and nuance. To troubleshoot SELinux’s issue though, try running grep caddy /var/log/audit/audit.log | audit2why for hints.

1 Like
6

I tried putting it in /var/web (just did the mv command) and it didn’t change. I sure reloaded the configuration with sudo systemctl reload caddy.service but my website still says 403 Forbidden.

Actual Caddyfile:

{
        email skyexplorewastaken@gmail.com
}
skyexplorewt.xyz {
        root * /var/web
        file_server
}
admin.skyexplorewt.xyz {
        reverse_proxy https://localhost:9090 {
                transport http {
                        tls_insecure_skip_verify
                }
        }
}
heatblock.skyexplorewt.xyz {
        reverse_proxy localhost:3000
}
http://192.168.1.194 {
        root * /var/web
        file_server
}
:80, :443 {
        respond "Nah, fuck you." 403
}
7

Double check the permissions on the directory and that it has the executable bit enabled. The Caddy systemd service runs as the user and group caddy.

8

I triple checked it, it still isn’t the problem. As I said it must be SELinux since when I disable enforce mode it works. But i don’t know what to do though.

sudo ls -l /var/web/
total 12
drwxr-xr-x. 8 caddy caddy   81 31 mai   04:45 assets
drwxr-xr-x. 2 caddy caddy   42 31 mai   04:45 documents
drwxr-xr-x. 2 caddy caddy   54 31 mai   04:45 errors
drwxr-xr-x. 2 caddy caddy   23 31 mai   04:45 gifs
-rwxr-xr-x. 1 caddy caddy 5304 31 mai   23:37 index.html
drwxr-xr-x. 4 caddy caddy   67 31 mai   04:45 modpacks
drwxr-xr-x. 2 caddy caddy   49 31 mai   04:45 pages
-rwxr-xr-x. 1 caddy caddy  102 25 mai   17:41 public-notepad.txt
drwxr-xr-x. 2 caddy caddy   34 31 mai   04:45 texturepacks
drwxr-xr-x. 2 caddy caddy   47 31 mai   04:45 videos
9

Can you run these two commands?

sudo audit2allow -a
sudo ls -lsaZ /var/web/

Also, @Mohammed90 asked you to run audit2why. What output did you get?

1 Like
10

Since the file with output of audit2why is pretty big, I put it here.

And for your two commands, here you go:

#============= httpd_t ==============
allow httpd_t default_t:dir read;
allow httpd_t default_t:file { getattr open read };

#============= init_t ==============
allow init_t bin_t:file unlink;
allow init_t user_home_t:file { append create execute execute_no_trans ioctl open read };

#!!!! This avc can be allowed using the boolean 'domain_can_mmap_files'
allow init_t user_home_t:file map;

total 16
0 drwxr-xr-x. 10 caddy caddy unconfined_u:object_r:default_t:s0  170 31 mai   23:37 .
4 drwxr-xr-x. 20 root  root  system_u:object_r:var_t:s0         4096  3 juin  08:44 ..
0 drwxr-xr-x.  8 caddy caddy unconfined_u:object_r:default_t:s0   81 31 mai   04:45 assets
0 drwxr-xr-x.  2 caddy caddy unconfined_u:object_r:default_t:s0   42 31 mai   04:45 documents
0 drwxr-xr-x.  2 caddy caddy unconfined_u:object_r:default_t:s0   54 31 mai   04:45 errors
0 drwxr-xr-x.  2 caddy caddy unconfined_u:object_r:default_t:s0   23 31 mai   04:45 gifs
8 -rwxr-xr-x.  1 caddy caddy unconfined_u:object_r:default_t:s0 5304 31 mai   23:37 index.html
0 drwxr-xr-x.  4 caddy caddy unconfined_u:object_r:default_t:s0   67 31 mai   04:45 modpacks
0 drwxr-xr-x.  2 caddy caddy unconfined_u:object_r:default_t:s0   49 31 mai   04:45 pages
4 -rwxr-xr-x.  1 caddy caddy unconfined_u:object_r:default_t:s0  102 25 mai   17:41 public-notepad.txt
0 drwxr-xr-x.  2 caddy caddy unconfined_u:object_r:default_t:s0   34 31 mai   04:45 texturepacks
0 drwxr-xr-x.  2 caddy caddy unconfined_u:object_r:default_t:s0   47 31 mai   04:45 videos

Thanks for trying to help me !

11

Can you please run:

sudo semanage fcontext -a -t httpd_sys_content_t "/var/web(/.*)?"
sudo restorecon -Rv /var/web
sudo ls -lsaZ /var/web

and see if you can run Caddy after that?

2 Likes
12

Yep it works ! Thank you so much !

1 Like