Problems with FileServer permissions when running as systemd

1. The problem I’m having is a file access issue:

You can access the website via “curl -vL https://www.duelify.com”, and the access seems ok. But when you load it via browser the index.html isn’t showing.

http.handlers.file_server	no index file in directory	{"path": "/var/www/html", "index_filenames": ["index.html", "index.txt"]}

2. Error messages and/or full log output:

2024/03/21 10:20:59.639	DEBUG	events	event	{"name": "tls_get_certificate", "id": "dec990a2-1fe0-44ba-a78b-03c3ac088848", "origin": "tls", "data": {"client_hello":{"CipherSuites":[19018,4865,4866,4867,49196,49195,52393,49200,49199,52392,49162,49161,49172,49171,157,156,53,47,49160,49170,10],"ServerName":"www.duelify.com","SupportedCurves":[6682,29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,515,2053,2053,1281,2054,1537,513],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[23130,772,771,770,769],"RemoteAddr":{"IP":"152.37.xx.xxx","Port":29901,"Zone":""},"LocalAddr":{"IP":"91.107.229.214","Port":443,"Zone":""}}}}
2024/03/21 10:20:59.641	DEBUG	tls.handshake	no matching certificates and no custom selection logic	{"identifier": "www.duelify.com"}
2024/03/21 10:20:59.641	DEBUG	tls.handshake	choosing certificate	{"identifier": "*.duelify.com", "num_choices": 1}
2024/03/21 10:20:59.642	DEBUG	tls.handshake	default certificate selection results	{"identifier": "*.duelify.com", "subjects": ["*.duelify.com"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "fea6e45fb3072ee7e2f1547f6c3bd8e374134164340e0a2ed51c69ad6304bb83"}
2024/03/21 10:20:59.642	DEBUG	tls.handshake	matched certificate in cache	{"remote_ip": "152.37.xx.xxx", "remote_port": "29901", "subjects": ["*.duelify.com"], "managed": true, "expiration": "2024/06/19 08:28:13.000", "hash": "fea6e45fb3072ee7e2f1547f6c3bd8e374134164340e0a2ed51c69ad6304bb83"}
2024/03/21 10:20:59.678	DEBUG	http.handlers.file_server	sanitized path join	{"site_root": "/var/www/html", "request_path": "/", "result": "/var/www/html"}
2024/03/21 10:20:59.678	DEBUG	http.handlers.file_server	no index file in directory	{"path": "/var/www/html", "index_filenames": ["index.html", "index.txt"]}
2024/03/21 10:20:59.679	DEBUG	http.log.error	{id=decjqua29} fileserver.(*FileServer).notFound (staticfiles.go:629): HTTP 404	{"request": {"remote_ip": "152.37.xx.xxx", "remote_port": "29901", "client_ip": "152.37.xx.xxx", "proto": "HTTP/2.0", "method": "GET", "host": "www.duelify.com", "uri": "/", "headers": {"Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"], "Sec-Fetch-Site": ["none"], "Accept-Encoding": ["gzip, deflate, br"], "Upgrade-Insecure-Requests": ["1"], "Sec-Fetch-Mode": ["navigate"], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15"], "Accept-Language": ["en-GB,en;q=0.9"], "Sec-Fetch-Dest": ["document"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "www.duelify.com"}}, "duration": 0.00106816, "status": 404, "err_id": "decjqua29", "err_trace": "fileserver.(*FileServer).notFound (staticfiles.go:629)"}

But as you can see the index.html is clearly there. Could it be an access issue?
At first the index.html was 501, staff. Then I changed it to caddy:caddy and it is still not working.

root@DE-NUR-X-P-1:/var/www/html# ls -lt
total 624
-rw-r--r-- 1 caddy caddy   4055 Mar 20 10:56 index.html
-rw-r--r-- 1   501 staff 144568 Nov 30 17:13 4.jpg
-rw-r--r-- 1   501 staff 108383 Nov 30 17:13 3.jpg
-rw-r--r-- 1   501 staff 166949 Nov 30 17:13 2.jpg
-rw-r--r-- 1   501 staff 205044 Nov 30 17:12 1.jpg

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

sudo groupadd --system caddy
sudo useradd --system \
    --gid caddy \
    --create-home \
    --home-dir /var/lib/caddy \
    --shell /usr/sbin/nologin \
    --comment "Caddy web server" \
    caddy
tar -zxvf caddy-linux-amd64.tar.gz
sudo mv caddy /usr/bin/
sudo chown caddy:caddy /usr/bin/caddy
sudo chmod 700 /usr/bin/caddy
sudo mkdir /etc/caddy
sudo mkdir /var/log/caddy
sudo chown caddy:caddy /var/log/caddy

a. System environment:

Debian 12, amd64, systemd

b. Command:

Running it as Daemon (systemd) doesn’t work. But if I run this manually as root as shown below - without changing anything - then it works, this seems to be a file permission problem. But it doesn’t make sense.

systemctl stop caddy
export CLOUDFLARE_API_TOKEN="xxx"
/usr/bin/caddy run --environ --config /etc/caddy/caddy.json

c. Service/unit/compose file:

echo "[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
Environment=CLOUDFLARE_API_TOKEN=${CLOUDFLARE_API_TOKEN}
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/caddy.json
ExecReload=/usr/bin/caddy reload --config /etc/caddy/caddy.json --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target
" | sudo tee /etc/systemd/system/caddy.service >/dev/null

d. My complete Caddy config:

{
  "admin": {
    "disabled": true,
    "config": {
      "persist": false
    }
  },
  "logging": {
    "logs": {
      "default": {
        "writer": {
          "output": "file",
          "filename": "/var/log/caddy/default.log"
        },
        "encoder": {
          "format": "console"
        },
        "level": "DEBUG"
      }
    }
  },
  "apps": {
    "http": {
      "servers": {
        "srvh1": {
          "listen": [":80"],
          "routes": [{
            "handle": [{
              "handler": "static_response",
              "headers": {
                "Location": ["https://{http.request.host}{http.request.uri}"]
              },
              "status_code": 301
            }]
          }],
          "protocols": ["h1"]
        },
        "srvh2": {
          "listen": ["127.0.0.1:7443"],
          "listener_wrappers": [{
            "wrapper": "proxy_protocol"
          },
          {
            "wrapper": "tls"
          },
          {
            "wrapper": "trojan"
          }],
          "routes": [{
            "match": [{
              "path": ["/7d8d47bb"],
              "header": {
                "Connection": ["*Upgrade*"],
                "Upgrade": ["websocket"]
              }
            }],
            "handle": [{
              "handler": "reverse_proxy",
              "upstreams": [{
                "dial": "127.0.0.1:2001"
              }]
            }]
          },
          {
            "match": [{
              "path": ["/521029e5"]
            }],
            "handle": [{
              "handler": "reverse_proxy",
              "transport": {
                "protocol": "http",
                "versions": ["h2c","2"]
              },
              "upstreams": [{
                "dial": "127.0.0.1:2005"
              }]
            }]
          },
          {
            "match": [{
              "protocol": "grpc",
              "path": ["/SALdGZ9k/*"]
            }],
            "handle": [{
              "handler": "reverse_proxy",
              "transport": {
                "protocol": "http",
                "versions": ["h2c","2"]
              },
              "upstreams": [{
                "dial": "127.0.0.1:2011"
              }],
              "headers": {
              	"request": {
                  "set": {
                    "X-Real-IP": ["{http.vars.client_ip}"]
                  }
              	}
              }
            }]
          },
          {
            "handle": [{
              "handler": "forward_proxy",
              "auth_user_deprecated": "user",
              "auth_pass_deprecated": "pass",
              "hide_ip": true,
              "hide_via": true,
              "probe_resistance": {}
            }]
          },
          {
            "handle": [{
              "handler": "trojan",
              "connect_method": true,
              "websocket": true
            }]
          },
          {
            "handle": [{
              "handler": "headers",
              "response": {
                "set": {
                  "Strict-Transport-Security": ["max-age=31536000; includeSubDomains; preload"]
                }
              }
            },
            {
              "handler": "file_server",
              "root": "/var/www/html"
            }]
          }],
          "tls_connection_policies": [{
            "match": {
              "sni": ["83e9c4f0.duelify.com"]
            },
            "protocol_min": "tls1.2",
            "protocol_max": "tls1.2",
            "cipher_suites": ["TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256"],
            "curves": ["secp521r1","secp384r1","secp256r1"]
          },
          {
            "match": {
              "sni": ["www.duelify.com","0755747c.duelify.com"]
            },
            "cipher_suites": ["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256"],
            "curves": ["x25519","secp521r1","secp384r1","secp256r1"]
          }],
          "trusted_proxies": {
            "source": "cloudflare",
            "interval": "12h",
            "timeout": "15s"
          },
          "protocols": ["h1","h2"]
        }
      }
    },
    "trojan": {
      "proxy": {
        "proxy": "no_proxy"
      },
      "upstream": {
        "upstream": "memory"
      },
      "users": ["diy443"]
    },
    "tls": {
      "certificates": {
        "automate": [
          "duelify.com",
          "*.duelify.com"
        ]
      },
      "automation": {
        "policies": [
          {
            "issuers": [
              {
                "module": "acme",
                "email": "info@vc.com",
                "challenges": {
                  "dns": {
                    "provider": {
                      "name": "cloudflare",
                      "api_token": "{env.CLOUDFLARE_API_TOKEN}"
                    }
                  }
                }
              },
              {
                "module": "zerossl",
                "email": "info@vc.com",
                "challenges": {
                  "dns": {
                    "provider": {
                      "name": "cloudflare",
                      "api_token": "{env.CLOUDFLARE_API_TOKEN}"
                    }
                  }
                }
              }
            ]
          }
        ]
      }
    }
  }
}

But if I run this manually as root as shown below - without changing anything - then it works, this seems to be a file permission problem. But it doesn’t make sense.

UPDATE: That’s not true actually. It doesn’t work either way. I keep digging…

Right now, it seems like your server is not accepting connections.

1 Like

I agree with @francislavoie assessment.
Both Ports 80 & 443 appear to be filtered, meaning not available from the Internet.

$ nmap -Pn -p80,443 www.duelify.com
Starting Nmap 7.80 ( https://nmap.org ) at 2024-03-22 02:44 UTC
Nmap scan report for www.duelify.com (91.107.229.214)
Host is up.
rDNS record for 91.107.229.214: static.214.229.107.91.clients.your-server.de

PORT    STATE    SERVICE
80/tcp  filtered http
443/tcp filtered https

Nmap done: 1 IP address (1 host up) scanned in 3.47 seconds

And from around the world Permanent link to this check report “Connection timed out”.

Sorry guys. I should have considered the time zone difference between us. I gave up at the end and deleted the server before going to bed. Hence no connection.

I hope the detailed logs are somewhat useful, but no worries.

Thank you

1 Like

No problem @houmie :slight_smile: