How can I serve symbolically linked files?

webdev · May 15, 2024, 2:02pm

1. The problem I’m having:

I’m trying to serve media files that reside outside of the site root by creating a symbolic link within the site root to the file outside, but it isn’t working. It does work with hard links, but I need symbolic links.

Symbolic link example (not working):
curl https://satoshidnc.com/media/valid-for-7-days/DMHC-E01-fa9de3c5abca8a667bbb4d703cd3f888d1bf5e7ab1306e8195e9d293d43b3fd6.mp3

Hardlink example (working):
curl https://satoshidnc.com/media/valid-for-7-days/DMHC-E01-f927f9cc6a199d6681a29625a1675d7485340389e519408e7eff91499fc76470.mp3

I verified that the caddy user has access to the target of the symbolic link using these commands:

$ groups caddy
caddy : caddy www-data mediastore
$ runuser -u caddy -- ls /home/mediastore/DMHC/E01.mp3
-rw-r--r-- 6 mediastore mediastore 3592717 May 14 08:16 /home/mediastore/DMHC/E01.mp3

as well as with the following (which messes up the terminal due to binary content):
$ runuser -u caddy -- cat /home/mediastore/DMHC/E01.mp3

2. Error messages and/or full log output:

[the problem does not result in any entries reported by journalctl -u caddy --no-pager | less +G]

The following is reported by curl -vL https://satoshidnc.com/media/valid-for-7-days/DMHC-E01-fa9de3c5abca8a667bbb4d703cd3f888d1bf5e7ab1306e8195e9d293d43b3fd6.mp3

*   Trying 127.0.0.1:443...
* Connected to satoshidnc.com (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=satoshidnc.com
*  start date: Apr 16 23:47:43 2024 GMT
*  expire date: Jul 15 23:47:42 2024 GMT
*  subjectAltName: host "satoshidnc.com" matched cert's "satoshidnc.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x5575daadeeb0)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /media/valid-for-7-days/DMHC-E01-fa9de3c5abca8a667bbb4d703cd3f888d1bf5e7ab1306e8195e9d293d43b3fd6.mp3 HTTP/2
> Host: satoshidnc.com
> user-agent: curl/7.81.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200
< accept-ranges: bytes
< access-control-allow-origin: *
< alt-svc: h3=":443"; ma=2592000
< content-type: text/html; charset=utf-8
< etag: "sbg1tcg1"
< last-modified: Fri, 05 Apr 2024 00:39:12 GMT
< server: Caddy
< content-length: 577
< date: Wed, 15 May 2024 13:16:01 GMT
<
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection #0 to host satoshidnc.com left intact
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="/favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="Satoshi, D.N.C. Community Website"/><link rel="apple-touch-icon" href="/logo192.png"/><link rel="manifest" href="/manifest.json"/><title>Satoshi, D.N.C.</title><script defer="defer" src="/static/js/main.0189c2c4.js"></script><link href="/static/css/main.1ebb28a4.css" rel="stylesheet"></head><body><div id="root"></div></body></html>

By contrast, it should report something like below from the hardlink example (note the warning about binary output): curl -vL https://satoshidnc.com/media/valid-for-7-days/DMHC-E01-f927f9cc6a199d6681a29625a1675d7485340389e519408e7eff91499fc76470.mp3

*   Trying 127.0.0.1:443...
* Connected to satoshidnc.com (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=satoshidnc.com
*  start date: Apr 16 23:47:43 2024 GMT
*  expire date: Jul 15 23:47:42 2024 GMT
*  subjectAltName: host "satoshidnc.com" matched cert's "satoshidnc.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x55dbc087beb0)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /media/valid-for-7-days/DMHC-E01-f927f9cc6a199d6681a29625a1675d7485340389e519408e7eff91499fc76470.mp3 HTTP/2
> Host: satoshidnc.com
> user-agent: curl/7.81.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200
< accept-ranges: bytes
< access-control-allow-origin: *
< alt-svc: h3=":443"; ma=2592000
< content-type: audio/mpeg
< etag: "sdh62v2505p"
< last-modified: Tue, 14 May 2024 12:16:07 GMT
< server: Caddy
< content-length: 3592717
< date: Wed, 15 May 2024 13:15:35 GMT
<
* TLSv1.2 (IN), TLS header, Supplemental data (23):
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
* Failure writing output to destination
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* stopped the pause stream!
* Connection #0 to host satoshidnc.com left intact

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

a. System environment:

$ uname -a
Linux satoshidnc.com 5.15.0-105-generic #115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

b. Command:

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

c. Service/unit/compose file:

$ cat /etc/systemd/system/caddy.service
# caddy.service

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateDevices=yes
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

$ caddy fmt
# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace ":80" below with your
# domain name.

#:80 {
#       # Set this path to your site's directory.
#       root * /usr/share/caddy
#
#       # Enable the static file server.
#       file_server
#
#       # Another common task is to set up a reverse proxy:
#       # reverse_proxy localhost:8080
#
#       # Or serve a PHP site through php-fpm:
#       # php_fastcgi localhost:9000
#}

(cors) {
        @cors_preflight{args.0} method OPTIONS
        @cors{args.0} header Origin {args.0}

        handle @cors_preflight{args.0} {
                header {
                        Access-Control-Allow-Origin "{args.0}"
                        Access-Control-Allow-Methods "GET, POST, PUT, PATCH, DELETE, OPTIONS"
                        Access-Control-Allow-Headers *
                        Access-Control-Max-Age "3600"
                        defer #turn on defer on your header directive to make sure the new header values are set after proxying
                }
                respond "" 204
        }

        handle @cors{args.0} {
                header {
                        Access-Control-Allow-Origin "{args.0}"
                        Access-Control-Expose-Headers *
                        defer
                }
        }
}

(cors_all) {
        @cors_preflight method OPTIONS

        header {
                #               Access-Control-Allow-Origin "{header.origin}"
                Access-Control-Allow-Origin "*"
                Vary Origin
                Access-Control-Expose-Headers "Authorization"
                Access-Control-Allow-Credentials "true"
        }

        handle @cors_preflight {
                header {
                        Access-Control-Allow-Methods "GET, POST, PUT, PATCH, DELETE"
                        Access-Control-Max-Age "3600"
                }
                respond "" 204
        }
}

satoshidnc.com {
        handle /.well-known/lnurlp/* {
                #    root * /var/www
                #    file_server
                reverse_proxy 127.0.0.1:5000
                #    import cors_all {header.origin}
                #    import cors https://ng.satoshidnc.com
                #    import cors https://dev-ng.satoshidnc.com
        }
        handle /lnurlp/* {
                reverse_proxy 127.0.0.1:5000
        }
        handle /.well-known/nostr.json* {
                rewrite /.well-known/nostr.json* /nostrnip5/api/v1/domain/hash/nostr.json
                reverse_proxy 127.0.0.1:5000
        }
        handle /message {
                reverse_proxy 127.0.0.1:9091
        }
        handle /media/* {
                header Access-Control-Allow-Origin "*"
                root * /var/www
                try_files {path} /index.html
                file_server
        }
        handle {
                root * /var/www
                try_files {path} /index.html
                file_server
        }
}
ng.satoshidnc.com {
        handle /.well-known/lnurlp/* {
                root * /var/www/ng
                file_server
                import cors_all {header.origin}
        }
        handle /emoji/96/* {
                #    header Access-Control-Allow-Origin "*"
                root * /var/www/ng
                file_server
        }
        handle {
                root * /var/www/ng
                try_files {path} /index.html
                file_server
        }
}
dev-ng.satoshidnc.com {
        handle /.well-known/lnurlp/* {
                root * /var/www/dev-ng
                file_server
                import cors_all {header.origin}
        }
        handle /emoji/96/* {
                #    header Access-Control-Allow-Origin "*"
                root * /var/www/dev-ng
                file_server
        }
        handle /api/v1/* {
                root * /var/www/dev-ng
                file_server
        }
        handle {
                root * /var/www/dev-ng/web
                try_files {path} /index.html
                file_server
        }
}
mercado.satoshidnc.com {
        @mercado_root {
                path /
                query ""
        }
        templates
        redir @mercado_root /nostrmarket/market?naddr=hash#/
        rewrite / /nostrmarket/market
        reverse_proxy 127.0.0.1:5000
}
lnbits.satoshidnc.com {
        handle /3rd-party/* {
                root * /var/www
                file_server
        }
        handle /api/v1/payments/sse* {
                reverse_proxy 127.0.0.1:5000 {
                        transport http {
                                keepalive off
                                compression off
                        }
                }
        }
        @corsable header_regexp host Referer (.*)/
        header @corsable {
                >Access-Control-Allow-Origin {re.host.1}
                #https://dev-ng.satoshidnc.com
        }
        reverse_proxy 0.0.0.0:5000 {
        }
}
relay.satoshidnc.com {
        reverse_proxy 127.0.0.1:7777
}

# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile
Error: Caddyfile:26: Caddyfile input is not formatted; Tip: use '--overwrite' to update your Caddyfile in-place instead of previewing it. Consult '--help' for more options

5. Links to relevant resources:

francislavoie · May 15, 2024, 10:14pm

If you’re running Caddy as a systemd service, it doesn’t have permissions to read files in /home, because Caddy runs as the caddy user, which won’t have permission to read inside of other users’ home directories.

Move your files somewhere else, like /srv or /var/uploads or something like that.

webdev · May 19, 2024, 2:04pm

So it’s a permission issue. I preferred to solve it by opening mediastore’s read/search permissions for “others”, but I’m puzzled as to why this was necessary since I already added the caddy user to the mediastore group and even verified accessibility of the file as caddy via the runuser command. Apparently I’m not fully understanding Linux file permissions, lol. Anyhow, thanks for your response, which got me back on track.

system · June 18, 2024, 2:04pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.