Caddy 2 routing for subdirectories

1. Caddy version (caddy version):

v2.0.0 h1:pQSaIJGFluFvu8KDGDODV8u4/QRED/OPyIR+MWYYse8=

2. How I run Caddy:

Note: My binary is built with the following modules:

  • Standard
  • Cloudflare
  • NTLM-Transport
  • Prometheus

a. System environment:

Caddy: Native Install
Hypervisor: Hyper-V
OS: Ubuntu Server 18.04.4

b. Command:

caddy run, caddy start, caddy stop

c. Service/unit/compose file:

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

{
    email certs@alexsguardian.net
#    acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
    default_sni centaurus
    admin localhost:2019
}
(header) {
        header / {
            Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
            X-Xss-Protection "1; mode=block"
            X-Content-Type-Options "nosniff"
            X-Frame-Options "DENY"
            Content-Security-Policy "upgrade-insecure-requests"
            Referrer-Policy "strict-origin-when-cross-origin"
            Cache-Control "public, max-age=15, must-revalidate"
            Feature-Policy "accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'self'; camera 'none'; encrypted-media 'none'; fullscreen 'self'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; midi 'none'; payment 'none'; picture-in-picture *; speaker 'none'; sync-xhr 'none'; usb 'none'; vr 'none'"
       }
}
(tls) {
    tls {
        dns cloudflare <token>
    }
}
lab.alexsguardian.net {
#        import header
        import tls
        log {
            output file /var/log/caddy/lab.log {
                roll_size 50mb
                roll_keep 5
                roll_keep_for 80h
            }
        }
        route / {
            reverse_proxy 192.168.9.2:3000
        }
        route /portainer {
            uri strip_prefix /portainer
            reverse_proxy 192.168.9.8:9000/portainer/
        }
        route /portainer/api/websocket/ {
            uri strip_prefix /portainer
            reverse_proxy 192.168.9.8:9000/api/websocket/
        }
        route /sonarr {
            reverse_proxy 10.8.8.79:8989
        }
        route /radarr {
            reverse_proxy 10.8.8.79:7878
        }
        route /tautulli {
            reverse_proxy 192.168.9.8:8181
        }
}

3. The problem I’m having:

I am trying to use routes to direct subdirectories to different services. All services (except portainer, which I don’t really care for at the moment) support subdirectories and are already configured on their side.

4. Error messages and/or full log output:

Latest Error on current config:

2020/05/07 20:50:30.871 info    http.log.access.log0    handled request {"request": {"method": "GET", "uri": "/radarr/login?returnUrl=/radarr/", "proto": "HTTP/1.1", "remote_addr": "172.69.50.51:31444", "host": "lab.alexsguardian.net", "headers": {"Cdn-Loop": ["cloudflare"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36 Edg/81.0.416.68"], "Sec-Fetch-Dest": ["document"], "Connection": ["Keep-Alive"], "Cf-Visitor": ["{\"scheme\":\"https\"}"], "Cache-Control": ["max-age=0"], "Upgrade-Insecure-Requests": ["1"], "Sec-Fetch-Mode": ["navigate"], "Accept-Language": ["en,en-US;q=0.9"], "Cf-Request-Id": ["02928085470000f17eb30d5200000001"], "Cf-Ipcountry": ["US"], "X-Forwarded-For": ["73.130.234.167"], "X-Forwarded-Proto": ["https"], "Dnt": ["1"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"], "Sec-Fetch-Site": ["cross-site"], "Sec-Fetch-User": ["?1"], "Cookie": ["tautulli_token_7d5762e1f10743c2a9c8746fd1aa071a=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyX2lkIjoxMzQxMzcyNCwidXNlciI6ImFsZXhhbmR6b3JzIiwiZXhwIjoxNTg5NjQ2NTIxLCJ1c2VyX2dyb3VwIjoiYWRtaW4ifQ.KD5ZAns20VfSz7VafOqHCTmQJFhN8cI0323Y2txIEnQ; __cfduid=d372c2274151a19adff0e04412003d9e71587138771; grafana_session=4af1617776b262b658a938b56f840e41"], "Accept-Encoding": ["gzip"], "Cf-Ray": ["58fdd04edfeff17e-PIT"], "Cf-Connecting-Ip": ["73.130.234.167"]}, "tls": {"resumed": false, "version": 772, "ciphersuite": 4865, "proto": "", "proto_mutual": true, "server_name": "lab.alexsguardian.net"}}, "common_log": "172.69.50.51 - - [07/May/2020:20:50:30 +0000] \"GET /radarr/login?returnUrl=/radarr/ HTTP/1.1\" 0 0", "duration": 0.000051801, "size": 0, "status": 0, "resp_headers": {"Server": ["Caddy"]}}
2020/05/07 20:50:34.370 info    http.log.access.log0    handled request {"request": {"method": "GET", "uri": "/tautulli/api/v2?apikey=183694165a7a404d8e2e7eec518ff851&cmd=get_activity", "proto": "HTTP/1.1", "remote_addr": "172.69.50.9:53816", "host": "lab.alexsguardian.net", "headers": {"User-Agent": ["python-requests/2.21.0"], "Accept": ["*/*"], "Connection": ["Keep-Alive"], "Accept-Encoding": ["gzip"], "X-Forwarded-Proto": ["https"], "Cookie": ["__cfduid=d2aef4f8d672bd9de92f546c7c7b84e641588882020; redirect_to=%2Ftautulli%2Fapi%2Fv2%3Fapikey%3D183694165a7a404d8e2e7eec518ff851%26cmd%3Dget_activity"], "Cdn-Loop": ["cloudflare"], "Cf-Connecting-Ip": ["73.130.234.167"], "Cf-Visitor": ["{\"scheme\":\"https\"}"], "Cf-Request-Id": ["02928092bd0000f186cf32d200000001"], "Cf-Ipcountry": ["US"], "X-Forwarded-For": ["73.130.234.167"], "Cf-Ray": ["58fdd0646c58f186-PIT"]}, "tls": {"resumed": false, "version": 772, "ciphersuite": 4865, "proto": "", "proto_mutual": true, "server_name": "lab.alexsguardian.net"}}, "common_log": "172.69.50.9 - - [07/May/2020:20:50:34 +0000] \"GET /tautulli/api/v2?apikey=183694165a7a404d8e2e7eec518ff851&cmd=get_activity HTTP/1.1\" 0 0", "duration": 0.0000287, "size": 0, "status": 0, "resp_headers": {"Server": ["Caddy"]}}

Radarr and Sonarr both return blank pages and the url in full is: https://lab.alexsguardian.net/radarr/login?returnUrl=/radarr/
https://lab.alexsguardian.net/sonarr/login?returnUrl=/sonarr/

Tautulli is just a blank page with the full url:
https://lab.alexsguardian.net/tautulli/

5. What I already tried:

I’ve tried the following route setups. Only using one route as an example.

route /sonarr {
    reverse_proxy 10.8.8.79:9898
}

route /sonarr/ {
     reverse_proxy 10.8.8.79:9898/sonarr
}

route /sonarr {
    reverse_proxy 10.8.8.79/sonarr/
}


6. Links to relevant resources:

Old v1 config for same domain:

lab.alexsguardian.net {
        gzip
        tls certs@alexsguardian.net {
                dns cloudflare
        }
#        header / {
#            Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
#            X-Xss-Protection "1; mode=block"
#            X-Content-Type-Options "nosniff"
#            X-Frame-Options "DENY"
#            Content-Security-Policy "upgrade-insecure-requests"
#            Referrer-Policy "strict-origin-when-cross-origin"
#            Cache-Control "public, max-age=15, must-revalidate"
#            Feature-Policy "accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'self'; camera 'none'; encrypted-media 'none'; fullscreen 'self'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; midi 'none'; payment 'none'; picture-in-picture *; speaker 'none'; sync-xhr 'none'; usb 'none'; vr 'none'"
#       }
        errors /srv/logs/lab/errors/E!-lab-agnet.log {
                rotate_size 50
                rotate_age 90
                rotate_keep 20
                rotate_compress
                502 /srv/erpages/500.html
        }
        log / /srv/logs/lab/lab-agnet.log {
                rotate_size 50
                rotate_age 90
                rotate_keep 20
                rotate_compress
        }
        proxy / http://192.168.9.2:3000 {
                transparent
                websocket
        }
        proxy /portainer/ 10.8.8.52:9000 {
                without /portainer
                transparent
                header_upstream -Connection
        }
        proxy /portainer/api/websocket/ portainer:9000 {
               without /portainer
                transparent
                websocket
        }
        proxy /sonarr 10.8.8.79:8989 {
                transparent
                websocket
        }
        proxy /radarr 10.8.8.79:7878 {
                transparent
                websocket
        }
        proxy /tautulli 192.168.9.8:8181 {
                transparent
                websocket
        }

Also note the first route works (port 3000 one).

I have no idea if I am using routes right so any help would be appreciated. Would also prefer not to setup subdomains for these apps as I’d like to keep them under my lab subdomain.

You’ve run into a bit of a snag with the paths.

In Caddy v1, all those paths are evaluated as base paths / path prefixes. So a proxy / http://192.168.9.2:3000 proxied any request. proxy /portainer/ 10.8.8.52:9000 would proxy /portainer/ but also /portainer/foo, /portainer/bar/stuff/ etc.

However, Caddy v2 is different:

  • Where you matched requests by path prefix in Caddy 1, path matching is now exact by default in Caddy 2. If you want to match a prefix like /foo/ , you’ll need /foo/* in Caddy 2.

—Upgrading to Caddy 2 — Caddy Documentation

So this snippet:

Only EVER acts on the exact request, https://lab.alexsguardian.net/portainer and never /portainer/ or /portainer/foo.

Same is true with all other snippets.

This part in particular, though, is weird:

Specifically, upstream URLs do not support paths. This Caddyfile should be generating errors when you try to load it. Something like: Error during parsing: for now, URLs for proxy upstreams only support scheme, host, and port components.

1 Like

Thanks for the info! And yeah it generates errors if I have an upstream URL added and try to request the resource.

So ignoring portainer (not sure why I even bothered bringing that over to the new config), is it possible to get radarr, sonarr, and tautulli working then?

Because if I have:

route /radarr {
    reverse_proxy 10.8.8.79:7878
}

and then request it via the full url:
https://lab.alexsguardian.net/radarr

I get a full white blank page but the URL updates to:
https://lab.alexsguardian.net/radarr/login?returnUrl=/radarr/

Okay, I’m a goof. I got Radarr, Sonarr and Tautulli working. However I can’t seem to get Grafana to load now on the base subdomain.

lab.alexsguardian.net {
        import header
        import tls
        log {
            output file /var/log/caddy/lab.log {
                roll_size 50mb
                roll_keep 5
                roll_keep_for 80h
            }
        }
        route /sonarr* {
            reverse_proxy 10.8.8.79:8989
        }
        route /radarr* {
            reverse_proxy 10.8.8.79:7878
        }
        route /tautulli* {
            reverse_proxy 192.168.9.8:8181
        }
        route /* {
            reverse_proxy 192.168.9.2:3000
        }
}

I have tried

route * {
    reverse_proxy 192.168.9.2:3000
}

route /* {
    reverse_proxy 192.168.9.2:3000
}

route / {
    reverse_proxy 192.168.9.2:3000
}

reverse_proxy 192.168.9.2:3000

What are you seeing instead?

root_url and subpath settings haven’t changed on the Grafana side. Its still expecting lab.alexsguardian.net

Seems Caddy isnt passing traffic properly to it.

Relevent portions of my grafana.ini:

# The public facing domain name used to access grafana from a browser
domain = lab.alexsguardian.net

# The full public facing url you use in browser, used for redirects and emails
# If you use reverse proxy and sub path specify full url (with sub path)
root_url = https://%(domain)s/

To narrow things down, open the network tab in your browser’s developer tools and look at some of the requests that are made. It seems like some of the JS assets aren’t loading for some reason.

Yeah not sure whats going on. Seems the JS is loading but its still popping that failed config page.

Could it be the route that I messed up? Do I need to have a route for the base domain if I have other routes specified? Still trying to understand how Caddy works with this new config.

Any errors in the console tab?

Thats all I’ve got. Keep in mind that these showed while using Caddy v1 and Grafana was loading properly. Also before I added the routes I was able to get Grafana to load but after adding them, well here we are atm.

That’s quite strange, I think that should work.

Of those, only route / is incorrect. All the others should work.

Could you check your grafana logs? There might be something there to help.

I’m not seeing anything in the grafana.log. Its mostly startup stuff and all the DB queries. The only request I saw in there was when I did a local query via the IP:PORT of the machine.

{"logger":"context","lvl":"info","method":"GET","msg":"Request Completed","orgId":0,"path":"/","referer":"","remote_addr":"10.8.8.113","size":29,"status":302,"t":"2020-05-08T03:19:08.183392325Z","time_ms":0,"uname":"","userId":0}

Which gave me the IP:PORT/login page:


I guess you were right though. Looks like its not correctly loading app files. This defiently is on Caddy’s side though, so somehwere in my config is messed up.

So I commented out the other routes and only had

lab.alexsguardian.net {
#        import header
        import tls
        log {
            output file /var/log/caddy/lab.log {
                roll_size 50mb
                roll_keep 5
                roll_keep_for 80h
            }
        }
        reverse_proxy 192.168.8.2:3000
}

and am still getting the same error page…

With that latest Caddyfile active, open up your web console to the Network tab, tick “Disable Cache”, and then reload the page.

All of those JS files are 62 bytes. Why is Caddy responding with 62 bytes every time?

Can you click one of those files and show us exactly where the request went and exactly what the response looks like? Headers, and content wise?

Headers for: RxZJdnzeo3R5zSexge8UUVtXRa8TVwTICgirnJhmVJw.woff2

Request URL: https://lab.alexsguardian.net/public/fonts/roboto/RxZJdnzeo3R5zSexge8UUVtXRa8TVwTICgirnJhmVJw.woff2
Request Method: GET
Status Code: 200 
Remote Address: 104.18.46.202:443
Referrer Policy: no-referrer-when-downgrade
accept-ranges: bytes
age: 5694
cache-control: max-age=14400
cf-cache-status: HIT
cf-ray: 590042d4bf5ba3e5-PIT
cf-request-id: 02940818ee0000a3e5372e3200000001
content-length: 0
date: Fri, 08 May 2020 03:58:13 GMT
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
status: 200
vary: Accept-Encoding
:authority: lab.alexsguardian.net
:method: GET
:path: /public/fonts/roboto/RxZJdnzeo3R5zSexge8UUVtXRa8TVwTICgirnJhmVJw.woff2
:scheme: https
accept: */*
accept-encoding: gzip, deflate, br
accept-language: en,en-US;q=0.9
cache-control: no-cache
cookie: tautulli_token_7d5762e1f10743c2a9c8746fd1aa071a=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyX2lkIjoxMzQxMzcyNCwidXNlciI6ImFsZXhhbmR6b3JzIiwiZXhwIjoxNTg5NjQ2NTIxLCJ1c2VyX2dyb3VwIjoiYWRtaW4ifQ.KD5ZAns20VfSz7VafOqHCTmQJFhN8cI0323Y2txIEnQ; __cfduid=d372c2274151a19adff0e04412003d9e71587138771; grafana_session=bd9b647bb9d3e524f617300b5aca2af7
dnt: 1
origin: https://lab.alexsguardian.net
pragma: no-cache
referer: https://lab.alexsguardian.net/
sec-fetch-dest: font
sec-fetch-mode: cors
sec-fetch-site: same-origin
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36 Edg/81.0.416.68

Response:

What about response headers? Nevermind, they’re there, looks like.

And can you share the same info for one of those JS files, too?

Yeah I did a CTRL+A in the Headers tab. Unfortunetly it does not bring the splitters. lol

Request URL: https://lab.alexsguardian.net/public/build/runtime.b63fb8634611947fae1b.js
Request Method: GET
Status Code: 200 
Remote Address: 104.18.46.202:443
Referrer Policy: no-referrer-when-downgrade
accept-ranges: bytes
age: 5910
cache-control: max-age=14400
cf-cache-status: HIT
cf-ray: 5900481b685ea3e5-PIT
cf-request-id: 02940b651f0000a3e5372dd200000001
content-length: 0
date: Fri, 08 May 2020 04:01:49 GMT
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
status: 200
vary: Accept-Encoding
:authority: lab.alexsguardian.net
:method: GET
:path: /public/build/runtime.b63fb8634611947fae1b.js
:scheme: https
accept: */*
accept-encoding: gzip, deflate, br
accept-language: en,en-US;q=0.9
cache-control: no-cache
cookie: tautulli_token_7d5762e1f10743c2a9c8746fd1aa071a=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyX2lkIjoxMzQxMzcyNCwidXNlciI6ImFsZXhhbmR6b3JzIiwiZXhwIjoxNTg5NjQ2NTIxLCJ1c2VyX2dyb3VwIjoiYWRtaW4ifQ.KD5ZAns20VfSz7VafOqHCTmQJFhN8cI0323Y2txIEnQ; __cfduid=d372c2274151a19adff0e04412003d9e71587138771; grafana_session=bd9b647bb9d3e524f617300b5aca2af7
dnt: 1
pragma: no-cache
referer: https://lab.alexsguardian.net/
sec-fetch-dest: script
sec-fetch-mode: no-cors
sec-fetch-site: same-origin
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36 Edg/81.0.416.68

For https://lab.alexsguardian.net/public/build/runtime.b63fb8634611947fae1b.js

Your request isn’t going to Caddy, it’s going to Cloudflare, and Cloudflare is cache hitting it. Cloudflare cached a zero byte response. Purge your Cloudflare cache for your site.

Same result for your JS.

I expect Cloudflare cached these zero byte responses from your origin back when you were testing prior to learning about the exact-path routing.

Strong suggestion: don’t use caching features when deploying, configuring, or troubleshooting web servers…

3 Likes