Caddy + Jellyfin + Inlets Tunnel

Jeff_Untalan · July 15, 2021, 9:12am

1. Caddy version (`caddy version`):

v2.4.3 h1:Y1FaV2N4WO3rBqxSYA8UZsZTQdN+PwcoOcAiZTM8C0I=

2. How I run Caddy:

Digital Ocean VPS

a. System environment:

Ubuntu 20.04, systemd

b. Command:

default ubuntu install

c. Service/unit/compose file:

default ubuntu unstall

d. My complete Caddyfile or JSON config:

pi.leveloneadventure.me {
    # PROXY ALL REQUEST TO PORT 1717
    reverse_proxy localhost:1717
}
books.leveloneadventure.me {
    # PROXY ALL REQUEST TO PORT 8083
    reverse_proxy localhost:8083
}
media.leveloneadventure.me {
    # PROXY ALL REQUEST TO PORT 8096
    reverse_proxy localhost:8096
}

3. The problem I’m having:

Hi there. I’ve been trying to make my Jellyfin install available outside of my home network. I’m using inlets to tunnel into a Digital Ocean droplet so I could use that machine’s IP for my stuff. It’s worked well for my FoundryVTT install (at pi.leveloneadventure.me) and my calibre-web install (books.leveloneadventure.me) but for Jellyfin I can’t seem to get it working on media.leveloneadventure.me via Caddy. I can access my Jellyfin through my the IP:port address of my droplet, but for some reason it won’t load the custom domain.

4. Error messages and/or full log output:

{"level":"error","ts":1626338882.3964524,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"136.158.31.6:616","proto":"HTTP/2.0","method":"GET","host":"media.leveloneadventure.me","uri":"/web/index.html","headers":{"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Sec-Ch-Ua-Mobile":["?0"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Edg/91.0.864.67"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Dest":["document"],"Sec-Ch-Ua":["\" Not;A Brand\";v=\"99\", \"Microsoft Edge\";v=\"91\", \"Chromium\";v=\"91\""],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["en-US,en;q=0.9"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"media.leveloneadventure.me"}},"duration":0.10731233,"status":502,"err_id":"cc3thj7gh","err_trace":"reverseproxy.statusError (reverseproxy.go:857)"}

5. What I already tried:

I’ve tried redoing my inlets install on my raspberry pi on my home network and retraced my steps when I initially setup my foundry and calibre-web which I replicated when setting up jellyfin. I’ve tried redoing my caddyfile but my efforts there may be limited to what I only know about reverse proxies (which is not a lot). I can access Jellyfin using my droplet’s IP address, and my inlets seems to be doing fine as per the log:

Jul 15 17:07:37 raspberrypi inlets[14613]: 2021/07/15 17:07:37 Starting client - version 3.0.2
Jul 15 17:07:37 raspberrypi inlets[14613]: 2021/07/15 17:07:37 Upstream:  => http://127.0.0.1:8096
Jul 15 17:07:37 raspberrypi inlets[14613]: time="2021/07/15 17:07:37" level=info msg="Connecting to proxy" url="wss://media.leveloneadventure.me/tunnel"

I’m almost about to give up and just settle on using Jellyfin using the droplet IP. I’m not very experienced with this. Any help would be appreciated!

6. Links to relevant resources:

Whitestrake · July 15, 2021, 11:39pm

Howdy @Jeff_Untalan, welcome to the Caddy community!

These can change over time and I run my Caddy in Docker or compiled from source, so I’m afraid I don’t have easy access to exactly what these will be for you. If you can retrieve them and add them to your post, that would be quite helpful for easy reference, and especially for posterity.

The command itself should be in the unit file, which you can view with systemctl cat caddy.service (or similar).

Getting more information on exactly how the page is failing to load is going to be critical to determining what steps are required to troubleshoot.

Specifically:

What action did you take? (e.g. navigate to media.leveloneadventure.me in my browser, Edge)
What result did you expect? (e.g. Jellyfin interface)
What result did you get instead? (What behaviour did you observe, what specific errors were displayed, etc - CTRL+C V or screenshots are awesome for this)

P.S.:

We see people use domain.com all the time - I’d like to encourage you to avoid this behaviour, since someone actually owns that domain and we have actually had people come to our forums with complaints in the past for other domains because they’ve been misused as placeholders.

The example.com and example.net domains are reserved specifically for purpose, and so is the .example TLD (e.g. mydomain.example) so if you absolutely must redact your own domain, we recommend using one of those.

Whitestrake · July 15, 2021, 11:54pm

P.S.S.:

I note that while the pi and books subdomains appear to have valid public DNS records, media does not. If that’s unintentional, you’ll probably need to address that first before you can connect.

I was able to access the HTTP redirector for Caddy at the expected domain with manual resolution to the IP address of the other two subdomains:

➜ curl -kIL http://media.leveloneadventure.me --resolve media.leveloneadventure.me:443:128.199.121.115 --resolve media.leveloneadventure.me:80:128.199.121.115
HTTP/1.1 308 Permanent Redirect
Connection: close
Location: https://media.leveloneadventure.me/
Server: Caddy
Date: Thu, 15 Jul 2021 23:51:06 GMT

curl: (35) error:14004438:SSL routines:CONNECT_CR_SRVR_HELLO:tlsv1 alert internal error

The HTTPS error occurs because Caddy presumably was unable to requisition a valid certificate and its default behaviour when no cert exists for a domain is to simply refuse to send any certificate, which results in a closed connection with a TLS error. I imagine once a DNS entry exists, things will clear right up.

Jeff_Untalan · July 16, 2021, 1:38am

Hi there, I replaced any references to the other domain from the main post. Sorry about that.

I removed the DNS record while I was trying to troubleshoot it. I re-created an A Record pointing to my droplet’s IP with subdomain media.

Here is my unit file for the caddy service:

#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

For the actions I took,

I navigate to media.leveloneadventure.me using Edge.
I expected to see the Jellyfin interface properly loaded with that address.
I got a 502 and the URL changed to https://media.leveloneadventure.me/web/index.html

image802×416 10.5 KB

After that, my caddy log now shows:

Jul 16 01:18:05 sparefoundry systemd[1]: Started Caddy.
Jul 16 01:18:05 sparefoundry caddy[6677]: {"level":"info","ts":1626398285.6863122,"msg":"serving initial configuration"}
Jul 16 01:18:05 sparefoundry caddy[6677]: {"level":"info","ts":1626398285.6865227,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Jul 16 01:18:05 sparefoundry caddy[6677]: {"level":"info","ts":1626398285.6941977,"logger":"tls","msg":"finished cleaning storage units"}
Jul 16 01:19:04 sparefoundry caddy[6677]: {"level":"error","ts":1626398344.657801,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"136.158.31.6:54670","proto":"HTTP/2.0","method":"GET","host":"media.leveloneadventure.me","uri":"/web/index.html","headers":{"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Sec-Ch-Ua":["\" Not;A Brand\";v=\"99\", \"Microsoft Edge\";v=\"91\", \"Chromium\";v=\"91\""],"Accept-Language":["en-US,en;q=0.9"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Edg/91.0.864.67"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Sec-Ch-Ua-Mobile":["?0"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"media.leveloneadventure.me"}},"duration":0.16292844,"status":502,"err_id":"wd65u6g13","err_trace":"reverseproxy.statusError (reverseproxy.go:857)"}
Jul 16 01:33:53 sparefoundry caddy[6677]: {"level":"error","ts":1626399233.891708,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"136.158.31.6:18811","proto":"HTTP/2.0","method":"GET","host":"media.leveloneadventure.me","uri":"/web/index.html","headers":{"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Site":["none"],"Sec-Ch-Ua":["\" Not;A Brand\";v=\"99\", \"Microsoft Edge\";v=\"91\", \"Chromium\";v=\"91\""],"Sec-Ch-Ua-Mobile":["?0"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Edg/91.0.864.67"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["en-US,en;q=0.9"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"media.leveloneadventure.me"}},"duration":0.111645616,"status":502,"err_id":"m74j07817","err_trace":"reverseproxy.statusError (reverseproxy.go:857)"}

However, I can access the Jellyfin interface using my droplet’s IP. http://128.199.121.115:8096/

Whitestrake · July 16, 2021, 1:52am

What’s really weird here is that it responds perfectly fine to curl:

HTTP/1.1 308 Permanent Redirect
Connection: close
Location: https://media.leveloneadventure.me/
Server: Caddy
Date: Fri, 16 Jul 2021 01:49:16 GMT

HTTP/2 302
date: Fri, 16 Jul 2021 01:49:16 GMT
location: /web/index.html
server: Caddy
server: Kestrel

HTTP/2 200
accept-ranges: bytes
content-type: text/html
date: Fri, 16 Jul 2021 01:49:17 GMT
etag: "1d74de5f7259d00"
last-modified: Fri, 21 May 2021 02:06:52 GMT
server: Caddy
server: Kestrel
x-response-time-ms: 0

But I get the exact same page issue you do in my browser - status 502!

Lets add debug logging: Global options (Caddyfile) — Caddy Documentation

{
  debug
}

And see if we can inspect a bit closer what result Caddy’s getting from the upstream server.

Jeff_Untalan · July 16, 2021, 1:55am

Here are some more log files. I think I’ve found the issue. Here are some log files from my droplet’s inlets tunnel server:

Jul 16 01:49:17 sparefoundry inlets[6653]: 2021/07/16 01:49:17 Proxy: media.leveloneadventure.me HEAD => /
Jul 16 01:49:17 sparefoundry inlets[6653]: 2021/07/16 01:49:17 Proxy: media.leveloneadventure.me HEAD => /web/index.html
Jul 16 01:49:51 sparefoundry inlets[6653]: 2021/07/16 01:49:51 Proxy: media.leveloneadventure.me HEAD => /
Jul 16 01:49:51 sparefoundry inlets[6653]: 2021/07/16 01:49:51 Proxy: media.leveloneadventure.me HEAD => /web/index.html
Jul 16 01:50:07 sparefoundry inlets[6653]: 2021/07/16 01:50:07 Proxy: media.leveloneadventure.me GET => /
Jul 16 01:50:08 sparefoundry inlets[6653]: 2021/07/16 01:50:08 Proxy: media.leveloneadventure.me GET => /web/index.html
Jul 16 01:50:08 sparefoundry inlets[6653]: 2021/07/16 01:50:08 Proxy: media.leveloneadventure.me GET => /
Jul 16 01:50:08 sparefoundry inlets[6653]: 2021/07/16 01:50:08 Proxy: media.leveloneadventure.me GET => /web/index.html
Jul 16 01:50:09 sparefoundry inlets[6653]: E0716 01:50:09.018473    6653 transport.go:246] Proxy encountered encoding br for text/html; can't understand this so not fixing links.
Jul 16 01:50:09 sparefoundry inlets[6653]: 2021/07/16 01:50:09 httputil: ReverseProxy read error during body copy: http: read on closed response body
Jul 16 01:50:09 sparefoundry inlets[6653]: 2021/07/16 01:50:09 Proxy: media.leveloneadventure.me GET => /web/index.html
Jul 16 01:50:09 sparefoundry inlets[6653]: E0716 01:50:09.081448    6653 transport.go:246] Proxy encountered encoding br for text/html; can't understand this so not fixing links.
Jul 16 01:50:09 sparefoundry inlets[6653]: 2021/07/16 01:50:09 httputil: ReverseProxy read error during body copy: http: read on closed response body

and here are the logs for the inlets tunnel client on my raspberry pi:

Jul 16 09:18:53 raspberrypi inlets[4625]: 2021/07/16 09:18:53 Starting client - version 3.0.2
Jul 16 09:18:53 raspberrypi inlets[4625]: 2021/07/16 09:18:53 Upstream:  => http://127.0.0.1:8096
Jul 16 09:18:53 raspberrypi inlets[4625]: time="2021/07/16 09:18:53" level=info msg="Connecting to proxy" url="wss://media.leveloneadventure.me/tunnel"

It seems like the inlets tunnel encountered an error Proxy encountered encoding br for text/html; can't understand this so not fixing links. which I’m still not sure what to make of. Is there any way to fix this? I apologize for this. This seems a bit out of scope for caddy.

Whitestrake · July 16, 2021, 2:04am

Does Inlets do HTML/header filtering to replace URLs?

It seems to imply that it does; but in this instance, can’t, because it doesn’t know how to handle Brotli encoding (br, as commonly opposed to gzip gz).

I’m not sure this is an issue at all, since Caddy is expecting the same URL as Inlets, so there should be no need to fix links.

Jeff_Untalan · July 16, 2021, 2:06am

As per request I’ve added the debug block at the top of my caddy file. Here is the output:

Jul 16 02:03:25 sparefoundry caddy[7127]: {"level":"info","ts":1626401005.757862,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0001b5030"}
Jul 16 02:03:25 sparefoundry caddy[7127]: {"level":"info","ts":1626401005.7579625,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Jul 16 02:03:25 sparefoundry caddy[7127]: {"level":"info","ts":1626401005.7785609,"logger":"tls","msg":"finished cleaning storage units"}
Jul 16 02:03:25 sparefoundry caddy[7127]: {"level":"info","ts":1626401005.7790053,"msg":"serving initial configuration"}
Jul 16 02:03:27 sparefoundry caddy[7127]: {"level":"debug","ts":1626401007.78108,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"localhost:8096","request":{"remote_addr":"136.158.31.6:15278","proto":"HTTP/1.1","method":"GET","host":"media.leveloneadventure.me","uri":"/tunnel","headers":{"Upgrade":["websocket"],"Sec-Websocket-Version":["13"],"X-Forwarded-For":["136.158.31.6"],"User-Agent":["Go-http-client/1.1"],"Authorization":["Bearer 8096"],"Connection":["Upgrade"],"Sec-Websocket-Key":["1RM0DBMglg9KM6Vdqw9h6g=="],"X-Inlets-Id":["749910c35dc24150b78f1069e5f5fbb9"],"X-Inlets-Upstream":["=http://127.0.0.1:8096"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","proto_mutual":true,"server_name":"media.leveloneadventure.me"}},"headers":{"Connection":["Upgrade"],"Sec-Websocket-Accept":["GugLZvZ6HqK2H4H771V8sPQk70Q="],"Upgrade":["websocket"]},"status":101}
Jul 16 02:03:27 sparefoundry caddy[7127]: {"level":"debug","ts":1626401007.7832007,"logger":"http.handlers.reverse_proxy","msg":"upgrading connection","upstream":"localhost:8096","request":{"remote_addr":"136.158.31.6:15278","proto":"HTTP/1.1","method":"GET","host":"media.leveloneadventure.me","uri":"/tunnel","headers":{"Upgrade":["websocket"],"Sec-Websocket-Version":["13"],"X-Forwarded-For":["136.158.31.6"],"User-Agent":["Go-http-client/1.1"],"Authorization":["Bearer 8096"],"Connection":["Upgrade"],"Sec-Websocket-Key":["1RM0DBMglg9KM6Vdqw9h6g=="],"X-Inlets-Id":["749910c35dc24150b78f1069e5f5fbb9"],"X-Inlets-Upstream":["=http://127.0.0.1:8096"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","proto_mutual":true,"server_name":"media.leveloneadventure.me"}}}
Jul 16 02:03:32 sparefoundry caddy[7127]: {"level":"debug","ts":1626401012.2437503,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"localhost:1717","request":{"remote_addr":"136.158.31.6:15432","proto":"HTTP/1.1","method":"GET","host":"pi.leveloneadventure.me","uri":"/tunnel","headers":{"X-Forwarded-Proto":["https"],"Authorization":["Bearer 1710"],"Sec-Websocket-Key":["G+2McrX1SazGNtstxsScqQ=="],"X-Inlets-Upstream":["=http://127.0.0.1:30000"],"X-Forwarded-For":["136.158.31.6"],"X-Inlets-Id":["a7bb5e56687649938550b4f5bedbab1f"],"Connection":["Upgrade"],"User-Agent":["Go-http-client/1.1"],"Upgrade":["websocket"],"Sec-Websocket-Version":["13"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","proto_mutual":true,"server_name":"pi.leveloneadventure.me"}},"headers":{"Upgrade":["websocket"],"Connection":["Upgrade"],"Sec-Websocket-Accept":["gWatMoN7h+DY1D57Oyirhj4JpUM="]},"status":101}
Jul 16 02:03:32 sparefoundry caddy[7127]: {"level":"debug","ts":1626401012.2442956,"logger":"http.handlers.reverse_proxy","msg":"upgrading connection","upstream":"localhost:1717","request":{"remote_addr":"136.158.31.6:15432","proto":"HTTP/1.1","method":"GET","host":"pi.leveloneadventure.me","uri":"/tunnel","headers":{"X-Forwarded-Proto":["https"],"Authorization":["Bearer 1710"],"Sec-Websocket-Key":["G+2McrX1SazGNtstxsScqQ=="],"X-Inlets-Upstream":["=http://127.0.0.1:30000"],"X-Forwarded-For":["136.158.31.6"],"X-Inlets-Id":["a7bb5e56687649938550b4f5bedbab1f"],"Connection":["Upgrade"],"User-Agent":["Go-http-client/1.1"],"Upgrade":["websocket"],"Sec-Websocket-Version":["13"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","proto_mutual":true,"server_name":"pi.leveloneadventure.me"}}}
Jul 16 02:03:32 sparefoundry caddy[7127]: {"level":"debug","ts":1626401012.2811499,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"localhost:8083","request":{"remote_addr":"136.158.31.6:15434","proto":"HTTP/1.1","method":"GET","host":"books.leveloneadventure.me","uri":"/tunnel","headers":{"Connection":["Upgrade"],"Authorization":["Bearer 1717"],"Upgrade":["websocket"],"Sec-Websocket-Key":["hN6ZOl02EC/QqDtVjNXnjA=="],"X-Forwarded-Proto":["https"],"X-Inlets-Upstream":["=http://127.0.0.1:8083"],"User-Agent":["Go-http-client/1.1"],"Sec-Websocket-Version":["13"],"X-Inlets-Id":["c0d474ae2371414faf83d2b7fa847336"],"X-Forwarded-For":["136.158.31.6"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","proto_mutual":true,"server_name":"books.leveloneadventure.me"}},"headers":{"Connection":["Upgrade"],"Sec-Websocket-Accept":["JUKjeZFv8MBubtB71wp/IFOcQaA="],"Upgrade":["websocket"]},"status":101}
Jul 16 02:03:32 sparefoundry caddy[7127]: {"level":"debug","ts":1626401012.2814639,"logger":"http.handlers.reverse_proxy","msg":"upgrading connection","upstream":"localhost:8083","request":{"remote_addr":"136.158.31.6:15434","proto":"HTTP/1.1","method":"GET","host":"books.leveloneadventure.me","uri":"/tunnel","headers":{"Connection":["Upgrade"],"Authorization":["Bearer 1717"],"Upgrade":["websocket"],"Sec-Websocket-Key":["hN6ZOl02EC/QqDtVjNXnjA=="],"X-Forwarded-Proto":["https"],"X-Inlets-Upstream":["=http://127.0.0.1:8083"],"User-Agent":["Go-http-client/1.1"],"Sec-Websocket-Version":["13"],"X-Inlets-Id":["c0d474ae2371414faf83d2b7fa847336"],"X-Forwarded-For":["136.158.31.6"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","proto_mutual":true,"server_name":"books.leveloneadventure.me"}}}
Jul 16 02:03:46 sparefoundry caddy[7127]: {"level":"debug","ts":1626401026.8164861,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"localhost:8096","request":{"remote_addr":"136.158.31.6:15984","proto":"HTTP/2.0","method":"GET","host":"media.leveloneadventure.me","uri":"/web/index.html","headers":{"Cache-Control":["max-age=0"],"Accept-Language":["en-US,en;q=0.9"],"X-Forwarded-For":["136.158.31.6"],"Sec-Ch-Ua":["\" Not;A Brand\";v=\"99\", \"Microsoft Edge\";v=\"91\", \"Chromium\";v=\"91\""],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Edg/91.0.864.67"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"X-Forwarded-Proto":["https"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Dest":["document"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Site":["none"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"media.leveloneadventure.me"}},"duration":0.061300777,"error":"EOF"}
Jul 16 02:03:46 sparefoundry caddy[7127]: {"level":"error","ts":1626401026.8173356,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"136.158.31.6:15984","proto":"HTTP/2.0","method":"GET","host":"media.leveloneadventure.me","uri":"/web/index.html","headers":{"Cache-Control":["max-age=0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Dest":["document"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Ch-Ua":["\" Not;A Brand\";v=\"99\", \"Microsoft Edge\";v=\"91\", \"Chromium\";v=\"91\""],"Sec-Ch-Ua-Mobile":["?0"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Edg/91.0.864.67"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"media.leveloneadventure.me"}},"duration":0.062304193,"status":502,"err_id":"32grcyzzw","err_trace":"reverseproxy.statusError (reverseproxy.go:857)"}

Whitestrake · July 16, 2021, 2:15am

Jellyfin appears to be discriminating based on… something.

When I curl it, it seems fine:

> GET /web/index.html HTTP/2
> Host: media.leveloneadventure.me
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/2 200
< accept-ranges: bytes
< content-type: text/html
< date: Fri, 16 Jul 2021 02:08:09 GMT
< etag: "1d74de5f7259d00"
< last-modified: Fri, 21 May 2021 02:06:52 GMT
< server: Caddy
< server: Kestrel
< vary: Accept-Encoding
< x-response-time-ms: 6

But when the request is made from a browser, Jellyfin abruptly cuts off:

{
  "level": "debug",
  "ts": 1626401026.8164861,
  "logger": "http.handlers.reverse_proxy",
  "msg": "upstream roundtrip",
  "upstream": "localhost:8096",
  "request": {
    "remote_addr": "136.158.31.6:15984",
    "proto": "HTTP/2.0",
    "method": "GET",
    "host": "media.leveloneadventure.me",
    "uri": "/web/index.html",
    "headers": {
      "Cache-Control": [
        "max-age=0"
      ],
      "Accept-Language": [
        "en-US,en;q=0.9"
      ],
      "X-Forwarded-For": [
        "136.158.31.6"
      ],
      "Sec-Ch-Ua": [
        "\" Not;A Brand\";v=\"99\", \"Microsoft Edge\";v=\"91\", \"Chromium\";v=\"91\""
      ],
      "Upgrade-Insecure-Requests": [
        "1"
      ],
      "User-Agent": [
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Edg/91.0.864.67"
      ],
      "Sec-Fetch-Mode": [
        "navigate"
      ],
      "Sec-Fetch-User": [
        "?1"
      ],
      "X-Forwarded-Proto": [
        "https"
      ],
      "Accept": [
        "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
      ],
      "Sec-Fetch-Dest": [
        "document"
      ],
      "Sec-Ch-Ua-Mobile": [
        "?0"
      ],
      "Sec-Fetch-Site": [
        "none"
      ],
      "Accept-Encoding": [
        "gzip, deflate, br"
      ]
    },
    "tls": {
      "resumed": false,
      "version": 772,
      "cipher_suite": 4865,
      "proto": "h2",
      "proto_mutual": true,
      "server_name": "media.leveloneadventure.me"
    }
  },
  "duration": 0.061300777,
  "error": "EOF"
}

Both of these requests are going through Caddy. The only feasible difference I can think of are user-agent / browser specific headers.

Whitestrake · July 16, 2021, 2:32am

Jeff_Untalan:

Jul 16 01:50:09 sparefoundry inlets[6653]: E0716 01:50:09.081448    6653 transport.go:246] Proxy encountered encoding br for text/html; can't understand this so not fixing links.
Jul 16 01:50:09 sparefoundry inlets[6653]: 2021/07/16 01:50:09 httputil: ReverseProxy read error during body copy: http: read on closed response body

Yeah, this is definitely the problem.

When curl hits, it doesn’t send Accept-Encoding, so we get an unencoded response along with a vary: Accept-Encoding (which just says don’t use this cached response if you change your requested encoding).

When the browser hits, it sends Accept-Encoding: gzip, deflate, br, Jellyfin sees it and responds with a Brotli-encoded body (br), and Inlets spits the dummy because it can’t get its little fingers in to do stream filtering because it doesn’t grok Brotli.

Pretty much confirmed by adding this header manually to the curl request:

> GET /web/index.html HTTP/2
> Host: media.leveloneadventure.me
> User-Agent: curl/7.64.1
> Accept: */*
> Accept-Encoding: gzip, deflate, br
>
< HTTP/2 502
< server: Caddy
< content-length: 0
< date: Fri, 16 Jul 2021 02:29:33 GMT

Can you disable link fixing in Inlets? I think this would be the ideal solution, since again I suspect no link fixing is required at all.

It seems to break on gzip, too, but deflate works fine.

In a worst-case scenario, you can have Caddy strip the Accept-Encoding header before sending it upstream. That will have Jellyfin respond to every request unencoded (a rather arbitrary and inconvenient waste of comparable bandwidth), but should stop Inlets from being silly about it.

Jeff_Untalan · July 16, 2021, 2:37am

I’ll definitely do this. Thank you for the lead! I’ll update you with what I get.

Jeff_Untalan · July 16, 2021, 2:52am

Hey Whitestrake! It’s fixed! I added a --disable-transport-wrapping to the inlets server and it’s working now. I apologize. It wasn’t caddy issue at all. Thank you very much for the help. The Caddy community rocks!

system · August 14, 2021, 9:12am

This topic was automatically closed after 30 days. New replies are no longer allowed.