Emby reverse proxy doesn't work

[rafaelagp (Rafael A. G. Pereira)]

1. Caddy version (caddy version):

v2.4.0-beta.2

2. How I run Caddy:

a. System environment:

Docker container built from builder-alpine official image with Cloudflare module.

b. Command:

Don’t know, it’s baked into the official Docker image. But here is my Dockerfile:

FROM caddy:builder-alpine AS builder

RUN xcaddy build \
    --with github.com/caddy-dns/cloudflare

FROM caddy:alpine

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

c. Service/unit/compose file:

version: "3.9"
services:
  caddy:
    container_name: caddy
    image: rafaelagp/caddy-cloudflare
    user: 1001:100
    volumes:
      - /mnt/solid/config/caddy:/config
      - /mnt/solid/config/caddy:/data/caddy
      - /mnt/solid/config/caddy/Caddyfile:/etc/caddy/Caddyfile
    ports:
      - 80:80
      - 443:443
      - 10001:10001
    network_mode: "bridge"
    restart: unless-stopped

d. My complete Caddyfile or JSON config:

Sorry for redacting but I do not want my domain made public.

{
        debug
        acme_dns cloudflare api_token
        email email@domain
}

server:80 {
        reverse_proxy 192.168.1.3:8002
}

yacht:80 {
        reverse_proxy 192.168.1.3:8001
}

transmission:80 {
        reverse_proxy 192.168.1.3:9091
}

jackett:80 {
        reverse_proxy 192.168.1.3:9117
}

radarr:80 {
        reverse_proxy 192.168.1.3:7878
}

sonarr:80 {
        reverse_proxy 192.168.1.3:8989
}

emby:80 {
        reverse_proxy 192.168.1.3:8096
}

https://emby.domain:10001 {
        reverse_proxy 192.168.1.3:8920 {
                header_up X-Real-IP {remote_host}
        }
        encode gzip
        header / {
                Strict-Transport-Security "max-age=31536000;"
                X-XSS-Protection "1; mode=block"
                X-Frame-Options "DENY"
                X-Robots-Tag "none"
                -Server
        }
}

3. The problem I’m having:

Can’t access Emby remotely through Caddy’s reverse proxy.

4. Error messages and/or full log output:

Again, sorry for redacting IPs and domain:

{
    "level": "debug",
    "ts": 1618770361.4402223,
    "logger": "http.handlers.reverse_proxy",
    "msg": "upstream roundtrip",
    "upstream": "192.168.1.3:8920",
    "request": {
        "remote_addr": "mobile_phone_ip:60962",
        "proto": "HTTP/2.0",
        "method": "GET",
        "host": "emby.domain:10001",
        "uri": "/",
        "headers": {
            "Accept": [
                "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
            ],
            "X-Real-Ip": [
                "mobile_phone_ip"
            ],
            "Sec-Fetch-Mode": [
                "navigate"
            ],
            "X-Forwarded-For": [
                "mobile_phone_ip"
            ],
            "Save-Data": [
                "on"
            ],
            "X-Forwarded-Proto": [
                "https"
            ],
            "User-Agent": [
                "Mozilla/5.0 (Linux; Android 11; SM-G973F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.66 Mobile Safari/537.36"
            ],
            "Sec-Fetch-Site": [
                "none"
            ],
            "Cookie": [
                "__cfduid=df3e2dd3ad9ecad0dc7a8d03fe7faddd81618721715"
            ],
            "Accept-Encoding": [
                "gzip, deflate, br"
            ],
            "Sec-Fetch-Dest": [
                "document"
            ],
            "Sec-Ch-Ua": [
                "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""
            ],
            "Accept-Language": [
                "en-US,en;q=0.9,pt;q=0.8,fr;q=0.7"
            ],
            "Sec-Fetch-User": [
                "?1"
            ],
            "Upgrade-Insecure-Requests": [
                "1"
            ],
            "Sec-Ch-Ua-Mobile": [
                "?1"
            ],
            "Purpose": [
                "prefetch"
            ]
        },
        "tls": {
            "resumed": false,
            "version": 772,
            "cipher_suite": 4865,
            "proto": "h2",
            "proto_mutual": true,
            "server_name": "emby.domain"
        }
    },
    "duration": 0.001930143,
    "error": "EOF"
}
    {
    "level": "error",
    "ts": 1618770361.4403605,
    "logger": "http.log.error",
    "msg": "EOF",
    "request": {
        "remote_addr": "mobile_phone_ip:60962",
        "proto": "HTTP/2.0",
        "method": "GET",
        "host": "emby.domain:10001",
        "uri": "/",
        "headers": {
            "Purpose": [
                "prefetch"
            ],
            "Sec-Fetch-Site": [
                "none"
            ],
            "Sec-Fetch-Mode": [
                "navigate"
            ],
            "Sec-Fetch-Dest": [
                "document"
            ],
            "Cookie": [
                "__cfduid=df3e2dd3ad9ecad0dc7a8d03fe7faddd81618721715"
            ],
            "Sec-Ch-Ua-Mobile": [
                "?1"
            ],
            "User-Agent": [
                "Mozilla/5.0 (Linux; Android 11; SM-G973F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.66 Mobile Safari/537.36"
            ],
            "Accept": [
                "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
            ],
            "Sec-Ch-Ua": [
                "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""
            ],
            "Accept-Language": [
                "en-US,en;q=0.9,pt;q=0.8,fr;q=0.7"
            ],
            "Save-Data": [
                "on"
            ],
            "Sec-Fetch-User": [
                "?1"
            ],
            "Accept-Encoding": [
                "gzip, deflate, br"
            ],
            "Upgrade-Insecure-Requests": [
                "1"
            ]
        },
        "tls": {
            "resumed": false,
            "version": 772,
            "cipher_suite": 4865,
            "proto": "h2",
            "proto_mutual": true,
            "server_name": "emby.domain"
        }
    },
    "duration": 0.002249738,
    "status": 502,
    "err_id": "sy77gwcj2",
    "err_trace": "reverseproxy.statusError (reverseproxy.go:843)"
}

5. What I already tried:

• I’ve got the port (10001) forwarded to the server in my router.
• I’ve tried forwarding straight to Emby (8920), and it works, I can access it from my mobile phone’s data connection (not wifi), which points the problem at the reverse proxy configuration.
• I’ve tried replacing the reverse proxy LAN IP for the Docker network IP with no change in response.
• I’ve tried removing all directives but the reverse proxy with no change in response.
• I’ve successfully ping’ed both the LAN IP and the Docker network IP for Emby from within Caddy’s container, so I know it’s reachable.

6. Links to relevant resources:

None yet.

[francislavoie]

@rafaelagp

      - /mnt/solid/config/caddy:/config
      - /mnt/solid/config/caddy:/data/caddy

You should no use the same volume for both those paths. This will cause data loss. You should be persisting /data and /config to different locations (and not /data/caddy).

@rafaelagp

header /

With this, you’re only matching requests to exactly / and nothing else, i.e. that site’s root. Remove the / to make those header options apply to all requests.

@rafaelagp

header_up X-Real-IP {remote_host}

I doubt you need this line. There’s evidence that Emby works fine when behind Caddy with a simple reverse_proxy, as shown by this topic on the Emby forums:

https://emby.media/community/index.php?/topic/84777-caddy-v2-update-and-warning

Caddy will proxy HTTP by default. Make sure emby is listening for HTTP requests (not HTTPS) on that port. Seeing as you have another site emby:80 which is proxying to port 8096, that makes me think port 8920 is the HTTPS port or something.

[rafaelagp (Rafael A. G. Pereira)]

@francislavoie

You should no use the same volume for both those paths. This will cause data loss. You should be persisting /data and /config to different locations (and not /data/caddy).

Ok, I’ll set it up differently.

@francislavoie

With this, you’re only matching requests to exactly / and nothing else, i.e. that site’s root. Remove the / to make those header options apply to all requests.

Ah, good to know, will do.

@francislavoie

I doubt you need this line. There’s evidence that Emby works fine when behind Caddy with a simple reverse_proxy, as shown by this topic on the Emby forums:

https://emby.media/community/index.php?/topic/84777-caddy-v2-update-and-warning

And I’ve tried it with only the reverse_proxy directive but it gave me the same 502 error.

@francislavoie

Caddy will proxy HTTP by default. Make sure emby is listening for HTTP requests (not HTTPS) on that port. Seeing as you have another site emby:80 which is proxying to port 8096, that makes me think port 8920 is the HTTPS port or something.

emby:80 works fine within my LAN, but I am trying to expose my Emby service to the internet.

8920 is the https port, yes, and it works when accessed directly through domain:8920. I am trying to reverse proxy port 10001 from Caddy to Emby’s https 8920 because I can’t yet forward 443 in my router but it is not working.

[francislavoie]

Well, you don’t really need to proxy HTTPS from Caddy to emby; you’re on the same machine, you can proxy over HTTP and it’s fine. Caddy terminates TLS.

Basically:

Browser → HTTPS → (Caddy → HTTP → Emby)

The bit in parentheses is what’s going on on that machine. There’s no real security advantage to proxying from Caddy to Emby over HTTPS because the only risk of interception is having other software on that same machine snooping the traffic. You shouldn’t be running untrusted software, and if something was already able to do that, it would be game over already (for other reasons).

So just change 192.168.1.3:8920 to 192.168.1.3:8096 and you’ll be good to go.

For completion’s sake, if you had a reason to proxy over HTTPS (like if emby was on another machine or in another network where you can’t trust the pipe between them) then you would need to do reverse_proxy https://192.168.1.3:8920 to tell Caddy to connect with TLS enabled, and then you would need to make sure Caddy trusts the certificate served by that service via reverse_proxy’s tls_trusted_ca_certs option (or turn off security altogether with tls_insecure_skip_verify which completely defeats the purpose of using TLS anyways because that would allow man-in-the-middle attacks to work).

This article might help explain these ideas:

Using Caddy as a reverse proxy in a home network

If you want to run a service inside a Local Area Network (LAN) such as your home or office – and especially if you want to be able to access it from outside that network – Caddy can help you accomplish this quite easily. This guide will show you how. It assumes you’ve never done this before, but that you have some technical proficiency and are somewhat knowledgable about your own network. Do this at your own risk. There are several ways to make private services accessible from other n…

[rafaelagp (Rafael A. G. Pereira)]

@francislavoie

Well, you don’t really need to proxy HTTPS from Caddy to emby; you’re on the same machine, you can proxy over HTTP and it’s fine. Caddy terminates TLS.

Basically:

Browser → HTTPS → (Caddy → HTTP → Emby)

The bit in parentheses is what’s going on on that machine. There’s no real security advantage to proxying from Caddy to Emby over HTTPS because the only risk of interception is having other software on that same machine snooping the traffic. You shouldn’t be running untrusted software, and if something was already able to do that, it would be game over already (for other reasons).

So just change 192.168.1.3:8920 to 192.168.1.3:8096 and you’ll be good to go.

Yes, you are totally right! That never occurred to me despite being pretty obvious. I’ll implement it as soon as I can and report back.

@francislavoie

For completion’s sake, if you had a reason to proxy over HTTPS (like if emby was on another machine or in another network where you can’t trust the pipe between them) then you would need to do reverse_proxy https://192.168.1.3:8920 to tell Caddy to connect with TLS enabled, and then you would need to make sure Caddy trusts the certificate served by that service via reverse_proxy’s tls_trusted_ca_certs option (or turn off security altogether with tls_insecure_skip_verify which completely defeats the purpose of using TLS anyways because that would allow man-in-the-middle attacks to work).

Thanks for being thorough, I appreciate learning some details.
I’ll report back once I try forwarding to Emby over http.

[rafaelagp (Rafael A. G. Pereira)]

Just found out my ISP blocks all the common ports (21, 80, 443…), so in the end that was the actual problem.

Now I’m researching how to implement a simple Cloudflare Workers reverse proxy to route port 443 to a different port I can actually open on my router.

[matt (Matt Holt)]

I wonder if this would help? https://blog.cloudflare.com/tunnel-for-everyone/

[rafaelagp (Rafael A. G. Pereira)]

It’s an alternative to doing a reverse proxy with Workers, but a bit more work. If the Workers thingy doesn’t pan out I’ll definitely try it out.

Thanks for pointing it out!