Emby reverse proxy doesn't work

1. Caddy version (caddy version):

v2.4.0-beta.2

2. How I run Caddy:

a. System environment:

Docker container built from builder-alpine official image with Cloudflare module.

b. Command:

Don’t know, it’s baked into the official Docker image. But here is my Dockerfile:

FROM caddy:builder-alpine AS builder

RUN xcaddy build \
    --with github.com/caddy-dns/cloudflare

FROM caddy:alpine

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

c. Service/unit/compose file:

version: "3.9"
services:
  caddy:
    container_name: caddy
    image: rafaelagp/caddy-cloudflare
    user: 1001:100
    volumes:
      - /mnt/solid/config/caddy:/config
      - /mnt/solid/config/caddy:/data/caddy
      - /mnt/solid/config/caddy/Caddyfile:/etc/caddy/Caddyfile
    ports:
      - 80:80
      - 443:443
      - 10001:10001
    network_mode: "bridge"
    restart: unless-stopped

d. My complete Caddyfile or JSON config:

Sorry for redacting but I do not want my domain made public.

{
        debug
        acme_dns cloudflare api_token
        email email@domain
}

server:80 {
        reverse_proxy 192.168.1.3:8002
}

yacht:80 {
        reverse_proxy 192.168.1.3:8001
}

transmission:80 {
        reverse_proxy 192.168.1.3:9091
}

jackett:80 {
        reverse_proxy 192.168.1.3:9117
}

radarr:80 {
        reverse_proxy 192.168.1.3:7878
}

sonarr:80 {
        reverse_proxy 192.168.1.3:8989
}

emby:80 {
        reverse_proxy 192.168.1.3:8096
}

https://emby.domain:10001 {
        reverse_proxy 192.168.1.3:8920 {
                header_up X-Real-IP {remote_host}
        }
        encode gzip
        header / {
                Strict-Transport-Security "max-age=31536000;"
                X-XSS-Protection "1; mode=block"
                X-Frame-Options "DENY"
                X-Robots-Tag "none"
                -Server
        }
}

3. The problem I’m having:

Can’t access Emby remotely through Caddy’s reverse proxy.

4. Error messages and/or full log output:

Again, sorry for redacting IPs and domain:

{
    "level": "debug",
    "ts": 1618770361.4402223,
    "logger": "http.handlers.reverse_proxy",
    "msg": "upstream roundtrip",
    "upstream": "192.168.1.3:8920",
    "request": {
        "remote_addr": "mobile_phone_ip:60962",
        "proto": "HTTP/2.0",
        "method": "GET",
        "host": "emby.domain:10001",
        "uri": "/",
        "headers": {
            "Accept": [
                "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
            ],
            "X-Real-Ip": [
                "mobile_phone_ip"
            ],
            "Sec-Fetch-Mode": [
                "navigate"
            ],
            "X-Forwarded-For": [
                "mobile_phone_ip"
            ],
            "Save-Data": [
                "on"
            ],
            "X-Forwarded-Proto": [
                "https"
            ],
            "User-Agent": [
                "Mozilla/5.0 (Linux; Android 11; SM-G973F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.66 Mobile Safari/537.36"
            ],
            "Sec-Fetch-Site": [
                "none"
            ],
            "Cookie": [
                "__cfduid=df3e2dd3ad9ecad0dc7a8d03fe7faddd81618721715"
            ],
            "Accept-Encoding": [
                "gzip, deflate, br"
            ],
            "Sec-Fetch-Dest": [
                "document"
            ],
            "Sec-Ch-Ua": [
                "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""
            ],
            "Accept-Language": [
                "en-US,en;q=0.9,pt;q=0.8,fr;q=0.7"
            ],
            "Sec-Fetch-User": [
                "?1"
            ],
            "Upgrade-Insecure-Requests": [
                "1"
            ],
            "Sec-Ch-Ua-Mobile": [
                "?1"
            ],
            "Purpose": [
                "prefetch"
            ]
        },
        "tls": {
            "resumed": false,
            "version": 772,
            "cipher_suite": 4865,
            "proto": "h2",
            "proto_mutual": true,
            "server_name": "emby.domain"
        }
    },
    "duration": 0.001930143,
    "error": "EOF"
}
    {
    "level": "error",
    "ts": 1618770361.4403605,
    "logger": "http.log.error",
    "msg": "EOF",
    "request": {
        "remote_addr": "mobile_phone_ip:60962",
        "proto": "HTTP/2.0",
        "method": "GET",
        "host": "emby.domain:10001",
        "uri": "/",
        "headers": {
            "Purpose": [
                "prefetch"
            ],
            "Sec-Fetch-Site": [
                "none"
            ],
            "Sec-Fetch-Mode": [
                "navigate"
            ],
            "Sec-Fetch-Dest": [
                "document"
            ],
            "Cookie": [
                "__cfduid=df3e2dd3ad9ecad0dc7a8d03fe7faddd81618721715"
            ],
            "Sec-Ch-Ua-Mobile": [
                "?1"
            ],
            "User-Agent": [
                "Mozilla/5.0 (Linux; Android 11; SM-G973F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.66 Mobile Safari/537.36"
            ],
            "Accept": [
                "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
            ],
            "Sec-Ch-Ua": [
                "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""
            ],
            "Accept-Language": [
                "en-US,en;q=0.9,pt;q=0.8,fr;q=0.7"
            ],
            "Save-Data": [
                "on"
            ],
            "Sec-Fetch-User": [
                "?1"
            ],
            "Accept-Encoding": [
                "gzip, deflate, br"
            ],
            "Upgrade-Insecure-Requests": [
                "1"
            ]
        },
        "tls": {
            "resumed": false,
            "version": 772,
            "cipher_suite": 4865,
            "proto": "h2",
            "proto_mutual": true,
            "server_name": "emby.domain"
        }
    },
    "duration": 0.002249738,
    "status": 502,
    "err_id": "sy77gwcj2",
    "err_trace": "reverseproxy.statusError (reverseproxy.go:843)"
}

5. What I already tried:

  • I’ve got the port (10001) forwarded to the server in my router.
  • I’ve tried forwarding straight to Emby (8920), and it works, I can access it from my mobile phone’s data connection (not wifi), which points the problem at the reverse proxy configuration.
  • I’ve tried replacing the reverse proxy LAN IP for the Docker network IP with no change in response.
  • I’ve tried removing all directives but the reverse proxy with no change in response.
  • I’ve successfully ping’ed both the LAN IP and the Docker network IP for Emby from within Caddy’s container, so I know it’s reachable.

6. Links to relevant resources:

None yet.

You should no use the same volume for both those paths. This will cause data loss. You should be persisting /data and /config to different locations (and not /data/caddy).

With this, you’re only matching requests to exactly / and nothing else, i.e. that site’s root. Remove the / to make those header options apply to all requests.

I doubt you need this line. There’s evidence that Emby works fine when behind Caddy with a simple reverse_proxy, as shown by this topic on the Emby forums:

https://emby.media/community/index.php?/topic/84777-caddy-v2-update-and-warning

Caddy will proxy HTTP by default. Make sure emby is listening for HTTP requests (not HTTPS) on that port. Seeing as you have another site emby:80 which is proxying to port 8096, that makes me think port 8920 is the HTTPS port or something.

Ok, I’ll set it up differently.

Ah, good to know, will do.

And I’ve tried it with only the reverse_proxy directive but it gave me the same 502 error.

emby:80 works fine within my LAN, but I am trying to expose my Emby service to the internet.

8920 is the https port, yes, and it works when accessed directly through domain:8920. I am trying to reverse proxy port 10001 from Caddy to Emby’s https 8920 because I can’t yet forward 443 in my router but it is not working.

Well, you don’t really need to proxy HTTPS from Caddy to emby; you’re on the same machine, you can proxy over HTTP and it’s fine. Caddy terminates TLS.

Basically:

Browser → HTTPS → (Caddy → HTTP → Emby)

The bit in parentheses is what’s going on on that machine. There’s no real security advantage to proxying from Caddy to Emby over HTTPS because the only risk of interception is having other software on that same machine snooping the traffic. You shouldn’t be running untrusted software, and if something was already able to do that, it would be game over already (for other reasons).

So just change 192.168.1.3:8920 to 192.168.1.3:8096 and you’ll be good to go.

For completion’s sake, if you had a reason to proxy over HTTPS (like if emby was on another machine or in another network where you can’t trust the pipe between them) then you would need to do reverse_proxy https://192.168.1.3:8920 to tell Caddy to connect with TLS enabled, and then you would need to make sure Caddy trusts the certificate served by that service via reverse_proxy's tls_trusted_ca_certs option (or turn off security altogether with tls_insecure_skip_verify which completely defeats the purpose of using TLS anyways because that would allow man-in-the-middle attacks to work).

This article might help explain these ideas:

2 Likes

Yes, you are totally right! That never occurred to me despite being pretty obvious. I’ll implement it as soon as I can and report back.

Thanks for being thorough, I appreciate learning some details.
I’ll report back once I try forwarding to Emby over http.

Just found out my ISP blocks all the common ports (21, 80, 443…), so in the end that was the actual problem.

Now I’m researching how to implement a simple Cloudflare Workers reverse proxy to route port 443 to a different port I can actually open on my router.

I wonder if this would help? https://blog.cloudflare.com/tunnel-for-everyone/

1 Like

It’s an alternative to doing a reverse proxy with Workers, but a bit more work. If the Workers thingy doesn’t pan out I’ll definitely try it out.

Thanks for pointing it out!

This topic was automatically closed after 30 days. New replies are no longer allowed.