Routing traffic through vpn giving 502 ERROR

1. Caddy version (caddy version): 2

2. How I run Caddy:

a. System environment:

Inside a docker container on a raspberry pi 4.

b. Command:

N/A

c. Service/unit/compose file:

version: "3.7"
services:

  caddy:
    image: caddy
    container_name: caddy
    hostname: caddy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    environment:
      - MY_DOMAIN
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - ./data:/data
      - ./config:/config

networks:
  default:
    external:
      name: proxy

d. My complete Caddyfile or JSON config:


{
    # testing
    # acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}

#make homer my homepage
{$MY_DOMAIN} {
    reverse_proxy homer:6941
}


portainer.{$MY_DOMAIN} {
    reverse_proxy portainer:9000
}

radarr.{$MY_DOMAIN} {
    reverse_proxy radarr:7878
}

sonarr.{$MY_DOMAIN} {
    reverse_proxy sonarr:8989
}

lidarr.{$MY_DOMAIN} {
    reverse_proxy lidarr:8686
}

ombi.{$MY_DOMAIN} {
    reverse_proxy ombi:3579
}

jellyfin.{$MY_DOMAIN} {
    reverse_proxy jellyfin:8096
}


jackett.{$MY_DOMAIN} {
    reverse_proxy mullvad:9117
}

qbittorrent.{$MY_DOMAIN} {
    reverse_proxy mullvad:8080
}

bitwarden.{$MY_DOMAIN} {
    encode gzip

    header {
        # Enable cross-site filter (XSS) and tell browser to block detected attacks
        X-XSS-Protection "1; mode=block"
        # Disallow the site to be rendered within a frame (clickjacking protection)
        X-Frame-Options "DENY"
        # Prevent search engines from indexing (optional)
        X-Robots-Tag "none"
        # Server name removing
        -Server
    }

    # Notifications redirected to the websockets server
    reverse_proxy /notifications/hub bitwarden:3012

    # Proxy the Root directory to Rocket
    reverse_proxy bitwarden:80
}

3. The problem I’m having:

I want to route certain container traffic (jackett and qbittorrent) through a vpn.

I have a wireguard container that is on the same network as caddy, and it connect to a mullvad sever. For those 2 containers I use network_mode: container:mullvad as the network.

It appears as though caddy correctly sees those 2 containers, but when I go to the actual address I get this: HTTP ERROR 502.

I have the relevant error logs posted below.

4. Error messages and/or full log output:

{"level":"error","ts":1600277313.0205107,"logger":"http.log.error","msg":"dial tcp 192.168.80.2:9117: i/o timeout","request":{"method":"GET","uri":"/UI/Dashboard","proto":"HTTP/2.0","remote_addr":"<my public ip>:44672","host":"jackett.rypi.software","headers":{"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"Accept-Language":["en-US,en;q=0.9"],"Cookie":["Jackett=CfDJ8KbBTytGR8NNsWSsHP4qUiycGDNo5Kw2pjx29gHYf2C8xL2YA_VTv3_BMkVMNYfJipSUmZcm8XD34Zx4juPRRjCsU9IncCIvL3MB79Cirj2zEBULKNWaW-ff5n-3jp1gPMl4Ys1tCkUDFdD2JE90AR6RK7vgfbQfgdFHUsgva0cUBijJIyIJVi_nI0YeRvh7-JAwAFW50jBmpgYxtcvUGX5nex2w7qAvl_pp8A9pfiMfaFN8zJpjxAevq1gtysI752Xo_5R4i3csBwkVMMzTSZ5gkpfWRre3SeuORYQzJfDbiipC1ZQetlfALQgqwY1s2tOvSaO-YElyPG5guYjsTAY"],"Dnt":["1"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Dest":["document"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"ciphersuite":4867,"proto":"h2","proto_mutual":true,"server_name":"jackett.rypi.software"}},"duration":10.001932598,"status":502,"err_id":"jfvrgzf3r","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:411)"}

5. What I already tried:

I have researched possible solutions on these forums and using a search engine, but have not been able to find a solution.

I have done a lot of tweaking the network and hostnames and such, but I am pretty confident that the vpn connects correctly to the server, the containers are connected to the vpn, and that caddy can see the containers. The problem seems to lie with caddy interacting with the vpn, I just dont know where to go from here.

This post seems to be a similar error to mine, but went unsolved so I don’t know if @The_Spartan ever found a solution.

6. Links to relevant resources:

502 means Caddy couldn’t connect to the proxied service.

Could you get a /bin/sh shell into the Caddy container and use curl -v to see if you can hit those services? You may need to apk add curl since the Caddy image is slim.

I’ve never considered running web services through a VPN, but that seems like a pretty interesting usecase.

I don’t think this is actually an issue with Caddy right now (not much evidence as such) but likely just a misconfiguration of your VPN container. :man_shrugging:

This is what i get when I curl the address defined in my caddyfile:

/srv # curl -v jackett.rypi.software
*   Trying <public ip>:80...
* Connected to jackett.rypi.software (<public ip>) port 80 (#0)
> GET / HTTP/1.1
> Host: jackett.rypi.software
> User-Agent: curl/7.69.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://jackett.rypi.software/
< Server: Caddy
< Date: Wed, 16 Sep 2020 19:56:57 GMT
< Content-Length: 0
< 
* Closing connection 0

I think you misunderstood, I’m asking that you try to connect to mullvad:9117 from inside of the caddy container, i.e. try to do what Caddy is trying to do.

Do you mean like this?

/srv # curl -v mullvad:9117
*   Trying 192.168.80.2:9117...

I performed both actions inside the caddy container. This one just hangs at this point.

Here is an example of another service connected to caddy on the network (not using the vpn):

/srv # curl -v jellyfin:8096
*   Trying 192.168.80.5:8096...
* Connected to jellyfin (192.168.80.5) port 8096 (#0)
> GET / HTTP/1.1
> Host: jellyfin:8096
> User-Agent: curl/7.69.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Date: Wed, 16 Sep 2020 20:06:34 GMT
< Server: Kestrel

< Content-Length: 0
< Location: /web/index.html
< 
* Connection #0 to host jellyfin left intact

That confirms it’s not an issue with Caddy, definitely an issue with either that mullvad container, since it’s not accepting connections.

Ah ok, thank you for the help. Would you happen to have any idea how to start troubleshooting the other container or what the problem might be?

I am able to ping google.com in both the vpn container and the jackett container with no packet loss, so it looks like it properly connects to the internet. I’m not really sure where to start with this problem now.

Sorry, I have no idea. I’ve never attempted doing this sort of thing using a container as a wireguard proxy. Maybe the people who make that container might be able to explain how to use it that way?

Would there be a simpler way to route the container traffic through the vpn? I assumed creating the container and routing traffic through that would be the easiest option, but now that may not be the case.

This topic was automatically closed after 30 days. New replies are no longer allowed.