Caddy in docker reverse proxy to a tunnel to local machine not working

Help
1

For development, a lot of platforms offer a webhook event but require HTTPS and a valid domain. A lot of people use ngrok, but I had an EC2 instance on hand and it allows me to get a non-randomized subdomain for the webhook.

I had the following working with nginx:

subdomain -> ec2 instance running nginx as a process -> tunnel -> local machine

The change is to move from nginx to caddy in a docker instance using caddy-docker-proxy:

subdomain -> docker/caddy-docker-proxy running on ec2 instance -> tunnel -> local machine

This is my caddy settings:

  caddy:
    image: lucaslorentz/caddy-docker-proxy:ci-alpine
    container_name: caddy
    ports:
      - 80:80
      - 443:443
    environment:
      - CADDY_INGRESS_NETWORKS=caddy
    networks:
      - caddy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - caddy_data:/data
    restart: unless-stopped
    labels:
      caddy_0: dev.domain.tld
      caddy_0.reverse_proxy: 172.17.0.1:9000

I can confirm that caddy gets the request from the subdomain based on the logs.
I can also confirm that if I curl localhost:9000/test from the EC2 instance, it works.

This means that there is a disconnect between the docker container running caddy-docker-proxy and the host machine/EC2 instance.

Here’s the error log from Caddy:

{
  "level": "error",
  "ts": 1695592513.784038,
  "logger": "http.log.error",
  "msg": "dial tcp :9000: connect: connection refused",
  "request": {
    "remote_ip": "redacted",
    "remote_port": "60083",
    "client_ip": "redacted",
    "proto": "HTTP/2.0",
    "method": "GET",
    "host": ".com",
    "uri": "/test",
    "headers": {
      "Sec-Fetch-User": [
        "?1"
      ],
      "Accept-Encoding": [
        "gzip, deflate, br"
      ],
      "Accept-Language": [
        "en-US,en;q=0.9,de;q=0.8"
      ],
      "Sec-Ch-Ua": [
        "\"Chromium\";v=\"116\", \"Not)A;Brand\";v=\"24\", \"Google Chrome\";v=\"116\""
      ],
      "Sec-Ch-Ua-Mobile": [
        "?0"
      ],
      "Sec-Ch-Ua-Platform": [
        "\"macOS\""
      ],
      "Upgrade-Insecure-Requests": [
        "1"
      ],
      "User-Agent": [
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"
      ],
      "Accept": [
        "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
      ],
      "Sec-Fetch-Site": [
        "none"
      ],
      "Sec-Fetch-Mode": [
        "navigate"
      ],
      "Cache-Control": [
        "max-age=0"
      ],
      "Sec-Fetch-Dest": [
        "document"
      ]
    },
    "tls": {
      "resumed": false,
      "version": 772,
      "cipher_suite": 4865,
      "proto": "h2",
      "server_name": "dev.domain.tld"
    }
  },
  "duration": 0.00028104,
  "status": 502,
  "err_id": "ra595z5i9",
  "err_trace": "reverseproxy.statusError (reverseproxy.go:1248)"
}

I run my tunnel from my Macbook with this:

tunneldev() {
  emulate -LR bash;
  echo "Tunneling $1 on core server to $2 on localhost";
  ssh -R $1:localhost:$2 -N lab;
}

tunneldev 9000 3000

Feels like I just need one more setting to allow the docker container to properly send traffic through the tunnel but can’t find it.

2

I think this will only accept connections coming from localhost, but containers are not in localhost, they’re in their own virtual network, i.e. 172.x.x.x. You probably want to use 0.0.0.0 instead of localhost here so it binds to all interfaces.

Either way, this isn’t anything to do with Caddy in particular, this is just a general networking thing.

3

@francislavoie thanks for the tip! I just tried changing my function to 0.0.0.0 but still the same behaviour. Wondering if I maybe need to set the host to 172.x.x.x?

4

No, that was just an example, that’s not a real thing.

You might need to change this to host.docker.internal:9000 to properly point to your host machine though. Google host.docker.internal for more info.

5

172.17.0.1 is the equivalent of that for older versions of docker engine on linux: What is linux equivalent of "host.docker.internal" - Stack Overflow

Probably worth it to note that I have some other services running directly on the EC2 instance (not in a container) that’s configured like this and Caddy is able to send traffic there.

6

Not necessarily. If you have more than one Docker network (e.g. more than one docker-compose project) then that won’t be true. Also, host.docker.internal should work on linux with latest versions of Docker. Check the dates on the things you read.

Either way, like I said, this isn’t really a Caddy problem, it’s a general networking problem. You might get better help elsewhere on those parts.

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.