Unable to reverse proxy traffic from Caddy (Docker) to Host Machine (Code-Server as Systemd)

[PancakePuncher]

1. The problem I’m having:

I have been absolutely beating my head against a wall trying to understand if I can get Caddy which is running in a docker container to send traffic to a non-dockerized application on the host machine.

Essentially, I have Code-Server hosted locally on my machine as a systemd. I do not wish to dockerize Code-Server because I want the benefit of being able to directly interact with the host machine when using it. (Ie, direct terminal access, actual files on the machine, etc.)

I do not wish to run the container in Host or Bridge mode because I want Caddy to also be able to send traffic to any dockers in the future I plan to run. (Ie. portainer, personal projects, etc.) I also want the benefits of Docker Networking for containers.

Code-Server is listening at 0.0.0.0:8080.

When I run Caddy as a system service, no issues.

But if I dockerize Caddy everything I’ve googled and tried does not work.

So from my Domain I want to be able to capture traffic to pancakepuncher.com/code to code-server.

All the resources I’ve read indicate it should be as simple as finding the host machines IP for the docker network. Based on my config this appears to be 172.20.0.1 (Gateway of the docker network). I confirmed this with “hostname -I” and “ip addr ls”.

I’ve tried so many things.

I’ll list off some of it:

1. I’ve tried using all the IP’s listed under hostname -I except for the Public IP.
2. I’ve tried configuring code-server to listen on the “Gateway” IP. I’ve tried giving it it’s own IP in that same “Network”. I’ve set it to 0.0.0.0, localhost, 127.0.0.1. Best I can discern is that the recommended for an application to accept traffic universally is 0.0.0.0.
3. I’ve configured a basic “whoami” container alongside Caddy and Code-Server and can confirm that Caddy sends the traffic to the whoami app at “/whoami” but it just can’t seem to route to Code-Server on the host machine.

I’ve exhausted all of the possible options I can consider and just genuinely can’t believe that what I am trying to do is impossible. I’ve read so many articles. I’ve even asked ChatGPT and I’ve gotten mixed information that it’s not possible for Docker to do this, yet other resources say “Just reverse proxy to the localhost IP.”

Any help is greatly appreciated.

2. Error messages and/or full log output:

Edit: I see in line 2 it’s saying the Caddyfile isn’t formatted. I went and formatted it and still fails.

caddy-caddy-1  | {"level":"info","ts":1703182225.8478878,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy-caddy-1  | {"level":"warn","ts":1703182225.8506181,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
caddy-caddy-1  | {"level":"info","ts":1703182225.8523543,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//[::1]:2019","//127.0.0.1:2019","//localhost:2019"]}
caddy-caddy-1  | {"level":"info","ts":1703182225.8528814,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy-caddy-1  | {"level":"info","ts":1703182225.8529165,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy-caddy-1  | {"level":"info","ts":1703182225.8530607,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00050ce80"}
caddy-caddy-1  | {"level":"info","ts":1703182225.8539634,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
caddy-caddy-1  | {"level":"info","ts":1703182225.854263,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddy-caddy-1  | {"level":"info","ts":1703182225.8543303,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddy-caddy-1  | {"level":"info","ts":1703182225.854342,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["pancakepuncher.com"]}
caddy-caddy-1  | {"level":"info","ts":1703182225.8547323,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy-caddy-1  | {"level":"info","ts":1703182225.8547592,"msg":"serving initial configuration"}
caddy-caddy-1  | {"level":"info","ts":1703182225.855001,"logger":"tls.obtain","msg":"acquiring lock","identifier":"pancakepuncher.com"}
caddy-caddy-1  | {"level":"warn","ts":1703182225.8559046,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"849feaeb-f208-4f38-b501-203acbee6c19","try_again":1703268625.8559024,"try_again_in":86399.999999436}
caddy-caddy-1  | {"level":"info","ts":1703182225.8560653,"logger":"tls","msg":"finished cleaning storage units"}
caddy-caddy-1  | {"level":"info","ts":1703182225.8569715,"logger":"tls.obtain","msg":"lock acquired","identifier":"pancakepuncher.com"}
caddy-caddy-1  | {"level":"info","ts":1703182225.8572795,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"pancakepuncher.com"}
caddy-caddy-1  | {"level":"info","ts":1703182225.8589325,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["pancakepuncher.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
caddy-caddy-1  | {"level":"info","ts":1703182225.8589544,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["pancakepuncher.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
caddy-caddy-1  | {"level":"info","ts":1703182226.383774,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"pancakepuncher.com","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
caddy-caddy-1  | {"level":"info","ts":1703182226.5527983,"logger":"http","msg":"served key authentication","identifier":"pancakepuncher.com","challenge":"http-01","remote":"172.70.179.103:9556","distributed":false}
caddy-caddy-1  | {"level":"info","ts":1703182226.7082229,"logger":"http","msg":"served key authentication","identifier":"pancakepuncher.com","challenge":"http-01","remote":"162.158.245.66:60992","distributed":false}
caddy-caddy-1  | {"level":"info","ts":1703182226.830831,"logger":"http","msg":"served key authentication","identifier":"pancakepuncher.com","challenge":"http-01","remote":"108.162.245.141:38818","distributed":false}
caddy-caddy-1  | {"level":"info","ts":1703182227.1281557,"logger":"http.acme_client","msg":"authorization finalized","identifier":"pancakepuncher.com","authz_status":"valid"}
caddy-caddy-1  | {"level":"info","ts":1703182227.1282039,"logger":"http.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme-v02.api.letsencrypt.org/acme/order/1477471916/230856017586"}
caddy-caddy-1  | {"level":"info","ts":1703182227.6197853,"logger":"http.acme_client","msg":"successfully downloaded available certificate chains","count":2,"first_url":"https://acme-v02.api.letsencrypt.org/acme/cert/03166f650dd285a08b4be12a2c0e1055d8ee"}
caddy-caddy-1  | {"level":"info","ts":1703182227.6204472,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"pancakepuncher.com"}
caddy-caddy-1  | {"level":"info","ts":1703182227.620613,"logger":"tls.obtain","msg":"releasing lock","identifier":"pancakepuncher.com"}
caddy-caddy-1  | {"level":"error","ts":1703182241.464922,"logger":"http.log.error","msg":"dial tcp 172.20.0.1:8080: i/o timeout","request":{"remote_ip":"172.71.22.45","remote_port":"10698","client_ip":"172.71.22.45","proto":"HTTP/2.0","method":"GET","host":"pancakepuncher.com","uri":"/code/","headers":{"Cf-Ipcountry":["US"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Cf-Connecting-Ip":["68.114.35.5"],"Cdn-Loop":["cloudflare"],"Sec-Ch-Ua-Mobile":["?0"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Accept-Language":["en-US,en;q=0.9,la;q=0.8"],"Cf-Ray":["8392153ded76b0d9-ATL"],"Sec-Ch-Ua":["\"Google Chrome\";v=\"119\", \"Chromium\";v=\"119\", \"Not?A_Brand\";v=\"24\""],"X-Forwarded-For":["68.114.35.5"],"X-Forwarded-Proto":["https"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-Site":["none"],"Accept-Encoding":["gzip"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"pancakepuncher.com"}},"duration":3.00356047,"status":502,"err_id":"u5naxwnsj","err_trace":"reverseproxy.statusError (reverseproxy.go:1267)"}
caddy-caddy-1  | {"level":"info","ts":1703182971.9205139,"msg":"shutting down apps, then terminating","signal":"SIGTERM"}
caddy-caddy-1  | {"level":"warn","ts":1703182971.9205818,"msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
caddy-caddy-1  | {"level":"info","ts":1703182971.9206572,"logger":"http","msg":"servers shutting down with eternal grace period"}
caddy-caddy-1  | {"level":"info","ts":1703182971.9211495,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
caddy-caddy-1  | {"level":"info","ts":1703182971.921201,"msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
caddy-caddy-1  | {"level":"info","ts":1703182977.7606215,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy-caddy-1  | {"level":"warn","ts":1703182977.7621322,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
caddy-caddy-1  | {"level":"info","ts":1703182977.763228,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//[::1]:2019","//127.0.0.1:2019","//localhost:2019"]}
caddy-caddy-1  | {"level":"info","ts":1703182977.7634773,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000689700"}
caddy-caddy-1  | {"level":"info","ts":1703182977.763528,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy-caddy-1  | {"level":"info","ts":1703182977.7635436,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy-caddy-1  | {"level":"info","ts":1703182977.7641044,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
caddy-caddy-1  | {"level":"info","ts":1703182977.764343,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddy-caddy-1  | {"level":"info","ts":1703182977.7644136,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddy-caddy-1  | {"level":"info","ts":1703182977.7644293,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["pancakepuncher.com"]}
caddy-caddy-1  | {"level":"info","ts":1703182977.7656374,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy-caddy-1  | {"level":"info","ts":1703182977.7656577,"msg":"serving initial configuration"}
caddy-caddy-1  | {"level":"warn","ts":1703182977.7658086,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"849feaeb-f208-4f38-b501-203acbee6c19","try_again":1703269377.7658052,"try_again_in":86399.999999484}
caddy-caddy-1  | {"level":"info","ts":1703182977.7659295,"logger":"tls","msg":"finished cleaning storage units"}
caddy-caddy-1  | {"level":"error","ts":1703183082.006614,"logger":"http.log.error","msg":"dial tcp 172.20.0.1:8080: i/o timeout","request":{"remote_ip":"172.71.26.26","remote_port":"39636","client_ip":"172.71.26.26","proto":"HTTP/2.0","method":"GET","host":"pancakepuncher.com","uri":"/code/","headers":{"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Sec-Fetch-Dest":["document"],"Cf-Ipcountry":["US"],"Sec-Ch-Ua":["\"Google Chrome\";v=\"119\", \"Chromium\";v=\"119\", \"Not?A_Brand\";v=\"24\""],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"],"Sec-Fetch-User":["?1"],"Accept-Language":["en-US,en;q=0.9,la;q=0.8"],"Cf-Connecting-Ip":["68.114.35.5"],"Cdn-Loop":["cloudflare"],"X-Forwarded-For":["68.114.35.5"],"Cf-Ray":["839229c2eb36243f-ATL"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-Mode":["navigate"],"Accept-Encoding":["gzip"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cache-Control":["max-age=0"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Site":["none"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"pancakepuncher.com"}},"duration":3.003401594,"status":502,"err_id":"g2henjp1x","err_trace":"reverseproxy.statusError (reverseproxy.go:1267)"}

3. Caddy version:

v2.7.6

4. How I installed and ran Caddy:

a. System environment:

Debian 12 Docker Compose

b. Command:

docker compose up -d

c. Service/unit/compose file:

version: "3.9"

services:
  caddy:
    image: caddy:latest
    restart: unless-stopped
    networks:
      - caddy_proxy
    cap_add:
      - NET_ADMIN
    ports:
      - "2019:2019"
      - "80:80"
      - "443:443"
      - "443:443/udp"
    volumes:
      - /home/admin/docker_stack/caddy/Caddyfile:/etc/caddy/Caddyfile
      - /home/admin/docker_stack/caddy/site:/srv
      - /home/admin/docker_stack/caddy/caddy_data:/data
      - /home/admin/docker_stack/caddy/caddy_config:/config

volumes:
  caddy_data:
    external: true
  caddy_config:

networks:
  caddy_proxy:
    external: true

d. My complete Caddy config:

pancakepuncher.com {

	handle_path /code* {
		reverse_proxy 172.20.0.1:8080
	}
}

[francislavoie]

@PancakePuncher

I have been absolutely beating my head against a wall trying to understand if I can get Caddy which is running in a docker container to send traffic to a non-dockerized application on the host machine.

Yes, use host.docker.internal to point to the host (instead of an IP address). Google it.

The docker host IP is not guaranteed to stay the same, so that hostname is provided to make it work.

@PancakePuncher

	handle_path /code* {
		reverse_proxy 172.20.0.1:8080
	}

Keep in mind that you don’t have any routes for anything other than /code* so you’ll get empty responses instead.

In general, I recommend using subdomains instead of subpaths. So I’d suggest making a code.pancakepuncher.com subdomain for this.

See this article which explains:

The "subfolder problem", OR, "why can't I reverse proxy my app into a subfolder?"

Have you ever tried to reverse proxy an app into its own little subfolder of your domain name? Makes things neat, doesn’t it? Using example.com/foo/ for one app, example.com/bar/ for another. If you’re coming here from one of the selfhosted communities, you might be thinking along the lines of example.com/sonarr/, example.com/radarr/ etc. Chances are, you’ve tried some configuration along these lines: example.com { redir /sonarr /sonarr/ handle_path /sonarr/* { reverse_proxy localhost…

[PancakePuncher]

Unfortunately I did try that, and I just re-tried it as well.

Documentation on host.docker.internal is honestly really hard to determine much about. I see lots of older documentation saying it’s not supported on Linux, but I see newer documentation saying it is.

I also see people saying I may need to add it into my Compose File? Are you familiar with this?

I added it into my Caddy Compose file just to see if that changed anything. It did not.

version: "3.9"

services:
  caddy:
    image: caddy:latest
    restart: unless-stopped
    networks:
      - caddy_proxy
    cap_add:
      - NET_ADMIN
    extra_hosts:
      - host.docker.internal:host-gateway
    ports:
      - "2019:2019"
      - "80:80"
      - "443:443"
      - "443:443/udp"
    volumes:
      - /home/admin/docker_stack/caddy/Caddyfile:/etc/caddy/Caddyfile
      - /home/admin/docker_stack/caddy/site:/srv
      - /home/admin/docker_stack/caddy/caddy_data:/data
      - /home/admin/docker_stack/caddy/caddy_config:/config

volumes:
  caddy_data:
    external: true
  caddy_config:

networks:
  caddy_proxy:
    external: true

Error I get

caddy-caddy-1  | {"level":"error","ts":1703202736.9951582,"logger":"http.log.error","msg":"dial tcp: lookup host.docker.internal on 127.0.0.11:53: no such host","request":{"remote_ip":"172.70.127.41","remote_port":"44602","client_ip":"172.70.127.41","proto":"HTTP/2.0","method":"GET","host":"pancakepuncher.com","uri":"/code/","headers":{"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Accept-Language":["en-US,en;q=0.9"],"Cf-Connecting-Ip":["68.184.14.24"],"Sec-Fetch-Dest":["document"],"Cdn-Loop":["cloudflare"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Sec-Gpc":["1"],"Sec-Fetch-Site":["none"],"Cf-Ipcountry":["US"],"Accept-Encoding":["gzip"],"Cf-Ray":["839409b1de0b2f2c-ORD"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Sec-Ch-Ua":["\"Not_A Brand\";v=\"8\", \"Chromium\";v=\"120\", \"Brave\";v=\"120\""],"Priority":["u=0, i"],"X-Forwarded-For":["68.184.14.24"],"X-Forwarded-Proto":["https"],"Upgrade-Insecure-Requests":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"pancakepuncher.com"}},"duration":0.001582163,"status":502,"err_id":"1ymqk6iaq","err_trace":"reverseproxy.statusError (reverseproxy.go:1267)"}

Caddyfile

pancakepuncher.com {

	handle_path /code* {
		reverse_proxy host.docker.internal:8080
	}
}

To your point about the sub-domains. I believe I need to be using TLS through a DNS like Cloudflare in order to use sub-domains, correct? As when using the built in TLS of Caddy any sub-domains I attempt to use fail on challenge. When I attempted to setup Cloudflare TLS I got a bunch of errors about not having the module. So if you’d like to assist with helping me set that up as well you can lol.

[francislavoie]

What version of Docker are you running?

@PancakePuncher

To your point about the sub-domains. I believe I need to be using TLS through a DNS like Cloudflare in order to use sub-domains, correct?

You just need a DNS A record for * (or the specific subdomain name) pointing to your IP address. That’s all. Caddy will automate issuance of the TLS cert as long as DNS and networking are set up correctly.