Debugging reverse proxy to localhost from docker

1. The problem I’m having:

I’m trying to set up Caddy (with security module) to secure and reverse proxy to a comsolserver. Caddy is running in a docker and the comsolserver is running on the host. I have set network to host in docker-compose, and yet still I get the " connection refused" (see below).

How can I debug this futher?

2. Error messages and/or full log output:

"http.log.error","msg":"dial tcp connect: connection refused"

from outside docker it responds fine:

$ curl -v http://localhost:2036
*   Trying
* Connected to localhost ( port 2036 (#0)
> GET / HTTP/1.1
> Host: localhost:2036
> User-Agent: curl/7.81.0
> Accept: */*
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Cache-Control: private
< Set-Cookie: JSESSIONID=A5EB26BD1E348BB6004F96548444681C; Path=/; HttpOnly
< Location: app-lib
< Content-Length: 0
< Date: Tue, 04 Apr 2023 07:28:36 GMT
< Server: Apache

3. Caddy version:

$ dc exec -it caddy caddy version
v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=

4. How I installed and ran Caddy:


a. System environment:

latest upgraded Ubuntu 22.04 with docker from PPA

b. Command:

docker compose up -d

c. Service/unit/compose file:

version: '3'
    container_name: caddy
      - secrets.env
      - 80:80
      - 443:443
    restart: unless-stopped
      - proxy
      - ./caddy_data:/data
      - ./Caddyfile:/etc/caddy/Caddyfile:ro

    driver: host

d. My complete Caddy config:


    order authenticate before respond
    order authorize before reverse_proxy
    security {
	local identity store localdb {
            realm local
            path /data/auth/users.json

        authentication portal myportal {
            crypto default token lifetime 3600
            crypto key sign-verify {env.JWT_SHARED_KEY}
            enable identity store localdb
            cookie domain {$DOMAIN}

	    ui {
		links {
			"COMSOL" https://server.{$DOMAIN} icon "las la-star"
			"My Identity" "/whoami" icon "las la-user"
	   transform user {
		match origin local
		action add role authp/user
		ui link "Portal Settings" /settings icon "las la-cog"

        authorization policy admin_policy {
            set auth url https://auth.{$DOMAIN}
            allow roles authp/admin authp/user
            crypto key verify {env.JWT_SHARED_KEY}


auth.{$DOMAIN}	 {
    authenticate with myportal

server.{$DOMAIN} {
	handle_path /files* {
	  authorize with admin_policy
	  file_server browse {
            root /tmp

	handle {
	  authorize with admin_policy
	  reverse_proxy * http://localhost:2036 {
             transport http {
               versions 1.1

I have stripped the mail/registration part of the Caddyfile security part, as I really think this issue is something obvious I can’t see right now with docker, and not about the security part. The file_server part works fine after I have authenticated…

5. Links to relevant resources:


the key was this:

However, you have to make sure your service on the actual host is listening on whatever host.docker.internal resolves to (usually - the docker0 interface).
Meaning, if you configured that service on the host to only listen on , you will have to change that.


So rolling with host mode on docker :confused:

When not using host mode, or localhost means “this container”, so Caddy tries to connect to something in the same container, which obviously doesn’t work.

Are you sure that using host.docker.internal wouldn’t work? That should only be the case if the service running on the host is specifically binding to, instead of all interfaces (which is typically the default).

I agree, I’m actually not sure what the problem really is, but at least working for host mode…

In bridge mode with the added host.docker.internal:host-gateway it actually seems to direct to the correct interface

outside container:

$ curl -v
*   Trying
* Connected to ( port 2036 (#0)
> GET / HTTP/1.1
> Host:
> User-Agent: curl/7.81.0
> Accept: */*
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Cache-Control: private
< Set-Cookie: JSESSIONID=4C39325B91C90AF717D98F4C8856C549; Path=/; HttpOnly
< Location: app-lib
< Content-Length: 0
< Date: Wed, 05 Apr 2023 08:13:55 GMT
< Server: Apache
* Connection #0 to host left intact

from caddy in docker bridge mode and doing reverse_proxy to http://host.docker.internal:2036

"logger":"http.log.error","msg":"dial tcp i/o timeout"

Any idea why this can be?

Is that Apache container running in Docker as well? In that case, you should use Docker’s networking to connect the two containers. Make sure they’re both in the same Docker network, then you can use the container’s name as the address.

Sadly no, the Apache server is served from a commercial tool running on the host (comsolserver, Reverse proxy setup for Model Manager server - Knowledge Base).

I would love not to have my Caddy container to run in host network mode, but for now it seems like the only way for some weird reason I don’t understand.

Is the apache server binding to instead of (i.e. all interfaces)? If so that would make it not accept requests coming from 172.x.x.x i.e. Docker.

The apache server is binding fine to the docker0 interface it seems, as I can get fine response from outside docker.
(using wget as curl is not inside the caddy docker)

From outside

$ wget -S --spider
Spider mode enabled. Check if remote file exists.
--2023-04-12 10:51:30--
Connecting to connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 302 
  P3P: CP="Not Applicable"
  Location: comsol-software-license-agreement
  Transfer-Encoding: chunked
  Date: Wed, 12 Apr 2023 10:51:30 GMT
  Keep-Alive: timeout=20
  Connection: keep-alive
  Server: Apache

From caddy docker in bridge mode with host-gateway added

/srv $ ping host.docker.internal:2036
PING host.docker.internal:2036 ( 56 data bytes
64 bytes from seq=0 ttl=64 time=0.112 ms
64 bytes from seq=1 ttl=64 time=0.112 ms
64 bytes from seq=2 ttl=64 time=0.130 ms
64 bytes from seq=3 ttl=64 time=0.128 ms
64 bytes from seq=4 ttl=64 time=0.135 ms
--- host.docker.internal:2036 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.112/0.123/0.135 ms

/srv $ wget host.docker.internal:2036 -S --spider
Connecting to host.docker.internal:2036 (

So I can ping fine host-gateway, but no response from the apache server from inside container?!? Any ideas?

I’m all out of ideas at this point.

You might get better help on this from the Docker forums

Yes I agree. Going with minimal curl image and standard python http.server and same problem - so NOT caddy :smiley:

I will return here and update and close when I figure out what is happening here.

Thanks for your effort though!

1 Like


ufw is blocking from the container as it is another network. I don’t want to expose the port widely, so instead I need to know the subnet of the network the container is using. Found the solution here:

in my case this solved it:

$ docker network inspect caddy_backend | grep Subnet
                    "Subnet": "",
$ sudo ufw allow in from
Rule added
1 Like

Nice, glad you figured it out! Didn’t realize ufw could affect stuff on the same machine, but makes sense.

You could allow instead which is the entire private range. That way you don’t need to worry if Docker uses a different subnet.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.