Debugging reverse proxy to localhost from docker

1. The problem I’m having:

I’m trying to set up Caddy (with security module) to secure and reverse proxy to a comsolserver. Caddy is running in a docker and the comsolserver is running on the host. I have set network to host in docker-compose, and yet still I get the " connection refused" (see below).

How can I debug this futher?

2. Error messages and/or full log output:

"http.log.error","msg":"dial tcp 127.0.0.1:2036: connect: connection refused"

from outside docker it responds fine:

$ curl -v http://localhost:2036
*   Trying 127.0.0.1:2036...
* Connected to localhost (127.0.0.1) port 2036 (#0)
> GET / HTTP/1.1
> Host: localhost:2036
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Cache-Control: private
< Set-Cookie: JSESSIONID=A5EB26BD1E348BB6004F96548444681C; Path=/; HttpOnly
< Location: app-lib
< Content-Length: 0
< Date: Tue, 04 Apr 2023 07:28:36 GMT
< Server: Apache
< 

3. Caddy version:

$ dc exec -it caddy caddy version
v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=

4. How I installed and ran Caddy:

docker

a. System environment:

latest upgraded Ubuntu 22.04 with docker from PPA

b. Command:

docker compose up -d

c. Service/unit/compose file:

version: '3'
services:
  caddy:
    image: ghcr.io/authp/authp
    container_name: caddy
    env_file:
      - secrets.env
    ports:
      - 80:80
      - 443:443
    restart: unless-stopped
    networks:
      - proxy
    volumes:
      - ./caddy_data:/data
      - ./Caddyfile:/etc/caddy/Caddyfile:ro

networks:
  proxy:
    driver: host

d. My complete Caddy config:

{
    debug

    order authenticate before respond
    order authorize before reverse_proxy
    
    security {
	local identity store localdb {
            realm local
            path /data/auth/users.json
        }

        authentication portal myportal {
            crypto default token lifetime 3600
            crypto key sign-verify {env.JWT_SHARED_KEY}
            enable identity store localdb
            cookie domain {$DOMAIN}

	    ui {
		links {
			"COMSOL" https://server.{$DOMAIN} icon "las la-star"
			"My Identity" "/whoami" icon "las la-user"
		}
	   }
	   transform user {
		match origin local
		action add role authp/user
		ui link "Portal Settings" /settings icon "las la-cog"
	   } 
        }

        authorization policy admin_policy {
            set auth url https://auth.{$DOMAIN}
            allow roles authp/admin authp/user
            crypto key verify {env.JWT_SHARED_KEY}
        }
    }


}

auth.{$DOMAIN}	 {
    authenticate with myportal
}

server.{$DOMAIN} {
	handle_path /files* {
	  authorize with admin_policy
	  file_server browse {
            root /tmp
          }
	}

	handle {
	  authorize with admin_policy
	  reverse_proxy * http://localhost:2036 {
             transport http {
               versions 1.1
             }
          }
	}
}

I have stripped the mail/registration part of the Caddyfile security part, as I really think this issue is something obvious I can’t see right now with docker, and not about the security part. The file_server part works fine after I have authenticated…

5. Links to relevant resources:

solved…

the key was this:

However, you have to make sure your service on the actual host is listening on whatever host.docker.internal resolves to (usually 172.17.0.1 - the docker0 interface).
Meaning, if you configured that service on the host to only listen on 127.0.0.1 , you will have to change that.

from:

So rolling with host mode on docker :confused:

When not using host mode, 127.0.0.1 or localhost means “this container”, so Caddy tries to connect to something in the same container, which obviously doesn’t work.

Are you sure that using host.docker.internal wouldn’t work? That should only be the case if the service running on the host is specifically binding to 127.0.0.1, instead of all interfaces (which is typically the default).

I agree, I’m actually not sure what the problem really is, but at least working for host mode…

In bridge mode with the added host.docker.internal:host-gateway it actually seems to direct to the correct interface 172.17.0.1

outside container:

$ curl -v 172.17.0.1:2036
*   Trying 172.17.0.1:2036...
* Connected to 172.17.0.1 (172.17.0.1) port 2036 (#0)
> GET / HTTP/1.1
> Host: 172.17.0.1:2036
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Cache-Control: private
< Set-Cookie: JSESSIONID=4C39325B91C90AF717D98F4C8856C549; Path=/; HttpOnly
< Location: app-lib
< Content-Length: 0
< Date: Wed, 05 Apr 2023 08:13:55 GMT
< Server: Apache
< 
* Connection #0 to host 172.17.0.1 left intact

from caddy in docker bridge mode and doing reverse_proxy to http://host.docker.internal:2036

"logger":"http.log.error","msg":"dial tcp 172.17.0.1:2036: i/o timeout"

Any idea why this can be?

Is that Apache container running in Docker as well? In that case, you should use Docker’s networking to connect the two containers. Make sure they’re both in the same Docker network, then you can use the container’s name as the address.

Sadly no, the Apache server is served from a commercial tool running on the host (comsolserver, Reverse proxy setup for Model Manager server - Knowledge Base).

I would love not to have my Caddy container to run in host network mode, but for now it seems like the only way for some weird reason I don’t understand.

Is the apache server binding to 127.0.0.1 instead of 0.0.0.0 (i.e. all interfaces)? If so that would make it not accept requests coming from 172.x.x.x i.e. Docker.

The apache server is binding fine to the docker0 interface it seems, as I can get fine response from outside docker.
(using wget as curl is not inside the caddy docker)

From outside

$ wget 172.17.0.1:2036 -S --spider
Spider mode enabled. Check if remote file exists.
--2023-04-12 10:51:30--  http://172.17.0.1:2036/
Connecting to 172.17.0.1:2036... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 302 
  P3P: CP="Not Applicable"
  Location: comsol-software-license-agreement
  Transfer-Encoding: chunked
  Date: Wed, 12 Apr 2023 10:51:30 GMT
  Keep-Alive: timeout=20
  Connection: keep-alive
  Server: Apache

From caddy docker in bridge mode with host-gateway added

/srv $ ping host.docker.internal:2036
PING host.docker.internal:2036 (172.17.0.1): 56 data bytes
64 bytes from 172.17.0.1: seq=0 ttl=64 time=0.112 ms
64 bytes from 172.17.0.1: seq=1 ttl=64 time=0.112 ms
64 bytes from 172.17.0.1: seq=2 ttl=64 time=0.130 ms
64 bytes from 172.17.0.1: seq=3 ttl=64 time=0.128 ms
64 bytes from 172.17.0.1: seq=4 ttl=64 time=0.135 ms
^C
--- host.docker.internal:2036 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.112/0.123/0.135 ms

/srv $ wget host.docker.internal:2036 -S --spider
Connecting to host.docker.internal:2036 (172.17.0.1:2036)
^C

So I can ping fine host-gateway, but no response from the apache server from inside container?!? Any ideas?

I’m all out of ideas at this point.

You might get better help on this from the Docker forums https://forums.docker.com/

Yes I agree. Going with minimal curl image and standard python http.server and same problem - so NOT caddy :smiley:

I will return here and update and close when I figure out what is happening here.

Thanks for your effort though!

1 Like

GOT IT!

ufw is blocking from the container as it is another network. I don’t want to expose the port widely, so instead I need to know the subnet of the network the container is using. Found the solution here:

in my case this solved it:

$ docker network inspect caddy_backend | grep Subnet
                    "Subnet": "172.22.0.0/16",
$ sudo ufw allow in from 172.22.0.0/16
Rule added
1 Like

Nice, glad you figured it out! Didn’t realize ufw could affect stuff on the same machine, but makes sense.

You could allow 172.16.0.0/12 instead which is the entire private range. That way you don’t need to worry if Docker uses a different subnet.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.