Reverse Proxy and Docker Socket

1. Caddy version (caddy version):

 caddy version
v2.4.3 h1:Y1FaV2N4WO3rBqxSYA8UZsZTQdN+PwcoOcAiZTM8C0I=

2. How I run Caddy:

Running caddy as a downloaded binary. What I’m trying to do is use Caddy as a reverse proxy in front of the Docker socket. Assuming I can get the basics working, I’m planning to add white-listing on specific paths to expose a safe subset of Docker’s functionality (e.g. for monitoring)

a. System environment:

OS: Ubuntu 20.04.2 LTS
Docker: 20.10.7

b. Command:

In one terminal

caddy run

In a second terminal

docker --debug -H tcp://localhost:2379 stats

d. My complete Caddyfile or JSON config:

{
  debug
}


http://:2379 {
  reverse_proxy unix///var/run/docker.sock {
    flush_interval -1
  }
}

3. The problem I’m having:

Some commands when run from the docker CLI will hang when run via Caddy as a reverse proxy. The stats command which is the example here and also the events command. Others like the info command work just fine, so the basic idea of reverse proxying the socket seems ok.

4. Error messages and/or full log output:

2021/08/15 13:44:27.244 INFO    using adjacent Caddyfile
2021/08/15 13:44:27.245 WARN    input is not formatted with 'caddy fmt' {"adapter": "caddyfile", "file": "Caddyfile", "line": 1}
2021/08/15 13:44:27.245 INFO    admin   admin endpoint started  {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2021/08/15 13:44:27.246 DEBUG   http    starting server loop    {"address": "[::]:2379", "http3": false, "tls": false}
2021/08/15 13:44:27.246 INFO    autosaved config (load with --resume flag)      {"file": "/home/rorym/.config/caddy/autosave.json"}
2021/08/15 13:44:27.247 INFO    serving initial configuration
2021/08/15 13:44:27.247 INFO    tls     cleaning storage unit   {"description": "FileStorage:/home/rorym/.local/share/caddy"}
2021/08/15 13:44:27.248 INFO    tls     finished cleaning storage units
2021/08/15 13:44:27.248 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc0003167e0"}
2021/08/15 13:44:41.184 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "unix///var/run/docker.sock", "request": {"remote_addr": "127.0.0.1:54704", "proto": "HTTP/1.1", "method": "HEAD", "host": "localhost:2379", "uri": "/_ping", "headers": {"User-Agent": ["Docker-Client/20.10.7 (linux)"], "X-Forwarded-Proto": ["http"], "X-Forwarded-For": ["127.0.0.1"]}}, "headers": {"Ostype": ["linux"], "Pragma": ["no-cache"], "Content-Type": ["text/plain; charset=utf-8"], "Docker-Experimental": ["false"], "Content-Length": ["0"], "Server": ["Docker/20.10.7 (linux)"], "Date": ["Sun, 15 Aug 2021 13:44:41 GMT"], "Api-Version": ["1.41"], "Cache-Control": ["no-cache, no-store, must-revalidate"]}, "status": 200}
2021/08/15 13:44:41.190 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "unix///var/run/docker.sock", "request": {"remote_addr": "127.0.0.1:54704", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:2379", "uri": "/v1.41/version", "headers": {"X-Forwarded-For": ["127.0.0.1"], "User-Agent": ["Docker-Client/20.10.7 (linux)"], "X-Forwarded-Proto": ["http"]}}, "headers": {"Server": ["Docker/20.10.7 (linux)"], "Date": ["Sun, 15 Aug 2021 13:44:41 GMT"], "Content-Length": ["851"], "Api-Version": ["1.41"], "Content-Type": ["application/json"], "Docker-Experimental": ["false"], "Ostype": ["linux"]}, "status": 200}
2021/08/15 13:44:41.191 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "unix///var/run/docker.sock", "request": {"remote_addr": "127.0.0.1:54704", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:2379", "uri": "/v1.41/events?filters=%7B%22type%22%3A%7B%22container%22%3Atrue%7D%7D", "headers": {"X-Forwarded-For": ["127.0.0.1"], "User-Agent": ["Docker-Client/20.10.7 (linux)"], "X-Forwarded-Proto": ["http"]}}, "headers": {"Content-Type": ["application/json"], "Docker-Experimental": ["false"], "Ostype": ["linux"], "Server": ["Docker/20.10.7 (linux)"], "Date": ["Sun, 15 Aug 2021 13:44:41 GMT"], "Api-Version": ["1.41"]}, "status": 200}

5. What I already tried:

I tried the flush_interval -1 command (as shown in the config file). I’ve also tried requesting the URL with curl directly (instead of just using the Docker CLI), with the same outcome (URL was curl http://127.0.0.1:2379/v1.41/events?filters=%7B%22type%22%3A%7B%22container%22%3Atrue%7D%7D)

6. Links to relevant resources:

Hmm, I’ve not heard of anyone trying this before.

We’ve recently made changes on the master branch which had to do with making flush_interval work better. It’s possible that the changes might resolve your issue.

Could you try with the build from this commit, from our CI artifacts?

Awesome thanks for that, I downloaded the build you linked and re-tried with the configuration otherwise identical and it’s working fine now!

So looks like the changes made have improved things, for this use case anyway :slight_smile:

1 Like

Perfect! Glad it was that simple.

We’ll have v2.4.4 out relatively soon, but in the meantime, I suppose you can use that build.

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.