How to listen on Unix scokets receiving cleartext HTTP and PROXY protocol

Help
1

1. Caddy version:

v2.6.3 h1:QRVBNIqfpqZ1eJacY44I6eUC1OcxQ8D04EKImzpj7S8=

2. How I installed, and run Caddy:

Docker

a. System environment:

Ubuntu 22.04

3. The problem I’m having:

I’m having trouble adapting the following NGINX config to Caddy:

server {
    listen unix:/dev/shm/h2c.sock http2 proxy_protocol; # H2C server monitor process and enable PROXY protocol reception
    set_real_ip_from unix:;
    real_ip_header proxy_protocol;
    server_name example.com cdn.com;

    # grpc settings
    grpc_read_timeout 1h;
    grpc_send_timeout 1h;
    grpc_set_header X-Real-IP $remote_addr;

    # website
    location / {
        add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always; # enable HSTS
        root /var/www/html;
        index index.html index.htm;
    }

    location /servicename/* {
        # POST returns 404 when negotiation fails
        if ($request_method != "POST") {
            return 404;
        }
        client_body_buffer_size 1m;
        client_body_timeout 1h;
        client_max_body_size 0;
        grpc_pass grpc://127.0.0.1:3001;

    }
}

The extra settings (timeout, etc) are not very important to me, the only important part is the first 4 lines of the server block!

5. What I already tried:

I have tried the nginx-adapter plugin but it throws errors saying it does not recognize the listen directive!

2

Your nginx config is not wrapped in http, so it was confused. Wrap the config as such

http {
 # the config you've posted
}

It works in this manner.

If you want to write the Caddyfile config from scratch, you should be looking at the bind directive. It corresponds to the listen field in the http server app.

3

Your nginx config is not wrapped in http, so it was confused.

Okay it passed but now it says

WARN	nginx	set_real_ip_from: unrecognized or unsupported nginx directive
WARN	nginx	real_ip_header: unrecognized or unsupported nginx directive

This was exactly the part I was having trouble with! Enabling the PROXY header and H2C. :sweat_smile:

4

I stumbled upon this plugin (not sure if it’s what I’m looking for) and added the listener_wrappers key to my json as the following, but still it’s not working:

        {
            "http": {
                "servers": {
                    "fallback": {
                        "listener_wrappers": [
                            {
                                "wrapper": "proxy_protocol"
                            }
                        ],
                        "listen": [
                            "unix//dev/shm/H2C.socket"
                        ],
                        "routes": [
                            {
                                "match": [
                                    {
                                        "host": [
                                            "sub.example.com"
                                        ]
                                    }
                                ],
                                "handle": [
                                    {
                                        "handler": "subroute",
                                        "routes": [
                                            {
                                                "match": [
                                                    {
                                                        "path": [
                                                            "/svc_name/*"
                                                        ]
                                                    }
                                                ],
                                                "handle": [
                                                    {
                                                        "flush_interval": -1,
                                                        "handler": "reverse_proxy",
                                                        "headers": {
                                                            "request": {
                                                                "set": {
                                                                    "X-Real-Ip": [
                                                                        "{http.request.remote.host}"
                                                                    ]
                                                                }
                                                            }
                                                        },
                                                        "upstreams": [
                                                            {
                                                                "dial": "a_docker_container:3443"
                                                            }
                                                        ]
                                                    }
                                                ]
                                            }
                                        ]
                                    }
                                ]
                            }
                        ]
                    }
                }
            }
        }
1 Like
5

The nginx-adapter provides best-effort translation, and not 100% perfect. I have just caught a few bugs in the nginx-adapter:

Here’s the full list of warnings after fixing the missed warnings issue if you’re interested in knowing them:

{"level":"warn","ts":1676844755.510939,"logger":"nginx","msg":"set_real_ip_from: unrecognized or unsupported nginx directive","file":"forum.conf","line":6}
{"level":"warn","ts":1676844755.511026,"logger":"nginx","msg":"real_ip_header: unrecognized or unsupported nginx directive","file":"forum.conf","line":7}
{"level":"warn","ts":1676844755.511032,"logger":"nginx","msg":"grpc_read_timeout: unrecognized or unsupported nginx directive","file":"forum.conf","line":11}
{"level":"warn","ts":1676844755.511036,"logger":"nginx","msg":"grpc_send_timeout: unrecognized or unsupported nginx directive","file":"forum.conf","line":12}
{"level":"warn","ts":1676844755.511042,"logger":"nginx","msg":"grpc_set_header: unrecognized or unsupported nginx directive","file":"forum.conf","line":13}
{"level":"warn","ts":1676844755.511132,"logger":"nginx","msg":"index: unrecognized or unsupported nginx directive","file":"forum.conf","line":19}
{"level":"warn","ts":1676844755.511141,"logger":"nginx","msg":"client_body_buffer_size: unrecognized or unsupported nginx directive","file":"forum.conf","line":27}
{"level":"warn","ts":1676844755.511146,"logger":"nginx","msg":"client_body_timeout: unrecognized or unsupported nginx directive","file":"forum.conf","line":28}
{"level":"warn","ts":1676844755.5111501,"logger":"nginx","msg":"client_max_body_size: unrecognized or unsupported nginx directive","file":"forum.conf","line":29}
{"level":"warn","ts":1676844755.511154,"logger":"nginx","msg":"grpc_pass: unrecognized or unsupported nginx directive","file":"forum.conf","line":30}

You beat me! :slightly_smiling_face:

Your Caddy JSON is close. You need to tell it to enable h2c, which is plaintext grpc, in the transport configuration of the reverse_proxy handler. Check the addition below.

{
    "http": {
        "servers": {
            "fallback": {
                "listener_wrappers": [
                    {
                        "wrapper": "proxy_protocol"
                    }
                ],
                "listen": [
                    "unix//dev/shm/H2C.socket"
                ],
                "routes": [
                    {
                        "match": [
                            {
                                "host": [
                                    "sub.example.com"
                                ]
                            }
                        ],
                        "handle": [
                            {
                                "handler": "subroute",
                                "routes": [
                                    {
                                        "match": [
                                            {
                                                "path": [
                                                    "/svc_name/*"
                                                ]
                                            }
                                        ],
                                        "handle": [
                                            {
                                                "flush_interval": -1,
                                                "handler": "reverse_proxy",
                                                "transport": {
                                                    "protocol": "http",
                                                    "versions": [
                                                        "h2c",
                                                        "2"
                                                    ]
                                                },
                                                "headers": {
                                                    "request": {
                                                        "set": {
                                                            "X-Real-Ip": [
                                                                "{http.request.remote.host}"
                                                            ]
                                                        }
                                                    }
                                                },
                                                "upstreams": [
                                                    {
                                                        "dial": "a_docker_container:3443"
                                                    }
                                                ]
                                            }
                                        ]
                                    }
                                ]
                            }
                        ]
                    }
                ]
            }
        }
    }
}

Depending on your setup, you might need the tls listener_wrapper, set the timeouts, and so on.

2 Likes
6

You’re the best! All is working now! :slight_smile:

The only concern I have is that this plugin goes unmaintained and incompatible with future releases! I wish this was merged as a standard feature in Caddy.

7

Don’t worry - it’s very stable code, the protocol doesn’t change. The listener wrapper API is also stable. If we need to make a change to the API, then we’ll be sure that it gets updated.

1 Like
8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.