Caddy behind NLB

Help
1

1. Caddy version (caddy version):

2.1.1

2. How I run Caddy:

We run Caddy inside of a docker container on AWS Fargate
with a json caddyfile that is generated form YAML on the fly (when starting the container)

a. System environment:

docker on fargate

b. Command:

/usr/bin/caddy run --config /etc/caddy/config.json

d. My complete Caddyfile or JSON config:

not easily possible as it is generated on the fly

3. The problem I’m having:

We are running caddy in a sandwich between an NLB and an ALB

NLB => Caddy => ALB => Applications

Caddy is mostly used for SSL termination. We recently noticed that the X-Forwareded-For Header that comes from caddy does not contain the actual web browser IP address, but the NLB’s internal IP Address. Is there any way to get the client IP address instead of the NLBs IP Address?

2

That’s a very old version. Please upgrade to v2.4.5! It might fix your problem with X-Forwarded-For.

3

Updated to 2.4.5. Still does not work.
Now https://<loadbalancer-url> just loads forever and in caddy I don’t see any logs at all
NLB in front of caddy has PROXY PROTOCOL v2 enabled

Building in docker:

ARG GO_VERSION="1.17.2"
FROM golang:${GO_VERSION}-alpine AS builder

ARG CADDY_VERSION="2.4.5"
ARG XCADDY_VERSION="0.2.0"
ARG DYNAMODB_STORAGE_VERSION="2.0.1"

RUN wget -O xcaddy.tar.gz "https://github.com/caddyserver/xcaddy/releases/download/v${XCADDY_VERSION}/xcaddy_${XCADDY_VERSION}_linux_amd64.tar.gz"; \
    tar x -z -f xcaddy.tar.gz -C /usr/bin xcaddy; \
    chmod +x /usr/bin/xcaddy;

RUN /usr/bin/xcaddy build v${CADDY_VERSION} \
    --output /usr/bin/caddy \
    --with github.com/silinternational/certmagic-storage-dynamodb/v2@${DYNAMODB_STORAGE_VERSION} \
    --with github.com/mastercactapus/caddy2-proxyprotocol

Full Caddyfile (generated)

{
   "apps":{
      "tls":{
         "automation":{
            "on_demand":{
               "ask":"http://127.0.0.1:8080/ask"
            },
            "policies":[
               {
                  "issuers":[
                     {
                        "email":"{env.EMAIL}",
                        "module":"acme"
                     }
                  ],
                  "on_demand":true
               }
            ]
         },
         "certificates":{
            "load_folders":[
               "/certs"
            ]
         }
      },
      "http":{
         "servers":{
            "secure":{
               "listener_wrappers":[
                  {
                     "wrapper":"proxy_protocol",
                     "allow":[
                        "192.168.0.0/16",
                        "10.0.0.0/8"
                     ]
                  },
                  {
                     "wrapper":"tls"
                  }
               ],
               "listen":[
                  ":443"
               ],
               "routes":[
                  {
                     "handle":[
                        {
                           "handler":"subroute",
                           "routes":[
                              {
                                 "handle":[
                                    {
                                       "handler":"headers",
                                       "response":{
                                          "delete":[
                                             "ALB"
                                          ],
                                          "set":{
                                             "Referrer-Policy":[
                                                "strict-origin-when-cross-origin"
                                             ],
                                             "Strict-Transport-Security":[
                                                "max-age=63072000; preload"
                                             ],
                                             "Content-Security-Policy":[
                                                "default-src 'self'; img-src data: *; media-src 'self' *; child-src blob: *; frame-src blob: *; style-src 'self' 'unsafe-inline' bitpub-euc1.s3.amazonaws.com bitpub-euc1.s3.eu-central-1.amazonaws.com bitpub-usw1-live.s3.us-west-1.amazonaws.com bitpub-use1-live.s3.us-east-1.amazonaws.com bitpub-euc1-staging.s3.amazonaws.com bitpub-euc1-staging.s3.eu-central-1.amazonaws.com bitpub-usw1-staging.s3.us-west-1.amazonaws.com bitpub-use1-staging.s3.us-east-1.amazonaws.com blinkit-branding.s3.eu-central-1.amazonaws.com s3-eu-central-1.amazonaws.com fonts.googleapis.com translate.googleapis.com; font-src 'unsafe-inline' data: *; script-src 'self' 'unsafe-inline' beacon-v2.helpscout.net app.satismeter.com; connect-src 'self' blob: *;"
                                             ]
                                          }
                                       }
                                    },
                                    {
                                       "handler":"reverse_proxy",
                                       "transport":{
                                          "protocol":"http",
                                          "tls":{
                                             "insecure_skip_verify":true
                                          }
                                       },
                                       "upstreams":[
                                          {
                                             "dial":"{env.ENDPOINT}"
                                          }
                                       ],
                                       "handle_response":[
                                          {
                                             "match":{
                                                "status_code":[
                                                   5
                                                ]
                                             },
                                             "routes":[
                                                {
                                                   "handle":[
                                                      {
                                                         "handler":"file_server",
                                                         "root":"/var/www/html",
                                                         "index_names":[
                                                            "500.html"
                                                         ]
                                                      }
                                                   ]
                                                }
                                             ]
                                          }
                                       ]
                                    }
                                 ]
                              }
                           ]
                        }
                     ],
                     "terminal":true
                  }
               ],
               "tls_connection_policies":[
                  {
                     "cipher_suites":[
                        "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
                        "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
                        "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
                        "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
                        "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",
                        "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256",
                        "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA",
                        "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA",
                        "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA",
                        "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA",
                        "TLS_RSA_WITH_AES_128_GCM_SHA256",
                        "TLS_RSA_WITH_AES_256_GCM_SHA384",
                        "TLS_RSA_WITH_AES_128_CBC_SHA",
                        "TLS_RSA_WITH_AES_256_CBC_SHA"
                     ]
                  }
               ]
            },
            "status-check":{
               "listen":[
                  ":8080"
               ],
               "routes":[
                  {
                     "match":[
                        {
                           "path":[
                              "{env.STATUS_ROUTE}"
                           ]
                        }
                     ],
                     "handle":[
                        {
                           "body":"OK!",
                           "handler":"static_response",
                           "status_code":200
                        }
                     ]
                  },
                  {
                     "match":[
                        {
                           "path":[
                              "/ask"
                           ]
                        }
                     ],
                     "handle":[
                        {
                           "handler":"reverse_proxy",
                           "headers":{
                              "request":{
                                 "add":{
                                    "X-Cloud":[
                                       "{env.CLOUD}"
                                    ]
                                 }
                              }
                           },
                           "upstreams":[
                              {
                                 "dial":"{env.TLS_ASK}"
                              }
                           ]
                        }
                     ]
                  }
               ]
            }
         }
      }
   },
   "logging":{
      "logs":{
         "default":{
            "level":"{env.LOG_LEVEL}"
         }
      }
   },
   "storage":{
      "module":"file_system",
      "root":"/efs-certs"
   }
}
4

How are you running that docker container? Are you actually publishing ports 80/443/8080? That’s my hunch, if connections aren’t reaching Caddy. Sounds like a network config issue if you don’t see any requests reaching Caddy.

FYI, we have a builder image variant that you can use, to avoid needing to do the xcaddy setup yourself. See the docs on Docker Hub, specifically the section “Adding custom Caddy modules”

FWIW, I’m not seeing anything in your config that requires you to use JSON, you could use a Caddyfile now without trouble. Earlier versions didn’t have support for listener_wrappers in Caddyfile, but now, that’s configurable via global options.

5

We are running the stack:

Client => NLB => Caddy => ALB => Applications

for some while now. The only problem was that X-Forwarded-For only contained the private IP of the NLB instead of the Client’s IP. We need the client’s IP for deciding where to deliver some data from (EU or US).

The Problem/Goal is: We want to known the Client IP on the Application.
I now added the proxy_protocol in order to let the NLB add the Proxy Protocol Header (and also enabled it in the HTTPS Target Group in AWS). I haven’t changed anything else in the configuration. The ports are still published.

Aside:
and the reason why we use JSON for Configuration is because the configuration is generated based on environment variables by a python file when the docker container starts up. it then logs the config to stdout and starts caddy with that config. it’s way easier to parse a YAML file and produce JSON from it than to write a custom Caddyfile parser. and also JSON/YAML has way better tooling support (e.g. syntax highlighting)
Also the building works just fine. We are doing that for a while now.

1 Like
6

Environment variables are supported in the Caddyfile as well, FYI Caddyfile Concepts — Caddy Documentation

There’s a VSCode plugin for Caddyfile syntax highlighting Caddyfile Support - Visual Studio Marketplace

And that should work.

I’m confused, because you said:

Are you saying requests aren’t reaching Caddy at all now, after having upgraded to v2.4.5?

Maybe you can try adding the timeout option to proxy_protocol. It might be hanging because it can’t find types that look like they’re from PROXY protocol.

1 Like
7

Yah, it might be a nice thing, but our Config Management works and I haven’t found a way to disable parts based on Environment variables. We are using YAML, which is compiled to JSON and this very likely won’t change as all the configuration management is part of a bigger system.

I am now deploying a caddy v2.4.5 without PROXY protocol to see if the problem comes form the update or from PROXY protocol. Our CI/ Deployment takes around 40 minutes, after that I’ll see the results.

So for better understanding and making sure that I have done everything correctly:

  • the listener_wrappers needs to go inside of the server that should support proxy protocol. The correct order is proxy_protocol, tls?
  • tls in the listener_wrappers means, that the caddy itself speaks TLS with the client, not that the NLB infront adds TLS already
  • the allow statements should state the IP Addresses or CIDR Blocks of the NLB in front of caddy that adds the PROXY Header?
  • i need to enable PROXY protocol v2 in the AWS NLB (it does not support v1 according to the docs)?
  • after adding that configuration the actual client IP address will show up in the X-Forwarded-For header on the application side behind Caddy?
8

Sounds good :+1:

Environment variables in the Caddyfile can actually expand to multiple tokens of config, if your env vars can be multi-line values. Just FYI.

That’s right. There’s a clearer explanation about this on the Caddyfile global options page Global options (Caddyfile) — Caddy Documentation

Yeah it means that Caddy expects PROXY protocol bytes, followed by TLS handshake bytes (the proxy_procotol listener wrapper is expected to consume the bytes relevant to itself before handing it off to the next listener wrapper).

I believe so, yes. I’m not sure what happens if it doesn’t match those CIDRs though. It might just skip wrapping the connection. Reading the code in proxyprotocol/listener.go at ae65f2ce73c948fdd0d8f00ffaa6c519373c38bf · mastercactapus/proxyprotocol · GitHub that’s what it seems like it would do.

That’s probably correct. The plugin should accept both anyways. It just reads the first few bytes to determine which one to parse: proxyprotocol/parse.go at ae65f2ce73c948fdd0d8f00ffaa6c519373c38bf · mastercactapus/proxyprotocol · GitHub

It should be appended to X-Forwarded-For, yeah. In your access/debug logs in Caddy you should be able to see the remote_addr and the X-Forwarded-For header being passed upstream. The remote_addr should show up as the original client’s IP when using the PROXY protocol plugin.

1 Like
9

I finally have deoloyed a version with enabled DEBUG and getting these log lines on the caddy server (the IP address 10.20.201.x is the Subnet the NLB resides in)

{"level":"debug","ts":1634749313.4648879,"logger":"http.stdlib","msg":"http: TLS handshake error from 10.20.201.113:12064: invalid length"}

what happens if I remove the tls from the listener_wrappers might it be that the stuff is actually unpacked twice (so the second does not get the tls header?)

in around 10 minutes the other version with no PROXY protocol should be deployed

10

That’s strange. Seems like the Go stdlib TLS stack isn’t happy with the bytes it received.

AWS might be doing something funky with the bytes they send for proxy protocol. I don’t use AWS, and I didn’t write that plugin, so I’m not sure I’ll be able to help debug the specifics.

But from Target groups for your Network Load Balancers - Elastic Load Balancing it reads like they might put in some custom data that the plugin might not properly handle. I dunno.

Maybe you should open an issue on the plugin to get help from mastercactapus.

11

I’ve created an issue on the plugin: Caddy behind AWS Network Load Balancer with proxyprotocol enabled leads to tls errors · Issue #8 · mastercactapus/caddy2-proxyprotocol · GitHub

1 Like
12

I think your issue might be related to this note at the bottom: GitHub - pires/go-proxyproto: A Go library implementation of the PROXY protocol, versions 1 and 2.

That README is for a different golang proxy implementation, but it might offer a clue.

1 Like
13

I have exactly the same issue :), please lemme know if you find solution

14

By any chance, is Caddy v2 going to support proxy protocol as build it feature like other web servers? i believe these are critical feature and it would be awesome if Caddy v2 has this as built-in feature

15

If you mean receiving PROXY protocol, then yes, there’s a plugin for that. See the listener_wrappers config:

16

Yes but this does not work well with cloud providers like AWS.

17

This topic was automatically closed after 30 days. New replies are no longer allowed.