Caddy with reverse proxy returns unexpected 403

Help
1

1. Caddy version (caddy version):

v2.3.0 h1:fnrqJLa3G5vfxcxmOH/+kJOcunPLhSBnjgIvjXV/QTA=

2. How I run Caddy:

N/A

a. System environment:

AWS, Docker

b. Command:

paste command here

c. Service/unit/compose file:

paste full file contents here

d. My complete Caddyfile or JSON config:

paste config here, replacing this text
use `caddy fmt` to make it readable
DO NOT REDACT anything except credentials
or helpers will be sad

3. The problem I’m having:

I have a rather complicated (and large) set up of many services and we are replacing Nginx with caddy as a key component. I almost have everything working - but our test suite fails a bunch of tests.

The failure is due to a request for a javascript asset. If you curl the resource it works fine. However, we found that in our test suite this is returning a 403.

Here are the two requests:

This one is just loading the js code by entering the url in chrome. It works fine:

{
  "level": "info",
  "ts": 1613176370.9133158,
  "logger": "http.log.access",
  "msg": "handled request",
  "request": {
    "remote_addr": "52.42.200.43:35674",
    "proto": "HTTP/2.0",
    "method": "GET",
    "host": "accounts-customer.rayj2.dev.tilia-inc.com",
    "uri": "/ui/v1/widget",
    "headers": {
      "Accept": [
        "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
      ],
      "Sec-Fetch-Site": [
        "none"
      ],
      "Sec-Fetch-User": [
        "?1"
      ],
      "Sec-Fetch-Dest": [
        "document"
      ],
      "Cache-Control": [
        "max-age=0"
      ],
      "Sec-Ch-Ua": [
        "\"Chromium\";v=\"88\", \"Google Chrome\";v=\"88\", \";Not A Brand\";v=\"99\""
      ],
      "Sec-Ch-Ua-Mobile": [
        "?0"
      ],
      "User-Agent": [
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36"
      ],
      "Cookie": [
        "_ga=GA1.2.385234868.1564785036"
      ],
      "Upgrade-Insecure-Requests": [
        "1"
      ],
      "Sec-Fetch-Mode": [
        "navigate"
      ],
      "Accept-Encoding": [
        "gzip, deflate, br"
      ],
      "Accept-Language": [
        "en-US,en;q=0.9"
      ]
    },
    "tls": {
      "resumed": true,
      "version": 772,
      "cipher_suite": 4865,
      "proto": "h2",
      "proto_mutual": true,
      "server_name": "accounts-customer.rayj2.dev.tilia-inc.com"
    }
  },
  "common_log": "52.42.200.43 - - [13/Feb/2021:00:32:50 +0000] \"GET /ui/v1/widget HTTP/2.0\" 200 2786",
  "duration": 0.022670303,
  "size": 2786,
  "status": 200,
  "resp_headers": {
    "Content-Type": [
      "application/javascript; charset=utf-8"
    ],
    "X-Powered-By": [
      "Express"
    ],
    "Date": [
      "Sat, 13 Feb 2021 00:32:50 GMT"
    ],
    "Etag": [
      "W/\"238a-up84g1q1MAtG33vOoWlOHqoYGvk\""
    ],
    "Cache-Control": [
      "private, no-cache, no-store, must-revalidate"
    ],
    "Expires": [
      "-1"
    ],
    "Content-Encoding": [
      "gzip"
    ],
    "Pragma": [
      "no-cache"
    ],
    "Vary": [
      "Accept-Encoding"
    ],
    "Server": [
      "Caddy"
    ]
  }
}

This one is generated by a selenium test using headless chrome. It fails!

{
  "level": "error",
  "ts": 1613175786.8073003,
  "logger": "http.log.access",
  "msg": "handled request",
  "request": {
    "remote_addr": "172.17.42.1:49848",
    "proto": "HTTP/2.0",
    "method": "GET",
    "host": "accounts-customer.rayj2.dev.tilia-inc.com",
    "uri": "/ui/v1/widget",
    "headers": {
      "Sec-Fetch-Mode": [
        "no-cors"
      ],
      "Sec-Fetch-Dest": [
        "script"
      ],
      "Referer": [
        "https://fake-integrator.rayj2.dev.tilia-inc.com/ui/tos/iframe?account_id=58a15c06-51f0-48a3-b3b9-75190bcb44e2"
      ],
      "Accept-Encoding": [
        "gzip, deflate, br"
      ],
      "Accept-Language": [
        "en-US"
      ],
      "User-Agent": [
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4324.96 Safari/537.36"
      ],
      "Accept": [
        "*/*"
      ],
      "Sec-Fetch-Site": [
        "same-site"
      ]
    },
    "tls": {
      "resumed": false,
      "version": 772,
      "cipher_suite": 4865,
      "proto": "h2",
      "proto_mutual": true,
      "server_name": "fake-integrator.rayj2.dev.tilia-inc.com"
    }
  },
  "common_log": "172.17.42.1 - - [13/Feb/2021:00:23:06 +0000] \"GET /ui/v1/widget HTTP/2.0\" 403 0",
  "duration": 2.457e-05,
  "size": 0,
  "status": 403,
  "resp_headers": {
    "Server": [
      "Caddy"
    ]
  }
}

The interesting thing is when things work the request makes it to my service. But when I’m getting this 403 - the request is never getting to my service!!! The 403 is returned by Caddy (and I assume the reverse proxy module).

Here is the relevant part of my caddy config:

          - match:
              - host:
                  - accounts-customer.rayj2.dev.tilia-inc.com
            handle:
              - handler: subroute
                routes:
                  - handle:
                      - handler: reverse_proxy
                        headers:
                          request:
                            set:
                              Host:
                                - '{http.request.host}'
                              X-Forwarded-For:
                                - '{http.request.remote}'
                              X-Real-Ip:
                                - '{http.request.remote}'
                        upstreams:
                          - dial: accounts-customer-web:3000
                    match:
                      - path:
                          - /ui/*
                          - /static/*
                    terminal: true
                  - handle:
                      - handler: reverse_proxy
                        headers:
                          request:
                            set:
                              Host:
                                - '{http.request.host}'
                              X-Forwarded-For:
                                - '{http.request.remote}'
                              X-Real-Ip:
                                - '{http.request.remote}'
                        upstreams:
                          - dial: accounts-customer:80
                    terminal: true
            terminal: true

So I certainly have nothing that is telling Caddy to return a 403. The only real difference between the two requests is the headers. (As far as I can tell anyway.)

Under what circumstance is caddy making a decision to return a 403 instead of passing the request along to the downstream service? How do I get it to stop doing that?

4. Error messages and/or full log output:

5. What I already tried:

6. Links to relevant resources:

2

It’s very unlikely that you need this stuff. Caddy sets Host and X-Forwarded-For automatically and correctly. I strongly recommend removing these.

The only situations where Caddy (at least with standard modules) returns a 403 (Forbidden) are:

  • Admin API, if origin checks are enabled
  • During TLS handshakes, if strict_sni_host is enabled
  • The file_server when the filesystem gives a permission error for a directory or file

So if none of those make sense here, then it must be from your backend. You can enable debug logging by setting the default named logger’s level to DEBUG. It should reveal a bit more info about what Caddy’s doing.

3

I have debug logging on and it does not log ANYTHING related to the request. In fact, for the longest time I thought the request must not be getting to caddy. Finally, I turned on the access logging and I ONLY see the request there. The request never reaches my server.

But it does work when I do a normal curl or make the request direct from Chrome.

I think one of those headers is causing Caddy or the reverse proxy module to return the 403. I’ll keep working to narrow the problem down more. Was kind of hoping I was missing something - but if this is unexpected to you as well - then I need to find a way to make it more reproducible…

4

So I you mentioned the places where Caddy could use a 403. I’m not using file_server and this is not against the admin_server. But you got me thinking about the pls handshake.

Then I noticed above in my two example requests above:
In the tls.server_name section the request that works has the same server_name as the host (which we would expect). However, in the one that fails the server_name is “fake-integrator” which is actually the referrer.

How can that happen?

I should mention I’m using a wildcard cert for all the sub-domains.

Why would that server_name be different if I hit the URL manually vs. when it is hit in our test suite?

Is the browser reusing an open connection perhaps?

5

BTW, I do not have strict_sni_host enabled.

But I’m been reading about some of the problems that can happen with http/2 and request coalescing. I’m betting the connection is getting reused. The TLS handshake is avoided - but the sni is now plain wrong. This can happen for various reasons - but a serve/proxyr should return error code 421 in such cases not - 403.

This makes this a little harder to reproduce this problem…

6

I came across this thread related to a differentt proxy:
https://github.com/dlundquist/sniproxy/issues/178

It suggested a way to disable http2.0 in headless chrome. I did that and everything works. So I am certain it is a bug in caddy with respect to support for http/2.

Specifically, I think that rather than returning 403 if it finds a resource miss-match it should be returning a 421 as per the http/2 RFC:

In such a case, Chrome would then open a new connection and everything would work. But with a 403 it thinks it is some other problem.

7

This topic was automatically closed after 30 days. New replies are no longer allowed.