Cloudflare Infinite Redirect (ERR_TOO_MANY_REDIRECTS)

Help
1

1. Caddy version (caddy version):

V2.4.6

2. How I run Caddy:

I run caddy from the digital ocean marketplace

a. System environment:

Ubuntu 18.04

b. Command:

d. My complete Caddyfile or JSON config:

{
  "admin": {
    "listen": "0.0.0.0:2019"
  },
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":80"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "151.260.28.728"
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "upstreams": [
                            {
                              "dial": "www.example.com:80"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ]
        },
        "srv1": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "reverse_proxy",
                  "upstreams": [
                    {
                      "dial": "www.example.com:80"
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
    },
    "tls": {
      "automation": {
        "policies": [
          {
            "on_demand": true,
            "issuers": [{
                "module": "acme",
                "challenges": {
                    "dns": {
                        "provider": {
                            "name": "cloudflare",
                            "api_token": "MY_CLOUDFLARE_TOKEN"
                        },
                        "resolvers": [
                            "1.1.1.1"
                        ]
                    }
                }
            }]
          }
        ],
        "on_demand": {
          "ask": "https://example.com/caddy/allowed-domain"
        }
      }
    }
  }
}

3. The problem I’m having:

I’m trying to reverse proxy to a site on the Cloudflare network with proxy on. I’ve installed the Caddy Cloudflare Module (dns.providers.cloudflare) from the Caddy download page.

I created an API token on Cloudlfare for use (How to setup Caddy with Cloudflare) and I can verify it was entered correctly, I also confirmed that the Cloudflare Module (dns.providers.cloudflare) is installed correctly by running the caddy list-modules command.

The error persist irrespective of my Cloudflare SSL setting (Full or Full Strict).

NB. Caddy works fine if I disable the Cloudflare proxy.

Thank you for your time.

4. Error messages and/or full log output:

ERR_TOO_MANY_REDIRECTS

5. What I already tried:

Google search, reading the docs, switching the Cloudlfare SSL settings.

6. Links to relevant resources:

2

@matt @francislavoie, kindly assist me on this challenge, thank you

3

The “classic” issue with Cloudflare proxy is when the client connects to Cloudflare over TLS but Cloudflare connects to Caddy over HTTP (Flexible mode), causing Caddy redirecting Cloudflare to HTTPS, but the end client is already on HTTPS, so they try again… and receive a redirect to what looks like the same place, ad nauseum.

This doesn’t look like that at all. My first thought was some conflict with the upstream itself, but:

Indicates that this is not likely (or at least, if it is, it’s some niche interaction between the upstream proxy and Cloudflare’s proxy when Caddy is in the middle).

All of this said, I’m not sure where Caddy would be issuing a redirect at all. At a glance it looks like you have configured both HTTP on port 80 and HTTPS on port 443 to dial upstream www.example.com:80.

At the moment my recommendation is to enable debug logging, try again, and observe/report the log output so you/we can take a better look at the mechanics of a request in-flight. I’d suggest doing tests and getting logs for HTTP, HTTPS attempts without Cloudflare and again with Cloudflare.

The other thing to look at, just in case, might be Cloudflare Page Rules - which it is possible to produce a redirect loop when poorly configured.

1 Like
4

Thank you for taking the time to respond to this, your contribution to the Caddy server project is appreciated.

Indicates that this is not likely (or at least, if it is, it’s some niche interaction between the upstream proxy and Cloudflare’s proxy when Caddy is in the middle).

In this case, Cloudflare is in the middle. Kindly find the flow below:

CaddyServer (reverse proxied) => Cloudflare (proxied) => Endpoint

At the moment my recommendation is to enable debug logging, try again, and observe/report the log output so you/we can take a better look at the mechanics of a request in-flight. I’d suggest doing tests and getting logs for HTTP, HTTPS attempts without Cloudflare and again with Cloudflare.

Kindly find my logs below:

Cloudflare proxied:

{"level":"debug","ts":1636900936.4246683,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"app.gemstarenterprise.com"}
{"level":"debug","ts":1636900936.4247296,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.gemstarenterprise.com"}
{"level":"debug","ts":1636900936.4247358,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.com"}
{"level":"debug","ts":1636900936.4247408,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.*"}
{"level":"debug","ts":1636900936.42673,"logger":"tls.cache","msg":"added certificate to cache","subjects":["app.gemstarenterprise.com"],"expiration":1644567941,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"a8164556b8056899baa430e55562d221648c852d5d1abb888903032272281b50","cache_size":1,"cache_capacity":10000}
{"level":"debug","ts":1636900936.4267735,"logger":"tls.handshake","msg":"loaded certificate from storage","subjects":["app.gemstarenterprise.com"],"managed":true,"expiration":1644567941,"hash":"a8164556b8056899baa430e55562d221648c852d5d1abb888903032272281b50"}
{"level":"debug","ts":1636900951.349732,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"pointer.gemstarenterprise.com:80","duration":0.005979106,"request":{"remote_addr":"MY_CADDY_IP:43605","proto":"HTTP/2.0","method":"GET","host":"app.gemstarenterprise.com","uri":"/","headers":{"Sec-Fetch-Mode":["navigate"],"Sec-Ch-Ua-Platform":["\"Windows\""],"X-Forwarded-For":["MY_CADDY_IP"],"Sec-Fetch-Dest":["document"],"X-Forwarded-Proto":["https"],"Cache-Control":["max-age=0"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-User":["?1"],"Accept-Language":["en-US,en;q=0.9"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Ch-Ua-Mobile":["?0"],"Cookie":["_ga_KL6T5BHMQD=GS1.1.1636797644.1.1.1636797656.0; _ga=GA1.2.1792007519.1636797645"],"Dnt":["1"],"Sec-Fetch-Site":["none"],"Sec-Ch-Ua":["\"Google Chrome\";v=\"95\", \"Chromium\";v=\"95\", \";Not A Brand\";v=\"99\""]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"app.gemstarenterprise.com"}},"headers":{"Nel":["{\"success_fraction\":0,\"report_to\":\"cf-nel\",\"max_age\":604800}"],"Vary":["Accept-Encoding"],"Cf-Ray":["6ae101c1eaa932f4-EWR"],"Alt-Svc":["h3=\":443\"; ma=86400, h3-29=\":443\"; ma=86400, h3-28=\":443\"; ma=86400, h3-27=\":443\"; ma=86400"],"Date":["Sun, 14 Nov 2021 14:42:31 GMT"],"Connection":["keep-alive"],"Cache-Control":["max-age=3600"],"Report-To":["{\"endpoints\":[{\"url\":\"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=oHfEBqQUHvq7HGBTdSLagefy7KMHbA22CxnPx7yWpha87wXj%2B6F%2Bi3dCKSSrnJom01cBYiOewS5KojJi2pkyOmqLYtp9WJOCkDmaFyo0NNl89oxgd18gn%2F%2FCtFbn%2BB2nkXkBxdViOmIPIIIhMub6etg%3D\"}],\"group\":\"cf-nel\",\"max_age\":604800}"],"Expires":["Sun, 14 Nov 2021 15:42:31 GMT"],"Location":["https://app.gemstarenterprise.com/"],"Server":["cloudflare"]},"status":301}

Cloudflare proxy off

{"level":"debug","ts":1636905135.6610572,"logger":"tls.handshake","msg":"choosing certificate","identifier":"app.gemstarenterprise.com","num_choices":1}
{"level":"debug","ts":1636905135.6620243,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"app.gemstarenterprise.com","subjects":["app.gemstarenterprise.com"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"a8164556b8056899baa430e55562d221648c852d5d1abb888903032272281b50"}
{"level":"debug","ts":1636905135.6620922,"logger":"tls.handshake","msg":"matched certificate in cache","subjects":["app.gemstarenterprise.com"],"managed":true,"expiration":1644567941,"hash":"a8164556b8056899baa430e55562d221648c852d5d1abb888903032272281b50"}
{"level":"debug","ts":1636905135.8892372,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"pointer.gemstarenterprise.com:80","duration":0.021737347,"request":{"remote_addr":"129.205.113.22:43611","proto":"HTTP/2.0","method":"GET","host":"app.gemstarenterprise.com","uri":"/","headers":{"Dnt":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Accept-Encoding":["gzip, deflate, br"],"X-Forwarded-For":["129.205.113.22"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"Windows\""],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36"],"Cookie":["_ga_KL6T5BHMQD=GS1.1.1636797644.1.1.1636797656.0; _ga=GA1.2.1792007519.1636797645"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Mode":["navigate"],"Sec-Ch-Ua":["\"Google Chrome\";v=\"95\", \"Chromium\";v=\"95\", \";Not A Brand\";v=\"99\""],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Accept-Language":["en-US,en;q=0.9"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"app.gemstarenterprise.com"}},"headers":{"Content-Type":["text/html; charset=UTF-8"],"Referrer-Policy":["no-referrer"],"Content-Length":["1561"],"Date":["Sun, 14 Nov 2021 15:52:15 GMT"]},"status":404}
{"level":"debug","ts":1636905136.4692311,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"pointer.gemstarenterprise.com:80","duration":0.001817528,"request":{"remote_addr":"129.205.113.22:43611","proto":"HTTP/2.0","method":"GET","host":"app.gemstarenterprise.com","uri":"/favicon.ico","headers":{"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-Site":["same-origin"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Ch-Ua":["\"Google Chrome\";v=\"95\", \"Chromium\";v=\"95\", \";Not A Brand\";v=\"99\""],"X-Forwarded-Proto":["https"],"X-Forwarded-For":["129.205.113.22"],"Dnt":["1"],"Sec-Fetch-Mode":["no-cors"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Dest":["image"],"Cookie":["_ga_KL6T5BHMQD=GS1.1.1636797644.1.1.1636797656.0; _ga=GA1.2.1792007519.1636797645"],"Sec-Ch-Ua-Mobile":["?0"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36"],"Accept":["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"app.gemstarenterprise.com"}},"headers":{"Content-Type":["text/html; charset=UTF-8"],"Referrer-Policy":["no-referrer"],"Content-Length":["1572"],"Date":["Sun, 14 Nov 2021 15:52:16 GMT"]},"status":404}

The other thing to look at, just in case, might be Cloudflare Page Rules - which it is possible to produce a redirect loop when poorly configured.

I currently do not have a page rule for this domain on Cloudflare

1 Like
5

Oh, whoops! I just assumed it was Cloudflare → Caddy → App. Bad assumption there.

Looks like we’ve got the exact same problem as the “classic” Cloudflare issue, but in reverse.

    "Location": [
      "https://app.gemstarenterprise.com/"
    ],
    "Server": [
      "cloudflare"
    ]

Cloudflare wants its client (Caddy) to connect via HTTPS, but you have configured your upstream to connect to port 80 (HTTP). Cloudflare sees an insecure client and tries to upgrade, Caddy faithfully passes the Location response back to the client, who follows the redirect to try again with Caddy which proxies again to HTTP regardless of the original scheme… prompting a Cloudflare upgrade redirect, it seems.

Consider proxying to https://www.example.com instead of www.example.com:80 when Cloudflare is in the middle.

2 Likes
6

Of note, if you’re using JSON config then you need to use app.gemstarenterprise.com:443 as the upstream address, then configure the transport with tls: {} at least.

Is there a particular reason you’re using JSON instead of Caddyfile here? I’m not seeing anything in your config that would require JSON. It would be simpler/easier to write the config with a Caddyfile.

{
	admin :2019
	on_demand_tls {
		ask https://example.com/caddy/allowed-domain
	}
}

https:// {
	tls {
		dns cloudflare {$CLOUDFLARE_TOKEN}
		resolvers 1.1.1.1
	}

	reverse_proxy https://example.com {
		header_up Host {upstream_hostport}
		header_up X-Forwarded-Host {host}
	}
}

Also having written this out, why are you using the DNS challenge at the same time as on-demand TLS?

The main reason for using the DNS challenge is either because you need wildcard certificates, or because your server isn’t directly reachable by Let’s Encrypt or ZeroSSL over ports 80 and/or 443, and you control the DNS for the domains.

The main reason for using On-Demand TLS is when you don’t know the domains you need to proxy ahead of time, and you may not control the DNS (usually for domains held by your customer, and they CNAME it to your domain).

So those two features are essentially mutually-exclusive.

2 Likes
7

It works :tada::tada:

Thank you for pointing this out and for taking the time to resolve this challenge, it fixed the max_redirect error instantly :+1:

Thank you for providing a working Caddyfile it fixed the 502 errors I was getting. :clap:

Yes, I’m currently using Caddy via the API.

The choice of API or config file workflow is orthogonal to the use of config adapters: you can use JSON but store it in a file and use the command line interface; conversely, you can also use the Caddyfile with the API.

But most people will use JSON+API or Caddyfile+CLI combinations.
https://caddyserver.com/docs/getting-started

Thank you @matt, @Whitestrake, @francislavoie for your dedication to this project.

3 Likes
8

This topic was automatically closed after 30 days. New replies are no longer allowed.