Https backend suddenly not reachable, getting 502 although site is up

Help
1

1. Caddy version (caddy version):

Latest 2.4 docker container

2. Caddy run via

Docker

a. System environment:

Docker running on Ubuntu 20 LTS

b. Command:

docker-compose up -d

c. Service/unit/compose file:

#docker-compose.yml

version: "3.7"

services:
  caddy:
    image: caddy:latest
    container_name: caddy_devcon_cc
    restart: always
#    command: --log stdout
    volumes:
      - /root/docker/caddyv2/Caddyfile:/etc/caddy/Caddyfile:ro
      - /root/docker/caddyv2/data:/data
      - /root/docker/caddyv2/config:/config
    ports:
      - 192.168.2.153:80:80
      - 192.168.2.153:443:443
    environment:
      ACME_AGREE: "true" # agree to Let's Encrypt Subscriber Agreement
#      DOMAIN: "vault.devcon.cc" # CHANGE THIS! Used for Auto Let's Encrypt SSL
#      EMAIL: "thomas.kofler@devcon.cc"  # CHANGE THIS! Optional, provided to Let's Encrypt
#volumes:
#  caddycerts:

d. My complete Caddyfile or JSON config:

www.affectedomain.com {
        tls info@affectedomain.com {
    }
    header {
        # Enable HTTP Strict Transport Security (HSTS)
        Strict-Transport-Security "max-age=31536000;"
        # Enable cross-site filter (XSS) and tell browser to block detected attacks
        X-XSS-Protection "1; mode=block"
        # Disallow the site to be rendered within a frame (clickjacking protection)
        X-Frame-Options "DENY"
    }
    reverse_proxy https://192.168.2.107 {
                       transport http {
                        tls_insecure_skip_verify
                }
    }
}

3. The problem I’m having:

Websites hosted on an older webserver serviced via https via Caddy are no longer reachable. Browser shows an 502 error generated by Caddy.
Before they worked fine, we are so far not aware of any change on the webserver.
That webserver uses private expired certificates.

4. Error messages and/or full log output:

method":"GET","host":"www.affectedomain.com","uri":"/","headers":{"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Accept-Language":["en-US,en;q=0.9,de;q=0.8"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Microsoft Edge\";v=\"100\""],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36 Edg/100.0.1185.50"],"Sec-Fetch-User":["?1"],"Pragma":["no-cache"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-Site":["none"],"Cache-Control":["no-cache"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Dest":["document"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"www.affectedomain.com"}},"duration":0.0011455,"status":502,"err_id":"y5w4afeqn","err_trace":"reverseproxy.statusError (reverseproxy.go:1166)"}

5. What I already tried:

Serving the backend website via http works fine
Different websites are affected, independent of the config
The Backend is up and running, its reachable from the docker host via lynx https:///same-ip-as-used-with-caddy-reverse-proxy-directive

6. Links to relevant resources:

2

The latest is v2.5.0. Please upgrade.

You don’t need this line. This was only needed in Caddy v1. In v2, simply using Caddy is implicit agreement to the LE terms.

That’s not the full log message. It’s cut off at the top. Please share full logs.

Why not just use HTTP then? If the upstream is within a private network, then you don’t need the overhead of proxying over HTTPS, you don’t get any safety advantages unless you have untrusted software running in the network.

3

Thanks for the reply. The backend website has to run with HTTPS else its malfunctioning with http proxying.

Full log:

{"level":"error","ts":1651177791.8492796,"logger":"http.log.error","msg":"tls: server selected unsupported protocol version 301","request":{"remote_ip":"213.47.187.187","remote_port":"59842","proto":"HTTP/2.0","method":"GET","host":"www.affectedomain.com","uri":"/","headers":{"Sec-Ch-Ua-Platform":["\"Windows\""],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Ch-Ua-Mobile":["?0"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36 Edg/100.0.1185.50"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Dest":["document"],"Accept-Language":["en-US,en;q=0.9,de;q=0.8"],"Cookie":[],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Microsoft Edge\";v=\"100\""]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"www.affectedomain.com"}},"duration":0.0033493,"status":502,"err_id":"63z13h48h","err_trace":"reverseproxy.statusError (reverseproxy.go:1166)"}
{"level":"error","ts":1651177795.9125957,"logger":"http.log.error","msg":"tls: server selected unsupported protocol version 301","request":{"remote_ip":"213.47.187.187","remote_port":"59842","proto":"HTTP/2.0","method":"GET","host":"www.affectedomain.com","uri":"/","headers":{"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-Dest":["document"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-User":["?1"],"Accept-Language":["en-US,en;q=0.9,de;q=0.8"],"Cache-Control":["no-cache"],"Accept-Encoding":["gzip, deflate, br"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36 Edg/100.0.1185.50"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Cookie":[],"Pragma":["no-cache"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Microsoft Edge\";v=\"100\""]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"www.affectedomain.com"}},"duration":0.0038252,"status":502,"err_id":"i8xfw4ca9","err_trace":"reverseproxy.statusError (reverseproxy.go:1166)"}
4

Ah, there you go. Your upstream is using too old a version of TLS. Caddy only supports TLS 1.2 and 1.3, anything older is insecure. Version 301 is actually TLS 1.1.

3 Likes
5

Strange, it worked fine for months with that upstream server.
Can it be that Caddy 2.5 does not support upstream TLS 1.1?

6

Caddy v2 never has.

7

Thanks, then there must be a breaking change for us in Caddy 2.5

Rechecked everything and somehow the image was updated to caddy:2.5.0-alpine
If we go back to caddy:2.4.6-alpine everything works fine again.

8

Beginning in the next release, Go 1.18, the Config.MinVersion for crypto/tls clients will default to TLS 1.2, disabling TLS 1.0 and TLS 1.1 by default. Applications will be able to override the change by explicitly setting Config.MinVersion . This will not affect crypto/tls servers.

Our official builds are using Go 1.18 (we always use the latest Go version, especially for security releases). I recommend upgrading your backend to TLS 1.3 rather than using weakened security.

2 Likes
9

Even as I agree with @matt and @francislavoie to update the backend servers maybe you can take a look into the JSON Config Structure - Caddy Documentation until the backend server is updated to the latest TLS Version.

I have created a JSON example as I don’t know if the Caddyfile supports the tls_connection_policies config.
You will need to adopt the sni match for your needs.

tls_policy.json

{
  "admin": {
    "listen": "127.0.0.1:8081"
  },
  "apps": {
    "http": {
      "http_port": 8082,
      "https_port": 8443,
      "servers": {
        "srv0": {
          "listen": [
            ":8080"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "headers",
                  "response": {
                    "set": {
                      "Strict-Transport-Security": [
                        "max-age=31536000;"
                      ],
                      "X-Frame-Options": [
                        "DENY"
                      ],
                      "X-Xss-Protection": [
                        "1; mode=block"
                      ]
                    }
                  }
                },
                {
                  "handler": "reverse_proxy",
                  "transport": {
                    "protocol": "http",
                    "tls": {
                      "insecure_skip_verify": true
                    }
                  },
                  "upstreams": [
                    {
                      "dial": "192.168.2.107:443"
                    }
                  ]
                }
              ]
            }
          ],
          "tls_connection_policies": [
            {
              "match": {
                "sni": [
                  "192.168.2.107"
                ]
              },
              "protocol_min": "tls1.1"
            }
          ]
        }
      }
    },
    "tls": {
      "automation": {
        "policies": [
          {
            "issuers": [
              {
                "email": "info@example.com",
                "module": "acme"
              },
              {
                "email": "info@example.com",
                "module": "zerossl"
              }
            ]
          }
        ]
      }
    }
  },
  "logging": {
    "logs": {
      "default": {
        "level": "DEBUG"
      }
    }
  }
}

You can upload the file via the JSON api API Tutorial — Caddy Documentation or use the -adapter jsonc

./caddy run -adapter jsonc -config tls_policy.json

Please also take a look into this doc what’s the difference between the JSON and the Caddyfile.

1 Like
10

Thanks Aleks. Although, TLS Connection policies are for server configuration, it doesn’t configure TLS for the reverse proxy as a client going to the upstream.

2 Likes
11

thanks @matt for clarification.

12

Thanks a lot everyone for feedback and support.
As the backend is internally in the same lan segment as the backend, TLS security wouldn’t be an issue.

But as GO did the decision, seems there is no workaround for us.

Would be nice if it could be mentioned in the release notes to make others aware as I was unable to find it explicitly there.

13

Even if Go did allow for it, we (Caddy) probably wouldn’t have.

TLS 1.2 is 14 years old now. There’s no excuse for not upgrading to support it at this point. Older versions of TLS are broken.

14

Sorry, that’s not the point - disabling without even documenting it is a absolute no go and an official announcement with official depreciation timeplan would be even better.

Else Caddy is not it enterprise ready as TLS 1.2 is still actively used in the Enterprise space - even it’s not the “current” version.

1 Like
15

TLS 1.2 is absolutely still supported.

You can always build Caddy with an older version of Go if you want less security. But we’re not in the business of making enterprises less secure.

1 Like
16

Not in the business supporting enterprises at least with documenting the change based in the default docker images.

I brought the topic to your attention, my task is done. If that breaking changes are not clearly documented it’s your decision.
For that specific use case caddy is then not the right tool for us, which is fine.

But overall your great work is really appreciated with Caddy. Thanks.

17

To clarify, it’s TLS 1.0 and 1.1 that are not supported. TLS 1.2 is definitely supported.

My point was that TLS 1.2 is already 14 years old at this point, so there’s really no excuse for not supporting at least TLS 1.2.

Obviously TLS 1.3 is even better, but Caddy has a minimum of TLS 1.2.

Well, “enterprise ready” is a buzzword we don’t subscribe to.

But Caddy is definitely in use by huge companies, which as Matt said a couple days ago:

High loads, and high stakes (hundreds of thousands of dollars worth of transactions per hour, is one metric I do know).

“Enterprise” does not mean “we’re still using legacy software and can’t spend the resources updating them”. I don’t mean to be rude, but I take issue with that positioning.

Either way, we weren’t aware there would be a “breaking change” because we assume users are using modern software stacks. This doesn’t feel like something notable that we should document. I’d consider relying on old TLS versions a bug, and we implicitly got a fix for it with this latest release.

1 Like
18

To reiterate, TLS 1.2 is still supported in Caddy and Go.

My understand is that’s only true if you need TLS 1.0 and 1.1 (which are broken). And if that’s the case, why not just build with Go 1.17? You don’t have to change anything about Caddy or its configuration. Caddy is ambivalent to what version of Go it is built with as long as it’s recent enough. And Go 1.17 is still supported.

1 Like
19

Thanks for the clarification, I will give it a try.

20

My point was that TLS 1.2 is already 14 years old at this point, so there’s really no excuse for not supporting at least TLS 1.2.
Obviously TLS 1.3 is even better, but Caddy has a minimum of TLS 1.2.

Thanks for clarification and I absolutely agree on that points.

But Caddy is definitely in use by huge companies, which as [Matt said a couple days ago]

Had never a doubt about it regarding functionality and innovation.

To clarify about “enterprise ready”: For me its not about supporting legacy/old/buggy TLS versions.
Its about trackable changes (new features, bugfixes, removed features/functions) for new versions, where until now Caddy did a great job.

So either its an bugfix (your wording) or a removal of a feature (my wording), it should be documented.
Exactly as Go documented in the release notes the disabling of TLS 1.0 and TLS 1.1 to make users aware of it and react on it, if necessary.