Random 502 errors when proxying to application with self-signed certificate

1. The problem I’m having:

I’m trying to run Kasm behind Caddy. By default, this application uses a self-signed certificate. I’m planning to let caddy handle the certificate and just proxy to Kasm running on the same machine (on port 8443).

According to their documentation there shouldn’t be much configuration required (annotation is mine):

 {
      local_certs # skipping this because i want caddy to create a cert
      default_sni kasm.example.local
   }
   https://kasm.example.local:443 {

           reverse_proxy https://127.0.0.1:8443 {
                   transport http {
                           tls_insecure_skip_verify
                   }

               header_up Host {host}
               header_up X-Real-IP {remote}
               header_up X-Forwarded-For {remote}
               header_up X-Forwarded-Port {server_port}
               header_up X-Forwarded-Proto {scheme}

           }
 }

When i load the application (you can try yourself), some requests randomly fail. There is no pattern to it but for each page loads there are a few.

The application is running fine without caddy and still works as expected when using SSH port forwarding to directly access it on port 8443.

image

Here is one of the failed requests:

# General
Request URL: https://kasm.pingbit.de/
Request Method: GET
Status Code: 502 
Remote Address: 130.61.217.97:443
Referrer Policy: strict-origin-when-cross-origin

# Response headers
alt-svc: h3=":443"; ma=2592000
content-length: 0
date: Sun, 16 Apr 2023 20:37:21 GMT
server: Caddy

# Request HEaders
:authority: kasm.pingbit.de
:method: GET
:path: /
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
accept-encoding: gzip, deflate, br
accept-language: de-DE,de;q=0.9
cache-control: no-cache
dnt: 1
pragma: no-cache
sec-ch-ua: "Chromium";v="112", "Google Chrome";v="112", "Not:A-Brand";v="99"
sec-ch-ua-mobile: ?1
sec-ch-ua-platform: "Android"
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: same-origin
sec-fetch-user: ?1
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Mobile Safari/537.36

While i found some similar threads abut seemingly related issues, most of them end up using tls_insecure_skip_verify and carry on with their day.

I hope i’m not barking up the wrong tree. To me it seems like the request is not even getting trough to Kasm, that’s why i’m asking for help here :slight_smile: Any help/pointer is appreciated.

2. Error messages and/or full log output:

Apr 16 20:36:30 bender caddy[553075]: {"level":"error","ts":1681677390.34172,"logger":"http.log.error.log0","msg":"tls: first record does not look like a TLS handshake","request"

More logs here: 0.g.gg .

3. Caddy version:

2.6.4

4. How I installed and ran Caddy:

a. System environment:

Linux bender 5.15.106 #1-NixOS SMP Wed Apr 5 09:25:02 UTC 2023 aarch64 GNU/Linux

b. Command:

I don’t think it is relevant but anyways, i just enabled Caddy as a service in NixOS.

  services.caddy.enable = true;
  services.caddy.email = "<snip>;
  services.caddy.globalConfig = ''
    debug
    default_sni kasm.pingbit.de
  '';

c. Service/unit/compose file:

cat /etc/systemd/system/caddy.service
# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/nix/store/y3f0ambxjqrr95jasdpi3b6dw7vnxxsd-caddy-2.6.4/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/nix/store/y3f0ambxjqrr95jasdpi3b6dw7vnxxsd-caddy-2.6.4/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateDevices=yes
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

{
        debug
        default_sni kasm.pingbit.de

        email <snip>
        acme_ca https://acme-v02.api.letsencrypt.org/directory
        log {
                level ERROR
        }
}
https://kasm.pingbit.de:443 {
        bind

        log {
                output file /var/log/caddy/access-https://kasm.pingbit.de:443.log
        }

        reverse_proxy : https://127.0.0.1:8443 {
                transport http {
                        tls_insecure_skip_verify
                }
                header_up Host {host}
                header_up X-Real-IP {remote}
                header_up X-Forwarded-For {remote}
                header_up X-Forwarded-Port {server_port}
                header_up X-Forwarded-Proto {scheme}
        }
}

5. Links to relevant resources:

https://kasm.pingbit.de/

I had to truncate the logs to the (what i think) crucial information because if i didn’t, this happened:


(of course i run into more errors when trying to ask for help :rofl: )

If that’s your actual Caddyfile and not some copy past error or whatever, remove the stray : in

reverse_proxy : https://127.0.0.1:8443 {

so the line looks like

reverse_proxy https://127.0.0.1:8443 {

You are essentially load balancing between https://127.0.0.1:8443 and https://:80

You can see this yourselves when running

$ caddy adapt | jq .apps.http.servers.srv0.routes[0].handle[0].routes[0].handle[0].upstreams
[
  {
    "dial": ":80"
  },
  {
    "dial": "127.0.0.1:8443"
  }
]

Edit: or curl localhost:2019/config/ on the server

1 Like

Yep, I agree with @emilylange, the problem is the : upstream address which makes half of the requests fail.

Remove all this, none of it is useful.

See reverse_proxy (Caddyfile directive) — Caddy Documentation which explains.

You probably don’t need this. It’s only needed if some clients don’t support SNI, but all browsers do at this point.

I strongly recommend not doing this. This hides important INFO and WARN level message from your logs, e.g. deprecation warnings and such.

This doesn’t do anything because you didn’t pass it an argument. Remove that.

I opened an issue for Kasm to update their docs, since they’re misleading:

Wow, what a silly mistake :see_no_evil: Thank you for looking into it and giving a detailed explaination, too!

I will follow your advice and clean up my config.
The superflous bind statement was generated by the Nix expression used to enable Caddy on NixOS. I will poke around a little why it is there but it’s good to know that it basically is a NOOP

You are essentially load balancing between https://127.0.0.1:8443 and https://:80

This bit caught my interest. How does the additional colon translate to upstream https://:80 ?

I’m surprised it passed validation too. I’ll make sure that becomes an error because it doesn’t make sense.

But essentially, if there’s no port, it defaults to the port of the scheme. If there’s no scheme, it defaults to http:// implicitly, so we add port 80 to be a valid address. But since there was no scheme in that one and there was an explicit scheme of https:// on the other one, it enabled tls for the transport, which means effectively that all upstreams are HTTPS.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.