HTTP 502 / SSL error 14094438

1. Output of caddy version:

v2.5.2 h1:eCJdLyEyAGzuQTa5Mh3gETnYWDClo1LjtQm2q9RNZrs=

2. How I run Caddy:

a. System environment:

Ubuntu 22.04 LTS → Nomad → Docker container

b. Command: caddy run --config /etc/caddy/Caddyfile --adapter caddyfile

c. Service/unit/compose file:

job "ingress" {
  datacenters = ["..."]

  group "g" {
    network {
      mode = "bridge"

      port "http" {
        static = 80
        to     = 80
      }
      port "https" {
        static = 443
        to     = 443
      }
    }

    service {
      name = "ingress"

      connect {
        sidecar_service {
          proxy {
            upstreams {
              destination_name = "aaa"
              local_bind_port  = 1111
            }
            upstreams {
              destination_name = "bbb"
              local_bind_port  = 2222
            }
          }
        }
      }
    }

    task "caddy" {
      driver = "docker"

      config {
        image   = "pikeas/caddy:2.5.2"
        volumes = ["local:/etc/caddy:ro"]
      }

      template {
        data        = local.caddy_config
        destination = "local/Caddyfile"
      }

      volume_mount {
        volume      = "ingress"
        destination = "/data"
      }
    }

    volume "ingress" {
      type   = "host"
      source = "ingress"
    }
  }
}

d. My complete Caddy config:

# inlined to job file as local template
{
  acme_dns cloudflare ...
}

aaa.int.example.com {
    reverse_proxy localhost:1111
}
bbb.int.example.com {
    reverse_proxy localhost:2222
}

3. The problem I’m having:

Caddy is proxying internal services. This is in a Nomad cluster, where Nomad makes those services available in the Caddy container’s namespace.

Those services work when hit directly, eg: curl http://localhost:1111.

Caddy thinks everything is fine:

{"level":"info","ts":...,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme-v02.api.letsencrypt.org/acme/order/.../..."}
{"level":"info","ts":...,"logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":2,"first_url":"https://acme-v02.api.letsencrypt.org/acme/cert/..."}
{"level":"info","ts":...,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"aaa.int.example.com"}

Everything is not fine:

$ curl https://localhost/ -H "Host: aaa.internal.example.com" -v
*   Trying 127.0.0.1:443...
* Connected to localhost (127.0.0.1) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, internal error (592):
* error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
* Closing connection 0
curl: (35) error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error

I have two clues:

  1. Curl with --resolve aaa.int.example.com:443:127.0.0.1 works, host header does not.
  2. Only some services are failing, despite identical Caddy config blocks.

After enabling debug, there’s one additional line logged:

{"level":"debug","ts":...,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"localhost:1111","total_upstreams":1}

I’m a bit stumped on how to debug further.

4. Error messages and/or full log output:

Paste logs/commands/output here.
USE THE PREVIEW PANE TO MAKE SURE IT LOOKS NICELY FORMATTED.

5. What I already tried:

6. Links to relevant resources:

As you noticed, this does not set the TLS ServerName when making the TLS handshake. Like with any TLS server, the ServerName needs to match a SAN on the certificate. If you do curl -v https://aaa.int.example.com then you will notice it will work (unless your DNS resolver or router is misconfigured).

PS. In the future please do not redact your domain names. That is a forum rule (see link above) and is often the cause of troubleshooting problems, delays, and frustrations. Usually people do not redact correctly/accurately, and domains are public information.

What you want is --resolve from curl as you thought

curl --resolve aaa.int.example.com:443:127.0.0.1 \
     -v \
     https://aaa.int.example.com/

You can read more about naming in curl here Name resolve tricks - Everything curl and the blog post from curls author curl another host | daniel.haxx.se

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.