TLS handshake error: no certificate available for 'IP'"

Help
1

1. The problem I’m having:

I’m trying to reverse-proxy to a backend app with an AWS load balancer in front of Caddy, like this:

Internet -> AWS Load Balancer:443 -> Caddy:443 -> http://localhost:1337

A user will request https://myservice.example.com, which points to the AWS Load Balancer. The AWS Load Balancer will terminate TLS, then will open a new HTTPS request to Caddy, Caddy will terminate TLS again, Caddy will proxy to http://localhost:1337, then Caddy will return the request to the AWS Load Balancer and the AWS Load Balancer will return to the user.

The problem I’m having is that the AWS Load Balancer will healthcheck Caddy using the private IP address of the Caddy container, without any context of the domain name. Caddy fails these health checks because it has no HTTPS certificate for its own private IP address, so the load balancer sees Caddy as being unhealthy.

Is there any way to allow Caddy to accept HTTPS requests to its own private IP, without hardcoding that IP or knowing its own private IP ahead of time?

Thanks.

2. Error messages and/or full log output:

{
    "level": "debug",
    "ts": 1712674878.363298,
    "logger": "tls.handshake",
    "msg": "no matching certificates and no custom selection logic",
    "identifier": "172.30.6.242"
}
{
    "level": "debug",
    "ts": 1712674878.3633108,
    "logger": "tls.handshake",
    "msg": "no certificate matching TLS ClientHello",
    "remote_ip": "172.30.5.206",
    "remote_port": "47752",
    "server_name": "",
    "remote": "172.30.5.206:47752",
    "identifier": "172.30.6.242",
    "cipher_suites": [
        49195,
        49199,
        49187,
        49191,
        49161,
        49171,
        49196,
        49200,
        49188,
        49192,
        49172,
        49162,
        156,
        60,
        47,
        157,
        61,
        53,
        255
    ],
    "cert_cache_fill": 0.0001,
    "load_or_obtain_if_necessary": true,
    "on_demand": false
}
{
    "level": "debug",
    "ts": 1712674878.3633814,
    "logger": "http.stdlib",
    "msg": "http: TLS handshake error from 172.30.5.206:47752: no certificate available for '172.30.6.242'"
}

3. Caddy version:

caddy:2.7-alpine

4. How I installed and ran Caddy:

a. System environment:

Running in AWS Fargate, where Caddy runs in one container and my backend service runs in another container, both on the same host. Caddy will proxy from :443 to http://localhost:1337. The Caddy “installation” is just the vanilla Docker image listed above, with no custom Caddyfile.

b. Command:

      command = [
        # https://caddyserver.com/docs/command-line#caddy-reverse-proxy
        "caddy", "reverse-proxy",
        "--to", "http://localhost:1337",
        "--disable-redirects", # Don't bind to an HTTP port, the ALB already handles HTTP redirects
        "--internal-certs",    # Self-signed certificates are good enough for internal communication
        "--insecure",
        "--debug",
      ]

c. Service/unit/compose file:

Not really relevant as I'm using Docker without any Caddyfile

d. My complete Caddy config:

Not really relevant as I'm using Docker without any Caddyfile

5. Links to relevant resources:

2

You really should use a Caddyfile. The reverse-proxy command is very limited, so you won’t be able to implement some of these adjustments.

You could enable On-Demand TLS which would issue a certificate using Caddy’s internal CA, but the certificates would not be trusted by the client automatically.

It won’t be possible to get a trusted IP cert, because no ACME issuers provide support for IP certs.

Are you sure AWS can’t be configured to set TLS-SNI? That would be surprising to me.

Why proxy over HTTPS from AWS to Caddy? I assume Caddy isn’t using a publicly trusted certificate (because automating TLS when behind a proxy which terminated TLS is difficult).

If you’re not using a trusted cert, then there’s no benefit at all to proxying over HTTPS, so you can simplify by proxying to Caddy over HTTP, then your health checks would just work.

Without trust, HTTPS has no benefit, because anyone at all could man-in-the-middle the connection, since the client ignores the issuer of the certificate (so someone could use their own cert and decrypt/reencrypt the traffic, reading it or manipulating it maliciously).

2 Likes
3

@francislavoie thanks for taking the time to read and reply. Much appreciated.

I found out today that the project that we planned to use Caddy with is being cancelled (unrelated to this question and unrelated to Caddy entirely), so I don’t need any additional help.

But I’ll still answer the questions about our use case it case it’s helpful to understand how users might want to use Caddy…

You really should use a Caddyfile.

Understandable, but the thing which was most appealing to me about Caddy is that it appeared to be possible to use without a Caddyfile. If I was able to use Caddy as a reverse proxy without a Caddyfile, then I can use the vanilla image from DockerHub and I can configure Caddy entirely by way of CLI commands and environment variables.

If I used a Caddyfile, then I’ll need a significant amount of infrastructure to make that happen. Eg. I’ll need a Caddyfile, a Dockerfile, some kind of CI/CD pipeline to build the image (Jenkins, AWS Codebuild, etc), and a image repository (ECR, etc), and the various IAM policies which accompany those things. And if I really truly do need those things, that’s fair, but it’s more appealing if I didn’t need them.

Why proxy over HTTPS from AWS to Caddy? If you’re not using a trusted cert, then there’s no benefit at all to proxying over HTTPS…

Yes, it’s accurate that enforcing HTTPS between the load balancer and the backend is of negligible technical benefit since that connection happens entirely within our private VPC, and so even if we used HTTP for that connection it wouldn’t be the end of the world. The benefit in doing so is mostly being able to tell our compliance teams and auditors that we use HTTPS everywhere.

And FWIW, we do have other applications which do use HTTPS between the load balancer and the backend where we reverse-proxy via nginx with a self-signed certificate and AWS load balancers are able to play nicely with that setup. Those other applications already required their own Dockerfile, CI/CD pipeline, image repository, etc, and so it wasn’t much trouble to bake in an nginx.conf with a self-signed certificate to handle HTTPS coming from the load balancer.

Thanks again for your reply. I’ll make this topic as closed/resolved/whatever.

1 Like
4

Can you not mount files in AWS? That sounds silly.

You can run Caddy with a Caddyfile from CLI:

echo "example.com {\n\treverse_proxy localhost:9000\n}" | caddy run -a caddyfile -c-

The reverse-proxy command is really just meant for quick-and-dirty local development, not for production use.

That would be a lie, because HTTPS without trust is the same as no security at all.

You can do that too with Caddy with tls internal to have Caddy issue certs using its internal issuer, but again, you still need to establish trust though (take Caddy’s root CA cert and configure the client/proxy with it).

2 Likes