TLS handshake error and no certificate available

tcurdt · March 10, 2021, 12:16am

1. Caddy version (`caddy version`):

v2.3.0 h1:fnrqJLa3G5vfxcxmOH/+kJOcunPLhSBnjgIvjXV/QTA=

2. How I run Caddy:

a. System environment:

Debian Buster from the command line for testing as root.

b. Command:

./caddy run -config Caddyfile

c. Service/unit/compose file:

d. My complete Caddyfile or JSON config:

{
  debug
}

dev3.foo.com {

  log {
    output stdout
    format console
    level debug
  }

  reverse_proxy /api/* {
    to srv+http://backend-staging.service.fsn1.consul
  }
  reverse_proxy {
    to srv+http://frontend-staging.service.fsn1.consul
  }
}

3. The problem I’m having:

A couple of problems:

When I open the admin on 127.0.0.1:2019 I am getting a 404.
When I use curl to access the server I am getting a SSL error. With both the Host header set or not set.

curl -v -H "Host: dev3.foo.com" https://127.0.0.1
* Expire in 0 ms for 6 (transfer 0x55e010cebfb0)
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x55e010cebfb0)
* Connected to 127.0.0.1 (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, internal error (592):
* error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
* Closing connection 0
curl: (35) error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error

in the log I see

2021/03/09 23:46:03.528	DEBUG	http.stdlib	http: TLS handshake error from 127.0.0.1:49064: no certificate available for '127.0.0.1'

In the full log (below) I am seeing continuous TLS handshake errors for quora.com a domain I don’t even own. I assume this is someone sending a requests with the host header set to quora.com but to that IP? I assume with 2) fixed this should be OK to ignore.

4. Error messages and/or full log output:

./caddy run -config Caddyfile 
2021/03/09 23:19:13.282 INFO  using provided configuration  {"config_file": "Caddyfile", "config_adapter": ""}
2021/03/09 23:19:13.285 INFO  admin admin endpoint started  {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["127.0.0.1:2019", "localhost:2019", "[::1]:2019"]}
2021/03/09 23:19:13.285 INFO  http  server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2021/03/09 23:19:13.285 INFO  http  enabling automatic HTTP->HTTPS redirects  {"server_name": "srv0"}
2021/03/09 23:19:13.285 INFO  tls.cache.maintenance started background certificate maintenance  {"cache": "0xc00039d2d0"}
2021/03/09 23:19:13.286 DEBUG http  starting server loop  {"address": "[::]:443", "http3": false, "tls": true}
2021/03/09 23:19:13.286 DEBUG http  starting server loop  {"address": "[::]:80", "http3": false, "tls": false}
2021/03/09 23:19:13.286 INFO  http  enabling automatic TLS certificate management {"domains": ["dev3.foo.com"]}
2021/03/09 23:19:13.287 DEBUG tls loading managed certificate {"domain": "dev3.foo.com", "expiration": "2021/06/07 22:04:30.000", "issuer_key": "acme-v02.api.letsencrypt.org-directory", "storage": "FileStorage:/root/.local/share/caddy"}
2021/03/09 23:19:13.287 INFO  tls cleaned up storage units
2021/03/09 23:19:13.301 INFO  autosaved config  {"file": "/root/.config/caddy/autosave.json"}
2021/03/09 23:19:13.301 INFO  serving initial configuration
2021/03/09 23:19:14.682 DEBUG http.stdlib http: TLS handshake error from 37.129.73.186:52876: no certificate available for 'quora.com'
2021/03/09 23:19:18.183 DEBUG http.stdlib http: TLS handshake error from 2.147.183.203:60811: no certificate available for 'quora.com'
2021/03/09 23:19:24.101 DEBUG http.stdlib http: TLS handshake error from 37.129.73.186:52878: no certificate available for 'quora.com'
2021/03/09 23:19:33.543 DEBUG http.stdlib http: TLS handshake error from 37.129.73.186:52880: no certificate available for 'quora.com'
...

5. What I already tried:

I search the forum. This sounded similar - but I have a domain specified.
Plus I see the information about the cert in the log.

6. Links to relevant resources:

Whitestrake · March 10, 2021, 12:43am

G’day @tcurdt, welcome to the Caddy community.

Looks like you’ve specified the host dev3.foo.com and the handshake errors are for quora.com.

This would appear to be completely expected behaviour since you’re not serving quora.com.

tcurdt · March 10, 2021, 12:46am

Yes, that’s rather clear:

I assume this is someone sending a requests with the host header set to quora.com but to that IP? I assume with 2) fixed this should be OK to ignore.

But the main problem is that even for the configured domain I am getting TLS errors.

Whitestrake · March 10, 2021, 12:49am

Oh, my bad!

no certificate available for '127.0.0.1'

This really heavily implies the Host header wasn’t set. That’s pretty weird considering the very obvious -H "Host: dev3.foo.com" in your curl command.

Try using --resolve instead of -H and see if curl sends SNI properly:

curl -v --resolve dev3.foo.com:443:127.0.0.1 https://dev3.foo.com/

tcurdt · March 10, 2021, 12:52am

That worked!

Whitestrake · March 10, 2021, 12:54am

Alright, so 2) was just a matter of Caddy rejecting the connection early because there was no SNI, a problem that shouldn’t pop up in production / when DNS is configured.

Getting requests for quora.com is pretty weird still, but not really in your power to do anything about (unless you know who the heck is sending those requests).

404s for the admin interface is also strange. Have you tried localhost:2019?

tcurdt · March 10, 2021, 1:03am

Odd. I tested in the browser first. Then went to curl to make it more reproducible. But fair enough. Maybe a glitch. And I learned something new with curl

…but why is the admin giving me a 404?

And is there a way to prevent adding the /api to the request to upstream?

tcurdt · March 10, 2021, 1:05am

I have a ssh tunnel into the machine to forwards to 127.0.0.1:2019.
But even a curl on the machine results in a 404.

$ curl http://127.0.0.1:2019
404 page not found
$ curl http://localhost:2019
404 page not found

Whitestrake · March 10, 2021, 1:09am

Do you mean, stop the client from requesting the /api path?

I don’t think there’s anything Caddy would be doing to explicitly add it, it should be passing the path through exactly as presented by the client.

This is very strange indeed, especially given the log output: 2021/03/09 23:19:13.285 INFO admin admin endpoint started {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["127.0.0.1:2019", "localhost:2019", "[::1]:2019"]}

Just to double confirm, you definitely don’t have Caddy containerized in any way?

tcurdt · March 10, 2021, 1:24am

Well, not quite. Right now Caddy does the “normal” thing. These two give the same result with the upstream being the 2nd line.

curl --resolve dev3.foo.com:443:127.0.0.1 https://dev3.foo.com/api/
curl 10.0.1.2:29194/

Our upstream is a bit - weird. And still needs to be fixed. But for now I need these two to be the same:

curl --resolve dev3.foo.com:443:127.0.0.1 https://dev3.foo.com/api/
curl 10.0.1.2:29194/api

I thought this should work:

  handle_path /api* {
    reverse_proxy {
      to srv+http://backend-staging.service.fsn1.consul
    }
  }

  reverse_proxy {
    to srv+http://frontend-staging.service.fsn1.consul
  }

But that doesn’t seem to do it either. Which I find a bit confusing.

tcurdt · March 10, 2021, 1:25am

While that is the plan down the road… nope - running on the command line:

# ./caddy run -config Caddyfile 
2021/03/10 01:25:31.907	INFO	using provided configuration	{"config_file": "Caddyfile", "config_adapter": ""}
...

Whitestrake · March 10, 2021, 1:36am

tcurdt:

Well, not quite. Right now Caddy does the “normal” thing. These two give the same result with the upstream being the 2nd line.
curl --resolve dev3.foo.com:443:127.0.0.1 https://dev3.foo.com/api/
curl 10.0.1.2:29194/

This strikes me as odd.

If you request the /api/ path from Caddy, then Caddy should request the /api/ path from upstream.

By default a request to https://dev3.foo.com/api/ should produce the same result as a request to 10.0.1.2:29194/api/.

Your handle_path attempt should have worked - that directive is designed to strip the prefix it matches.

Just to clarify, what is the desired outcome?

Strip the /api prefix for all requests (i.e. no upstream request starts with /api regardless of whether the client requested it)
Ensure all requests have the /api prefix (i.e. every upstream request starts with /api, regardless of client request)
Leave the prefix up to the client

tcurdt · March 10, 2021, 1:41am

Leave the prefix up to the client. That means no stripping or adding. Just these two being the same:

curl --resolve dev3.foo.com:443:127.0.0.1 https://dev3.foo.com/api
curl 10.0.1.2:29194/api

Whitestrake · March 10, 2021, 1:43am

If you’d like to leave it up to the client, then, I suspect you do not need two proxy handlers at all.

Simply:

reverse_proxy srv+http://frontend-staging.service.fsn1.consul

(i.e. empty/catch-all matcher) should reverse proxy all requests (regardless of whether they start with /api) and should not modify the URI in any way when proxying them upstream (so the URI the client requests should be the exact URI the upstream receives).

Whitestrake · March 10, 2021, 1:47am

Actually, wait. You’ve got a frontend and a backend I missed. Give me a quick sec

tcurdt · March 10, 2021, 1:48am

Actually - I think I misunderstood your options. All requests going to the one upstream should have the prefix /api.

We have two upstreams:

srv+http://backend-staging.service.fsn1.consul
srv+http://frontend-staging.service.fsn1.consul

Which is why I don’t see how caddy should know which one to pick if not matching based on the /api.

Whitestrake · March 10, 2021, 1:51am

A matcher, on its own, never modifies a request. It’s only ever a conditional.

I’m not sure what the problem with your original post’s config would be:

To my understanding, with this configuration, a request to (for example):

dev3.foo.com/api/bar

Should result in Caddy sending a request to the upstream with the URI /api/bar (i.e. unmodified).

Since the matcher for the backend handler requires the path /api/*, by definition, the only requests being sent to this upstream should always be prefixed with /api/.

Can you give me any specific examples of requests that aren’t being handled correctly with this setup?

tcurdt · March 10, 2021, 1:59am

This is how it is at the moment:

# curl --resolve dev3.foo.com:443:127.0.0.1 https://dev3.foo.com/api
<h1>Welcome</h1>

# curl --resolve dev3.foo.com:443:127.0.0.1 https://dev3.foo.com/api/api
{"message":"Welcome to the API"}

This is how it should be

# curl --resolve dev3.foo.com:443:127.0.0.1 https://dev3.foo.com/api
{"message":"Welcome to the API"}

And this is how the upstream behaves:

# curl 10.0.1.2:29194
<h1>Welcome</h1>

# curl 10.0.1.2:29194/api
{"message":"Welcome to API"}

It’s late and I am getting tired - but to me it looks like Caddy is adding the mount point to the request.

tcurdt · March 10, 2021, 2:02am

That’s exactly what I was expecting. It’s just not what I am seeing at the moment.
I am wondering if there is easy way to intercept the traffic. Maybe with tcpdump?

Whitestrake · March 10, 2021, 2:14am

This request isn’t being sent to the backend, it’s being sent to the frontend.

A request to /api does not match the path matcher /uri/* (pay close attention to that last slash before the wildcard).

If you requested /api/, that should get a backend response.

If you wanted to get a backend response for the literal URI /api, you’ll need to change your config a bit. You could set the backend path matcher to /api*, but then you’d filter in requests for /apifoo etc, which might not be a practical problem but most likely isn’t correct.

Instead, a common config pattern for handling this is:

rewrite /api /api/
reverse_proxy /api/* <backend>
reverse proxy * <frontend>

This effectively “includes” the literal URI /api in the backend matcher.