Why does Caddy work on localhost and internal IP, but not public IP?

Help
1

1. Caddy version (caddy version):

2.4.3

2. How I run Caddy:

sudo caddy run

a. System environment:

Running in a GCE VM, Debian.

b. Command:

sudo caddy run

c. Service/unit/compose file:

Paste full file contents here.
Make sure backticks stay on their own lines,
and the post looks nice in the preview pane.

d. My complete Caddyfile or JSON config:

I’ve tried 3 variations:

localhost

respond "hello"

<internal_ip>

respond "hello"

<public_ip>

respond "hello"

3. The problem I’m having:

When I use the localhost Caddyfile and I do curl -k https://localhost, it works, great.
When I use the internal_ip Caddyfile and I do curl -k https://<internal_ip>, it works, great.
When I use the public_ip Caddyfile and I do curl -k https://<public_ip>, I get
curl: (35) error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
Similarly, Chrome gives ERR_SSL_PROTOCOL_ERROR for this case. (I was expecting a “cannot validate authenticity of this cert, do you wish to proceed” type prompt.
Mixing and matching the above combinations yielded no scenario where the public_ip case worked. Always the same error.

This is mostly for my curiosity as for my use case I don’t need the public IP case to work. But I’d like to know why this happens.

4. Error messages and/or full log output:

2022/03/02 03:23:16.802 INFO    using adjacent Caddyfile
2022/03/02 03:23:16.804 INFO    admin   admin endpoint started  {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2022/03/02 03:23:16.804 INFO    http    server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS       {"server_name": "srv0", "https_port": 443}
2022/03/02 03:23:16.804 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2022/03/02 03:23:16.811 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc0002fab60"}
2022/03/02 03:23:16.815 INFO    http    enabling automatic TLS certificate management   {"domains": ["<public_ip_redacted>"]}
2022/03/02 03:23:16.816 WARN    tls     stapling OCSP   {"error": "no OCSP stapling for [<public_ip_redacted>]: no OCSP server specified in certificate"}
2022/03/02 03:23:16.819 INFO    tls     cleaning storage unit   {"description": "FileStorage:/root/.local/share/caddy"}
2022/03/02 03:23:16.820 INFO    tls     finished cleaning storage units
2022/03/02 03:23:16.850 INFO    pki.ca.local    root certificate is already trusted by system   {"path": "storage:pki/authorities/local/root.crt"}
2022/03/02 03:23:16.850 INFO    autosaved config (load with --resume flag)      {"file": "/root/.config/caddy/autosave.json"}
2022/03/02 03:23:16.850 INFO    serving initial configuration
2

Please upgrade to v2.4.6!

You’re likely not actually hitting Caddy with that request. Try to make the request over HTTP, with curl -v to see the headers you get back. Do you get a redirect (Location header), and do you see Server: Caddy, or do you get something else?

3

Thank you for the suggestions. I’ve upgraded to 2.4.6.

Also I tried what you said and the result of curl -v http://<public_ip> is the following:

* Expire in 0 ms for 6 (transfer 0x558b56cf1fb0)
*   Trying <public_ip>...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x558b56cf1fb0)
* Connected to <public_ip> (<public_ip>) port 80 (#0)
> GET / HTTP/1.1
> Host: <public_ip>
> User-Agent: curl/7.64.0
> Accept: */*
> 
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://<public_ip>/
< Server: Caddy
< Date: Wed, 02 Mar 2022 21:00:16 GMT
< Content-Length: 0
< 
* Closing connection 0

As you can see it’s exactly what you said, a 308 with Location and Server: Caddy.
Same problem persists with https.

4

Okay. So turn on the debug global option in your Caddyfile by adding this at the top:

{
	debug
}

Then make a request with curl -vk over HTTPS and see what curl outputs gives and what Caddy logs. There might be a better explanation.

5

curl -vk https://<public_ip> yielded on the Caddy side:

2022/03/03 00:52:43.334 DEBUG   http    starting server loop    {"address": "[::]:443", "http3": false, "tls": true}
2022/03/03 00:52:43.334 DEBUG   http    starting server loop    {"address": "[::]:80", "http3": false, "tls": false}
2022/03/03 00:52:43.335 INFO    http    enabling automatic TLS certificate management   {"domains": ["34.133.83.194"]}
2022/03/03 00:52:43.335 WARN    tls     stapling OCSP   {"error": "no OCSP stapling for [34.133.83.194]: no OCSP server specified in certificate"}
2022/03/03 00:52:43.335 DEBUG   tls.cache       added certificate to cache      {"subjects": ["34.133.83.194"], "expiration": "2022/03/03 06:33:16.000", "managed": true, "issuer_key": "local", "hash": "5857d59002978b446e8c39d1e80b1c99acd578be58f4e6b27e043792181b58f9", "cache_size": 1, "cache_capacity": 10000}
2022/03/03 00:52:43.336 INFO    autosaved config (load with --resume flag)      {"file": "/root/.config/caddy/autosave.json"}
2022/03/03 00:52:43.336 INFO    serving initial configuration
2022/03/03 00:52:51.720 DEBUG   tls.handshake   no matching certificates and no custom selection logic  {"identifier": "10.128.0.3"}
2022/03/03 00:52:51.721 DEBUG   tls.handshake   no certificate matching TLS ClientHello {"server_name": "", "remote": "34.133.83.194:38190", "identifier": "10.128.0.3", "cipher_suites": [4866, 4867, 4865, 49196, 49200, 159, 52393, 52392, 52394, 49195, 49199, 158, 49188, 49192, 107, 49187, 49191, 103, 49162, 49172, 57, 49161, 49171, 51, 157, 156, 61, 60, 53, 47, 255], "cert_cache_fill": 0.0001, "load_if_necessary": true, "obtain_if_necessary": true, "on_demand": false}
2022/03/03 00:52:51.721 DEBUG   http.stdlib     http: TLS handshake error from 34.133.83.194:38190: no certificate available for '10.128.0.3'

It looks like it’s trying to match on the internal IP despite me curling the public IP and returning no certificate available for '10.128.0.3' (which is my internal IP). So I used the private IP Caddyfile again and with debug turned on.

This time, curl -vk https://<private_ip> yielded as expected

2022/03/03 00:57:25.144 DEBUG   tls.handshake   choosing certificate    {"identifier": "10.128.0.3", "num_choices": 1}
2022/03/03 00:57:25.144 DEBUG   tls.handshake   default certificate selection results   {"identifier": "10.128.0.3", "subjects": ["10.128.0.3"], "managed": true, "issuer_key": "local", "hash": "86bce8c52a0937f83a03ea78be75068616a44dc663193ccf7f6a00cf04a96fb2"}
2022/03/03 00:57:25.144 DEBUG   tls.handshake   matched certificate in cache    {"subjects": ["10.128.0.3"], "managed": true, "expiration": "2022/03/03 12:41:53.000", "hash": "86bce8c52a0937f83a03ea78be75068616a44dc663193ccf7f6a00cf04a96fb2"}
2022/03/03 00:57:25.148 INFO    http.log.access handled request {"request": {"remote_addr": "10.128.0.3:56150", "proto": "HTTP/2.0", "method": "GET", "host": "10.128.0.3", "uri": "/", "headers": {"User-Agent": ["curl/7.64.0"], "Accept": ["*/*"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "proto_mutual": true, "server_name": ""}}, "common_log": "10.128.0.3 - - [03/Mar/2022:00:57:25 +0000] \"GET / HTTP/2.0\" 200 5", "user_id": "", "duration": 0.000163331, "size": 5, "status": 200, "resp_headers": {"Server": ["Caddy"], "Content-Type": []}}

But curl -vk https://<public_ip> didn’t fail with the same error but returned no response, looks like HTTP status code 0.

2022/03/03 01:00:06.860 DEBUG   tls.handshake   choosing certificate    {"identifier": "10.128.0.3", "num_choices": 1}
2022/03/03 01:00:06.860 DEBUG   tls.handshake   default certificate selection results   {"identifier": "10.128.0.3", "subjects": ["10.128.0.3"], "managed": true, "issuer_key": "local", "hash": "86bce8c52a0937f83a03ea78be75068616a44dc663193ccf7f6a00cf04a96fb2"}
2022/03/03 01:00:06.860 DEBUG   tls.handshake   matched certificate in cache    {"subjects": ["10.128.0.3"], "managed": true, "expiration": "2022/03/03 12:41:53.000", "hash": "86bce8c52a0937f83a03ea78be75068616a44dc663193ccf7f6a00cf04a96fb2"}
2022/03/03 01:00:06.863 INFO    http.log.access handled request {"request": {"remote_addr": "34.133.83.194:38610", "proto": "HTTP/2.0", "method": "GET", "host": "34.133.83.194", "uri": "/", "headers": {"User-Agent": ["curl/7.64.0"], "Accept": ["*/*"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "proto_mutual": true, "server_name": ""}}, "common_log": "34.133.83.194 - - [03/Mar/2022:01:00:06 +0000] \"GET / HTTP/2.0\" 0 0", "user_id": "", "duration": 0.000019734, "size": 0, "status": 0, "resp_headers": {"Server": ["Caddy"]}}

Below is the curl output for curl -vk https://<public_ip>

* Expire in 0 ms for 6 (transfer 0x55926d605fb0)
*   Trying 34.133.83.194...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x55926d605fb0)
* Connected to 34.133.83.194 (34.133.83.194) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: [NONE]
*  start date: Mar  3 00:41:53 2022 GMT
*  expire date: Mar  3 12:41:53 2022 GMT
*  issuer: CN=Caddy Local Authority - ECC Intermediate
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55926d605fb0)
> GET / HTTP/2
> Host: 34.133.83.194
> User-Agent: curl/7.64.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200 
< server: Caddy
< content-length: 0
< date: Thu, 03 Mar 2022 01:00:41 GMT
< 
* Connection #0 to host 34.133.83.194 left intact
6

Right, so HTTP clients don’t set IP addresses in TLS-SNI (which Caddy uses to figure out which cert to load for the handshake), so it uses the local network address instead as a best-guess, so that LAN connections can still work (I think that’s the reasoning, anyway).

Basically, TLS certificates for IP addresses doesn’t work too well due to a confluence of reasons. Trust is hard to establish, because nobody really “owns” an IP address (except big ISPs or network operators), but people can own and prove ownership of a domain.

7

Yep, and the difference in IP (curling one, then caddy getting it on another) could likely be the router doing something funky. I have seen that happen where you use a public IP from inside and it actually has that in it’s routing table to an internal device, so it routes it on the internal IP.

8

I see, thank you both. I wonder if the fact that I’m running all this on a GCP VM has anything to do with it (some sort of internal kink with GCP’s networking perhaps). So is it because the request being routed to the internal address that I receive no response when I curl the public IP?

9

No, that’s just how it works. This is the code that tries to grab the hostname to match a certificate:

If it’s not in TLS-SNI (which it isn’t because it’s an IP address, and by the RFC spec, TLS-SNI should never contain an IP address, only domains), then certmagic tries to fallback to the local IP address from the connection.

10

This topic was automatically closed after 30 days. New replies are no longer allowed.