Caddy reverse proxy curl works internally but externally returns content-length: 0

Help
1

1. Output of caddy version:

v2.5.2 h1:eCJdLyEyAGzuQTa5Mh3gETnYWDClo1LjtQm2q9RNZrs=

2. How I run Caddy:

a. System environment:

Ubuntu 22.04 LTS headless server behind NETGEAR R6250 router.

b. Command:

caddy run --watch

c. Service/unit/compose file:

N/A

d. My complete Caddy config:

{
        default_sni 192.168.1.5
}

https://192.168.1.5:443 {
        tls internal

        handle /static/* {
                root * /srv
                file_server browse
        }

        handle /foundry/* {
                reverse_proxy /foundry/* localhost:30000
        }

        handle /foundry {
                reverse_proxy /foundry localhost:30000
        }

        handle {
                reverse_proxy * localhost:8000
        }
}

3. The problem I’m having:

I’m trying to set up Caddy as a reverse proxy between two web applications and a static file server (all on one machine). When I curl the internal IP, it works as expected, but when I try to curl the external IP, it returns content-length: 0. Ultimately, my question is why is this happening?

I’m not using a domain name, just straight IP. I know I could get a free domain from a bunch of different DNS hosts; I’d really rather not.

Network Setup

I have a router connected to a single single physical server that is home to Caddy and the two web applications. I’m forwarding port 443 from my router to the server where Caddy is installed.

4. Error messages and/or full log output:

caddy run --watch 

2022/08/21 20:18:50.848 INFO    watcher config file changed; reloading  {"config_file": "Caddyfile"}
2022/08/21 20:18:50.848 INFO    using provided configuration    {"config_file": "Caddyfile", "config_adapter": ""}
2022/08/21 20:18:50.851 WARN    Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies       {"adapter": "caddyfile", "file": "Caddyfile", "line": 2}
2022/08/21 20:18:50.853 INFO    admin   admin endpoint started  {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2022/08/21 20:18:50.853 INFO    admin   stopped previous server {"address": "tcp/localhost:2019"}
2022/08/21 20:18:50.854 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc00016d7a0"}
2022/08/21 20:18:50.855 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2022/08/21 20:18:50.857 INFO    pki.ca.local    root certificate is already trusted by system   {"path": "storage:pki/authorities/local/root.crt"}
2022/08/21 20:18:50.857 DEBUG   http    starting server loop    {"address": "[::]:80", "http3": false, "tls": false}
2022/08/21 20:18:50.857 DEBUG   http    starting server loop    {"address": "[::]:443", "http3": false, "tls": true}
2022/08/21 20:18:50.857 INFO    http    enabling automatic TLS certificate management   {"domains": ["192.168.1.5"]}
2022/08/21 20:18:50.858 WARN    tls     stapling OCSP   {"error": "no OCSP stapling for [192.168.1.5]: no OCSP server specified in certificate", "identifiers": ["192.168.1.5"]}
2022/08/21 20:18:50.858 DEBUG   tls.cache       added certificate to cache      {"subjects": ["192.168.1.5"], "expiration": "2022/08/22 04:48:25.000", "managed": true, "issuer_key": "local", "hash": "0e0049589aae4e422aced249b7b26fedc44e4929fdef4ea4b7e48989115e908b", "cache_size": 1, "cache_capacity": 10000}
2022/08/21 20:18:50.861 INFO    tls.cache.maintenance   stopped background certificate maintenance      {"cache": "0xc0001678f0"}
2022/08/21 20:18:50.862 INFO    autosaved config (load with --resume flag)      {"file": "/root/.config/caddy/autosave.json"}
2022/08/21 20:19:11.612 DEBUG   tls.handshake   choosing certificate    {"identifier": "192.168.1.5", "num_choices": 1}
2022/08/21 20:19:11.612 DEBUG   tls.handshake   default certificate selection results   {"identifier": "192.168.1.5", "subjects": ["192.168.1.5"], "managed": true, "issuer_key": "local", "hash": "0e0049589aae4e422aced249b7b26fedc44e4929fdef4ea4b7e48989115e908b"}
2022/08/21 20:19:11.612 DEBUG   tls.handshake   matched certificate in cache    {"subjects": ["192.168.1.5"], "managed": true, "expiration": "2022/08/22 04:48:25.000", "hash": "0e0049589aae4e422aced249b7b26fedc44e4929fdef4ea4b7e48989115e908b"}

curl on Internal IP

admin@server:~$ curl -vk https://192.168.1.5/
*   Trying 192.168.1.5:443...
* TCP_NODELAY set
* Connected to 192.168.1.5 (192.168.1.5) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: [NONE]
*  start date: Aug 21 00:46:28 2022 GMT
*  expire date: Aug 21 12:46:28 2022 GMT
*  issuer: CN=Caddy Local Authority - ECC Intermediate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x561a91115210)
> GET / HTTP/2
> Host: 192.168.1.5
> user-agent: curl/7.68.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200 
< content-type: text/html; charset=utf-8
< cross-origin-opener-policy: same-origin
< referrer-policy: same-origin
< server: Caddy
< x-content-type-options: nosniff
< x-frame-options: DENY
< content-length: 2921
< date: Sun, 21 Aug 2022 05:07:17 GMT
 
<!doctype html>
<html lang="en">
  <body>
    <h1>Hello World!</h1>
  </body>
</html>
* Connection #0 to host 192.168.1.5 left intact

curl on External IP

admin@server:~$ curl -vk https://203.0.113.0/
*   Trying 203.0.113.0:443...
* TCP_NODELAY set
* Connected to 203.0.113.0 (203.0.113.0) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: [NONE]
*  start date: Aug 21 00:46:28 2022 GMT
*  expire date: Aug 21 12:46:28 2022 GMT
*  issuer: CN=Caddy Local Authority - ECC Intermediate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55e3ae7aa210)
> GET / HTTP/2
> Host: 203.0.113.0
> user-agent: curl/7.68.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200 
< server: Caddy
< content-length: 0
< date: Sun, 21 Aug 2022 05:08:16 GMT
< 
* Connection #0 to host 203.0.113.0 left intact

5. What I already tried:

6. Links to relevant resources:

Same issue on Server Fault

2

Your request from external has the wrong IP address as the Host, so it doesn’t match your site address’s host matcher which is looking for 192.168.1.5.

3

Hey thanks @francislavoie for replying!

So based on what you said, I reconfigured my Caddyfile to this:

{
        default_sni 192.168.1.5
        debug
}
 
https://192.168.1.5:443 {
        tls internal

        handle /static/* {
                root * /srv
                file_server browse
        }

        handle /foundry/* {
                reverse_proxy /foundry/* localhost:30000
        }

        handle /foundry {
                reverse_proxy /foundry localhost:30000
        }

        handle {
                reverse_proxy * localhost:8000
        }
} 

https://203.0.113.0:443 {
        tls internal

        handle /static/* {
                root * /srv
                file_server browse
        }

        handle /foundry/* {
                reverse_proxy /foundry/* localhost:30000
        }

        handle /foundry {
                reverse_proxy /foundry localhost:30000
        }

        handle {
                reverse_proxy * localhost:8000
        }
}

I ran curl again, but not much change:

admin@server:~$ curl -vk https://203.0.113.0/
*   Trying 203.0.113.0:443...
* TCP_NODELAY set
* Connected to 203.0.113.0 (203.0.113.0) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: [NONE]
*  start date: Aug 21 16:48:25 2022 GMT
*  expire date: Aug 22 04:48:25 2022 GMT
*  issuer: CN=Caddy Local Authority - ECC Intermediate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x563e5f407210)
> GET / HTTP/2
> Host: 203.0.113.0
> user-agent: curl/7.68.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200 
< server: Caddy
< content-length: 0
< date: Sun, 21 Aug 2022 21:28:47 GMT
< 
* Connection #0 to host 203.0.113.0 left intact

I also tried removing tls internal from the https://203.0.113.0:443 route; no change.

I’m still learning about how networking works, so I’m going to try to explain what I think is supposed to happen, and hopefully you can tell me where I’m messing this up:

  1. A request originates from some machine connected to the internet and is routed to my IP, which is https://203.0.113.0/
  2. The router receives the request for https://203.0.113.0/. It takes that request, determines that it needs to go to the internal IP https://192.168.1.5/ and forwards it there.
  3. Caddy, which is running at https://192.168.1.5/, receives the request and reroutes it to https://localhost:8000/.
  4. The application listening to https://localhost:8000/ gets the request and sends a response to Caddy at https://192.168.1.5/
  5. Caddy recieves the response and sends it to the router.
  6. The router sends the response off to my ISP who routes it back to the original requester.

Based on that, it seems like this chain is breaking between step 2 and 3, where the router is just forwarding the original request (I’m assuming without modifying the headers, but I really don’t understand headers very well…), and so Caddy is seeing a request for https://203.0.113.0/ instead of (what I would think it should see) https://192.168.1.5/.

Is that accurate?

Thanks again for your help!

4

@francislavoie, you were right, I just didn’t do a very good job of implementing your answer. My (working!) Caddyfile now looks like this:

{
        default_sni 192.168.1.5
}

https://203.0.113.0:443, https://192.168.1.5:443 {

        handle /static/* {
                root * /srv
                file_server browse
        }

        handle /foundry/* {
                reverse_proxy /foundry/* localhost:30000
        }

        handle /foundry {
                reverse_proxy /foundry localhost:30000
        }

        handle {
                reverse_proxy * localhost:8000
        }
}

A few important things I learned:

  1. As @francislavoie pointed out, the request forwarded from the router does not change the Host header to the internal IP address; it just forwards the whole request to the internal IP address. So instead of receiving a request for 192.168.1.5, Caddy was actually receiving a request for 203.0.113.0. Since there wasn’t a route for 203.0.113.0, it was just sending back a blank page.
  2. The router was configured to listen on port :580 (I forgot to mention this in the original post). It was forwarding requests from :580 to 192.168.1.5:443. The Caddyfile is structured such that the site address takes the form:
<http | https>://<your external IP>:<port that Caddy is listening on>

I had tried https://203.0.113.0:580 as a site address, but it failed to match because that told Caddy to listen on port 580 and the request was being sent by the router to port 443.

Bottom Line

If you’re trying to use Caddy without a domain name, make sure the site address in your Caddyfile looks like this:

<http | https>://<your external IP>:<port that Caddy is listening on>
1 Like
5

Glad you figured it out!

I strongly recommend just using a domain though, tbh.

You can use something like DuckDNS which will keep your external IP up to date.

Then if necessary (i.e. your router doesn’t support NAT loopback/hairpinning), then you can set up a local DNS server to make machines in your local network resolve your domain to your LAN IP instead of the WAN IP.

You can simplify this to either:

	handle /foundry* {
		reverse_proxy localhost:30000
	}

or if you really must not match things like /foundryfoo, then:

	@foundry path /foundry /foundry/*
	handle @foundry {
		reverse_proxy localhost:30000
	}

You can simplify your site address (scheme and port default to HTTPS and 443):

203.0.113.0, 192.168.1.5 {
1 Like
6

This topic was automatically closed after 30 days. New replies are no longer allowed.