Configuring HTTPS for Multiple Services on Caddy: Empty Response Issue

1. The problem I’m having:

Hi everybody, I’m currently facing a challenge in using Caddy to securely connect to three different web services, which operate on multiple ports (26657, 1317, 8545), over HTTPS. For my node to be eligible for listing, it must maintain a healthy status, verifiable on port 26657. When conducting a status check via HTTP, everything appears as expected:

HTTP Request Example:
curl -iL http://rollapp.ilanihas.xyz:26657/status -H "Host:rollapp.ilanihas.xyz"

Expected HTTP Response:

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Vary: Origin
X-Content-Type-Options: nosniff
Date: Sun, 11 Feb 2024 22:23:51 GMT
Content-Length: 1012

{"jsonrpc":"2.0","result":{"node_info":{"protocol_version":{"p2p":"8","block":"11","app":"0"},"id":"0024080112207736451f5f36e62a6b2de80eef384263a98d5c64a410699e4d82a26c8f16403c","listen_addr":"/ip4/0.0.0.0/tcp/26656","network":"ilanihas_8485150-1","version":"\u003cversion\u003e","channels":"01","moniker":"vmd128706.contaboserver.net","other":{"tx_index":"on","rpc_address":"tcp://127.0.0.1:26657"}},"sync_info":{"latest_block_hash":"E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855","latest_app_hash":"8B7845573516FF459D0A2E07BCC10A1BD98DFEA7F150370E827BF1F8E41AA28A","latest_block_height":"2644","latest_block_time":"2024-02-11T22:23:50.254037976Z","earliest_block_hash":"","earliest_app_hash":"","earliest_block_height":"0","earliest_block_time":"0001-01-01T00:00:00Z","catching_up":false},"validator_info":{"address":"B59FDF13A7A0D7D22856D4DABE465AC081CB97CF","pub_key":{"type":"tendermint/PubKeyEd25519","value":"MQW6TxyuL0FXp0L3e0BVM/ZjW7stldP3FrHX8cPvopY="},"voting_power":"1"}},"id":-1}

However, trying to an HTTPS GET request leads to a 200 status as well, but with an empty response body, contrary to my expectation of receiving the same JSON:

HTTPS Request Attempt:
curl -iL https://rollapp.ilanihas.xyz/status -H "Host:rollapp.ilanihas.xyz"

HTTPS Response Observed:

HTTP/2 200
alt-svc: h3=":443"; ma=2592000
date: Sun, 11 Feb 2024 22:25:25 GMT
server: Caddy
vary: Origin
content-length: 0

Just to let you know: The RPC endpoints I’m working with are currently overloaded, so now and then, I run into an HTTP 503 error that mentions “Error broadcasting batch[error Failed broadcasting tx: Priority is too low:).” You might encounter the same issue if you’re testing it. This error suggests something’s wrong with my node. Oddly enough, I get this message for both HTTP and HTTPS, which confuses me because I’m not seeing the expected response when the call is supposed to be successful. Thanks a lot for your assistance.

2. Error messages and/or full log output:

No error messages:

Feb 12 13:12:19 {"level":"info","ts":1707739939.220786,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000617600"}
Feb 12 13:12:19 {"level":"info","ts":1707739939.2223523,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Feb 12 13:12:19 {"level":"info","ts":1707739939.2233033,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Feb 12 13:12:19 {"level":"info","ts":1707739939.2234688,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2",>
Feb 12 13:12:19 {"level":"info","ts":1707739939.2235634,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["rollapp.ilanihas.xyz"]}
Feb 12 13:12:19 {"level":"info","ts":1707739939.2281837,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Feb 12 13:12:19 {"level":"info","ts":1707739939.230063,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Feb 12 13:12:19 systemd[1]: Started Caddy.
Feb 12 13:12:19 caddy[421739]: {"level":"info","ts":1707739939.2304993,"logger":"tls","msg":"finished cleaning storage units"}
Feb 12 13:12:19 caddy[421739]: {"level":"info","ts":1707739939.2313025,"msg":"serving initial configuration"}

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy
sudo systemctl start caddy

a. System environment:

Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:        22.04
Codename:       jammy

b. Command:

c. Service/unit/compose file:

d. My complete Caddy config:

https://rollapp.ilanihas.xyz:443 {
  handle_path /status* {
    reverse_proxy http://127.0.0.1:26657
  }
  handle_path /rest* {
    reverse_proxy http://127.0.0.1:1317
  }
  handle_path /evm* {
    reverse_proxy http://127.0.0.1:8545
  }
  tls {
    protocols tls1.2 tls1.3
  }
}

5. Links to relevant resources:

Thank you very much.

handle_path strips the matched path prefix before proxying. That means that a request to /status will become / before it reaches the upstream.

Use handle instead if you don’t want the path to be stripped.

Remove this from your config, you shouldn’t re-state the defaults. Imagine for example later TLS1.4 exists, with this config you would prevent it from being enabled, which is not good.

Enable the debug global option, it’s helpful to see what’s going on in situations like this. You would see in your logs that the proxy request has / as the path, and you would see a log from the handle_path rewrite.

1 Like

You’re amazing, thank you so much for the prompt assistance! We can consider this issue resolved as everything is functioning as expected now. Have a great day, Francis.

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.