Reverse double proxy with IP address + https?

Help
1

1. The problem I’m having:

For most of my usage of Caddy, I reverse_proxy to localhost for containers. Simple enough, and I can just use http as its all internal to the machine. I’ve recently expanded my “fleet” a little bit, so to speak, and want one of my containers (and subsequently, domains) to run on a different machine. In other words:

domain (WAN) → router (port forward) → server with caddy (reverse proxies a specific subdomain) → another machine with caddy listening on port 443

Higher security is needed for this service, and as such I am not willing for mere HTTP traffic across the LAN, hence my conundrum of requiring HTTPS between the two machines.

2. Error messages and/or full log output:

Here is what caddy logs when a new connection to this domain is attempted.

Feb 06 23:38:01 centauri caddy[1401534]: {"level":"error","ts":1738913881.234303,"logger":"http.log.error","msg":"remote error: tls: internal error","request":{"remote_ip":"<IP>","remote_port":"39012","client_ip":"<IP>","proto":"HTTP/2.0","method":"GET","host":"<DOMAIN>","uri":"/","headers":{"Accept-Language":["en-US,en;q=0.5"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Site":["none"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101 Firefox/134.0"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Dnt":["1"],"Sec-Gpc":["1"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Priority":["u=0, i"],"Te":["trailers"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"<DOMAIN>"}},"duration":0.018268014,"status":502,"err_id":"nvx8rg4rt","err_trace":"reverseproxy.statusError (reverseproxy.go:1269)"}
Feb 06 23:38:01 centauri caddy[1401534]: {"level":"error","ts":1738913881.6438465,"logger":"http.log.error","msg":"remote error: tls: internal error","request":{"remote_ip":"<IP>","remote_port":"39012","client_ip":"<IP>","proto":"HTTP/2.0","method":"GET","host":"<DOMAIN>","uri":"/favicon.ico","headers":{"Accept-Language":["en-US,en;q=0.5"],"Sec-Fetch-Mode":["no-cors"],"Priority":["u=6"],"Sec-Fetch-Dest":["image"],"Sec-Fetch-Site":["same-origin"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101 Firefox/134.0"],"Accept":["image/avif,image/webp,image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Dnt":["1"],"Sec-Gpc":["1"],"Referer":["https://<DOMAIN>/favicon.ico"],"Te":["trailers"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"<DOMAIN>"}},"duration":0.027526728,"status":502,"err_id":"4wg8udbw4","err_trace":"reverseproxy.statusError (reverseproxy.go:1269)"}

Here are some log lines from the secondary machine (likely unhelpful, just shows the SSL challenge is failing, and I’m pretty sure all the issues are with the primary server config).

Feb 06 23:32:25 alpha caddy[176926]: {"level":"error","ts":1738913545.6700125,"logger":"http.acme_client","msg":"challenge failed","identifier":"<DOMAIN>","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"<DOMAIN IP>: remote error: tls: internal error","instance":"","subproblems":[]}}
Feb 06 23:32:25 alpha caddy[176926]: {"level":"error","ts":1738913545.6700525,"logger":"http.acme_client","msg":"validating authorization","identifier":"<DOMAIN>","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"<DOMAIN IP>: remote error: tls: internal error","instance":"","subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/183808574/22459095454","attempt":2,"max_attempts":3}
Feb 06 23:32:25 alpha caddy[176926]: {"level":"error","ts":1738913545.6700685,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"<DOMAIN>","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:tls - <DOMAIN IP>: remote error: tls: internal error"}
Feb 06 23:32:25 alpha caddy[176926]: {"level":"error","ts":1738913545.6701024,"logger":"tls.obtain","msg":"will retry","error":"[<DOMAIN>] Obtain: [<DOMAIN>] solving challenge: <DOMAIN>: [<DOMAIN>] authorization failed: HTTP 400 urn:ietf:params:acme:error:tls - <DOMAIN IP>: remote error: tls: internal error (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":5,"retrying_in":600,"elapsed":612.420105734,"max_duration":2592000}

3. Caddy version:

2.8.4 on both machines (Fedora 41 latest package)

4. How I installed and ran Caddy:

a. System environment:

Fedora 41, x86_64, systemd, native package

b. Command:

systemctl start caddy

c. Service/unit/compose file:

I’m just using what shipped with the OS.

# caddy.service
#
# For using Caddy with a config file.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy web server
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStartPre=/usr/bin/caddy validate --config /etc/caddy/Caddyfile
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectHome=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

I cannot provide the subdomain for this service. If I could help it, it wouldn’t even be internet facing, but here we are. I’ve tried both of the following files on the main (internet facing) caddy instance:

https://<DOMAIN>:443 {
  reverse_proxy https://<LAN IP (secondary)>:443
}

https://<DOMAIN>:443 {
  reverse_proxy https://<LAN IP (secondary)>:443 {
    header_up Host {header.X-Forwarded-Host}
    header_up X-Forwarded-Host {Host}
  }
}

And for the secondary (LAN only) caddy instance (this is effectively what the primary server used to have):

https://<DOMAIN>:443 {
  reverse_proxy 10.88.0.1:10007
}

All <DOMAIN> strings match - I’m not that stupid.

5. Links to relevant resources:

2

Can you use DNS challenge to obtain the certificate for the internal Caddy?

3

I’m unaware of how to do so, is that not what Caddy does automatically?

Regardless, when the container was running on the internet-facing server, Caddy was always able to obtain certificates for my subdomains.

I don’t think there’s any issues aside from I just literally do not know how to accomplish my goal here of supporting https across the first proxy jump point.

I’ll check into the documentation about setting up DNS challenge but they aren’t the easiest to read sometimes (at least for silly me) - and more importantly I’m not sure if you’re asking me to place that DNS challenge on the first server or the second (actually now that I think about it you probably meant the LAN only one by internal).

I found that Caddy has a plugin for my registrar (Porkbun). GitHub - caddy-dns/porkbun

So I added the global config to the LAN-server Caddyfile. But then I discovered that this plugin does not ship with the official binary despite being in the official(?) GitHub. So, I downloaded a custom build from the website, and it appeared to pass the challenge - however this is not an acceptable long term solution as the package manager cannot update this.

Still, I don’t mind doing it for a short time, but now when I go to my domain in the browser I still just get a white screen (reverse proxy fail, as before), but this time no additional log entries are generated on the “internal” server. So it seems traffic may have never been forwarded? Not sure. Either way, no more certificate failure (but if the traffic was never making it there to begin with, that explains why the automatic SSL failed).

So, your suggestion has certainly helped narrow down the issue, thank you.

When I try to connect from my web browser this is what the WAN facing server reports:

Feb 07 16:35:31 centauri caddy[2274968]: {"level":"error","ts":1738974931.778648,"logger":"http.log.error","msg":"remote error: tls: internal error","request":{"remote_ip":"<IP>","remote_port":"33334","client_ip":"<IP>","proto":"HTTP/2.0","method":"GET","host":"<DOMAIN>","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101 Firefox/134.0"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Dnt":["1"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-User":["?1"],"Te":["trailers"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"Accept-Language":["en-US,en;q=0.5"],"Sec-Gpc":["1"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"],"Priority":["u=0, i"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"<DOMAIN>"}},"duration":0.054393451,"status":502,"err_id":"2zb33hvr6","err_trace":"reverseproxy.statusError (reverseproxy.go:1269)"}

Clearly there is an error in the WAN-Caddy reverse proxy setup, though I’m not clear what the error is (I guess I can go check the source code, it seems to have a line number… nevermind, can’t find anything).

4

Caddy, by default, supports HTTP-01 and TLS-ALPN-01 challenges. For these to work, Caddy needs to be accessible from the internet.

From your logs, it looks like your internal Caddy is trying to obtain a certificate via the TLS-ALPN-01 challenge, but it’s failing because it isn’t reachable from the internet. That’s why I asked whether you could use the DNS-01 challenge instead, as it doesn’t require the server to be publicly accessible.

For the DNS-01 challenge, as you mentioned, Caddy will need the appropriate DNS module that works with your registrar.

One option is to install the internal Caddy as a Docker container and set up an automated build and update process.

Alternatively, you could have your external Caddy obtain the certificate for your internal domain - assuming the internal domain is publicly resolvable and routable to your external Caddy. You’d then use a scheduled script to copy the key and certificate to your internal Caddy, which would reference them using the tls directive.

Another approach is to run a separate Caddy instance with your DNS registrar module, solely for obtaining the certificate via the DNS-01 challenge, and then passing it to your internal Caddy.

You could also use tls internal on your internal Caddy and configure your external Caddy to trust its CA.

Just throwing out some possible solutions.

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.