Caddy LXC instance reverse proxying its own host

1. The problem I’m having:

I’m using Proxmox VE (8.2.2) with Caddy and Pihole running as LXC containers on the same Host.

In my current setup:

  • Caddy is set up to get a wildcard cert which I can successfully use to reverse proxy to other LXC containers that expose either http sources or self-signed certs over https. See Caddyfile below.

  • I forward everything to Caddy’s IP using a rule I set up in Pihole

  • I’m running a split-dns setup, with NO external access using above.

What I want to do:

  • Access the PVE instance using a valid certificate from the internal network.
  • Use a host specific certificate obtained via Proxmox ACME implementation
  • Access the service over port 80 instead of 8006

The three requirements appear to be in conflict, unless another reverse-proxy is added to the mix, such as the one documented here: Caddy as a Proxmox Reverse Proxy - Wiki - Caddy Community

I would prefer to use the LXC container for this purpose, exposing the certificate assigned to the host is also not ideal.

Things I have tried:

  • I can directly assign a Local DNS entry in PiHole to override the normal routing to Caddy. This is the easiest fix with the caveat that the port remains as 8006

  • I can put an entry in the Caddyfile that will reverse proxy the PVE host at the cost of ignoring the host-specific certificate and just depending on the wildcard one requested from Caddy.

  • I tried putting in a rule that would reverse proxy the PVE instance, but this setup led to error messages as below

2. Error messages and/or full log output:

Jun 14 10:37:28 caddy caddy[10032]: {"level":"error","ts":1718375848.2796555,"logger":"http.log.error","msg":"tls: failed to verify certificate: x509: cannot validate certificate for 192.168.1.7 because it doesn't contain any IP SANs","request":{"remote_ip":"192.168.1.163","remote_port":"59085","client_ip":"192.168.1.163","proto":"HTTP/3.0","method":"GET","host":"pve.example.com","uri":"/","headers":{"Cookie":["REDACTED"],"Cache-Control":["max-age=0"],"Accept-Language":["en-US,en;q=0.7"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Sec-Fetch-Mode":["navigate"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Gpc":["1"],"Sec-Fetch-Dest":["document"],"Priority":["u=0, i"],"Sec-Ch-Ua":["\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Brave\";v=\"126\""],"Sec-Ch-Ua-Mobile":["?0"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"pve.example.com"}},"duration":0.012056783,"status":502,"err_id":"k7nw2x29g","err_trace":"reverseproxy.statusError (reverseproxy.go:1269)"}

3. Caddy version:

2.8.4

4. How I installed and ran Caddy:

rm -rf /usr/local/go && tar -C /usr/local -xzf go1.22.4.linux-amd64.tar.gz
xcaddy build --with github.com/caddy-dns/porkbun
sudo dpkg-divert --divert /usr/bin/caddy.default --rename /usr/bin/caddy
sudo mv ./caddy /usr/bin/caddy.custom
sudo update-alternatives --install /usr/bin/caddy caddy /usr/bin/caddy.default 10
sudo update-alternatives --install /usr/bin/caddy caddy /usr/bin/caddy.custom 50
sudo systemctl restart caddy

The original caddy installation that created the system scripts was done using Proxmox helper scripts here: Proxmox VE Helper-Scripts

But this did not include my DNS provider’s ACME plugin, so I had to recompile as above.

a. System environment:

Debian 12 LXC running on Proxmox VE

b. Command:

This just starts as a service but I can restart when needed using the command

sudo systemctl restart caddy

c. Service/unit/compose file:

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.targe

d. My complete Caddy config:

*.example.com, example.com {
        tls {
                dns porkbun {
                        api_key ""
                        api_secret_key ""
                }
        }

        @homepage host homepage.example.com
        handle @homepage {
                reverse_proxy http://192.168.1.103:3000
        }

        @pihole host pihole.example.com
        handle @pihole {
                reverse_proxy http://192.168.1.8
        }

        @ubnt host ubnt.example.com
        handle @ubnt {
                reverse_proxy https://192.168.1.6:8443 {
                        transport http {
                                tls
                                tls_insecure_skip_verify
                        }
                }
        }

        @gotify host gotify.example.com
        handle @gotify {
                reverse_proxy http://192.168.1.91
        }

        @pve host pve.example.com
        handle @pve {
                reverse_proxy https://192.168.1.7:8006 {
                        transport http {
                                tls
                        }
                }
        }

        # Fallback for otherwise unhandled domains
        handle {
                abort
        }
}

5. Links to relevant resources:

Seems like whatever certificate your upstream has doesn’t contain 192.168.1.7. What exactly is running there? What domain(s) are its certificate valid for?

You can use the tls_server_name transport option to set which domain to use for TLS-SNI, see reverse_proxy (Caddyfile directive) — Caddy Documentation

Does whatever thing is running there have an HTTP port (that doesn’t redirect HTTP->HTTPS) that you could proxy to instead? It’s preferred to proxy over HTTP because HTTPS has some overhead.

3 Likes

Seems like whatever certificate your upstream has doesn’t contain 192.168.1.7. What exactly is running there? What domain(s) are its certificate valid for?

This is the Proxmox VE host. That certificate is issued (also by porkbun, on the same domain) but to a concrete host (not a wildcard, like what I am using in Caddy). I would not expect it to have any mention of my internal network.

You can use the tls_server_name transport option to set which domain to use for TLS-SNI, see reverse_proxy (Caddyfile directive) — Caddy Documentation

Brilliant! This was exactly what I was looking for. By adding the concrete hostname inside the Caddyfile using this directive it works exactly as I intended.

Does whatever thing is running there have an HTTP port (that doesn’t redirect HTTP->HTTPS) that you could proxy to instead? It’s preferred to proxy over HTTP because HTTPS has some overhead.

No it doesn’t. But that’s ok because this is an administration interface, so I expect one user (me) to use it sparingly, with maybe some API access for heartbeats. I get your point though.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.