Problems with SSL and IPv6 reverse proxy

Help
1

1. Caddy version (caddy version):

v2.4.6 h1:HGkGICFGvyrodcqOOclHKfvJC0qTU7vny/7FhYp9hNw=

2. How I run Caddy:

a. System environment:

Debian 10, systemd enabled, no docker - Caddy installed with official non-Docker instructions for Debian

b. Command:

sudo systemctl start caddy

c. Service/unit/compose file:

The default systemd service file from /lib/systemd/system/caddy.service:

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

Caddyfile for server #1:

# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace ":80" below with your
# domain name.
{
	debug
}

https://hc.klo.ink {
	reverse_proxy 127.0.0.1:8000
}

https://hc2.klo.ink {
	reverse_proxy [2001:470:1f07:4d5:123::]:80
}

Caddyfile for server #2:

# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace ":80" below with your
# domain name.
{
	debug
}

:80 {
	respond "Hello there"
}

3. The problem I’m having:

I am running Caddy on two identical virtual servers, both on Debian 10. Both are NAT VPS with only limited IPv4 port ranges available to me so I am using Cloudflare to proxy my subdomains to the servers’ IPv6 addresses. On Cloudflare I have the “Full” SSL/TSL Encryption Mode on and two subdomains, hc and hc2, which are both AAAA records pointing to server #1’s IPv6 address with the “orange cloud” proxy on.

The reverse proxy for subdomain hc is working great as it points to a local application on server #1.

The subdomain hc2 points to server #2’s IPv6 address at port 80 where another Caddy instance is running, and I am only getting 502 Bad Gateway trying to access it through the h2 subdomain. I have also tried serving some actual files instead of just respond but it changed nothing. I believe this is a problem with HTTPS but I have tried different configurations for hours without any results.

So how can I make both subdomains work over HTTPS when the hc2 subdomain is served unencrypted HTTP content from an IPv6 address on an another server?

4. Error messages and/or full log output:

Journal logs for the Caddy service on server #1:

Jan 02 22:47:08 hippa caddy[1207]: {"level":"info","ts":1641163628.6994066,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_addr":"127.0.0.1:56896","headers":{"Accept-Encoding":["gzip"],"Content-Length":["489"],"Content-Type":["application/json"],"Origin":["localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
Jan 02 22:47:08 hippa caddy[1207]: {"level":"info","ts":1641163628.6996996,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Jan 02 22:47:08 hippa caddy[1207]: {"level":"info","ts":1641163628.6997654,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Jan 02 22:47:08 hippa caddy[1207]: {"level":"info","ts":1641163628.6997743,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Jan 02 22:47:08 hippa caddy[1207]: {"level":"debug","ts":1641163628.6998854,"logger":"http","msg":"starting server loop","address":"[::]:443","http3":false,"tls":true}
Jan 02 22:47:08 hippa caddy[1207]: {"level":"debug","ts":1641163628.6998932,"logger":"http","msg":"starting server loop","address":"[::]:80","http3":false,"tls":false}
Jan 02 22:47:08 hippa caddy[1207]: {"level":"info","ts":1641163628.6998963,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["hc.klo.ink","hc2.klo.ink"]}
Jan 02 22:47:08 hippa caddy[1207]: {"level":"debug","ts":1641163628.7082253,"logger":"tls","msg":"loading managed certificate","domain":"hc.klo.ink","expiration":1648924536,"issuer_key":"acme-v02.api.letsencrypt.org-directory","storage":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Jan 02 22:47:08 hippa caddy[1207]: {"level":"debug","ts":1641163628.7084117,"logger":"tls.cache","msg":"added certificate to cache","subjects":["hc.klo.ink"],"expiration":1648924536,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"dd80abf01626ce658b941d98198d67cbb99dc733a2ad94516821d21e0c6650f9","cache_size":1,"cache_capacity":10000}
Jan 02 22:47:08 hippa caddy[1207]: {"level":"debug","ts":1641163628.7085612,"logger":"tls","msg":"loading managed certificate","domain":"hc2.klo.ink","expiration":1648943999,"issuer_key":"acme.zerossl.com-v2-DV90","storage":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Jan 02 22:47:08 hippa caddy[1207]: {"level":"debug","ts":1641163628.7086964,"logger":"tls.cache","msg":"added certificate to cache","subjects":["hc2.klo.ink"],"expiration":1648943999,"managed":true,"issuer_key":"acme.zerossl.com-v2-DV90","hash":"5aeeed5e12aae3bae5c99515028f3097cebb7aaf63a2269b661eb8eaa1ae05f8","cache_size":2,"cache_capacity":10000}
Jan 02 22:47:08 hippa caddy[1207]: {"level":"info","ts":1641163628.7087462,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0001a84d0"}
Jan 02 22:47:08 hippa caddy[1207]: {"level":"info","ts":1641163628.7109673,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0000f4540"}
Jan 02 22:47:08 hippa caddy[1207]: {"level":"info","ts":1641163628.7157412,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Jan 02 22:47:08 hippa caddy[1207]: {"level":"info","ts":1641163628.7157743,"logger":"admin.api","msg":"load complete"}
Jan 02 22:47:08 hippa caddy[1207]: {"level":"info","ts":1641163628.7209148,"logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}

Journal logs for the Caddy service on server #2:

Jan 02 22:55:33 vinyl systemd[1]: Starting Caddy...
Jan 02 22:55:33 vinyl caddy[12241]: caddy.HomeDir=/var/lib/caddy
Jan 02 22:55:33 vinyl caddy[12241]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Jan 02 22:55:33 vinyl caddy[12241]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Jan 02 22:55:33 vinyl caddy[12241]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Jan 02 22:55:33 vinyl caddy[12241]: caddy.Version=v2.4.6 h1:HGkGICFGvyrodcqOOclHKfvJC0qTU7vny/7FhYp9hNw=
Jan 02 22:55:33 vinyl caddy[12241]: runtime.GOOS=linux
Jan 02 22:55:33 vinyl caddy[12241]: runtime.GOARCH=amd64
Jan 02 22:55:33 vinyl caddy[12241]: runtime.Compiler=gc
Jan 02 22:55:33 vinyl caddy[12241]: runtime.NumCPU=1
Jan 02 22:55:33 vinyl caddy[12241]: runtime.GOMAXPROCS=1
Jan 02 22:55:33 vinyl caddy[12241]: runtime.Version=go1.17.2
Jan 02 22:55:33 vinyl caddy[12241]: os.Getwd=/
Jan 02 22:55:33 vinyl caddy[12241]: LANG=C.UTF-8
Jan 02 22:55:33 vinyl caddy[12241]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Jan 02 22:55:33 vinyl caddy[12241]: NOTIFY_SOCKET=/run/systemd/notify
Jan 02 22:55:33 vinyl caddy[12241]: HOME=/var/lib/caddy
Jan 02 22:55:33 vinyl caddy[12241]: LOGNAME=caddy
Jan 02 22:55:33 vinyl caddy[12241]: USER=caddy
Jan 02 22:55:33 vinyl caddy[12241]: INVOCATION_ID=f98192ebbe7d44498802680d68505c63
Jan 02 22:55:33 vinyl caddy[12241]: JOURNAL_STREAM=7:585638051
Jan 02 22:55:33 vinyl caddy[12241]: {"level":"info","ts":1641164133.8453805,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Jan 02 22:55:33 vinyl caddy[12241]: {"level":"info","ts":1641164133.849394,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Jan 02 22:55:33 vinyl caddy[12241]: {"level":"info","ts":1641164133.8501492,"logger":"http","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv0","http_port":80}
Jan 02 22:55:33 vinyl caddy[12241]: {"level":"debug","ts":1641164133.850857,"logger":"http","msg":"starting server loop","address":"[::]:80","http3":false,"tls":false}
Jan 02 22:55:33 vinyl caddy[12241]: {"level":"info","ts":1641164133.8535688,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00027cd20"}
Jan 02 22:55:33 vinyl caddy[12241]: {"level":"info","ts":1641164133.8537667,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Jan 02 22:55:33 vinyl caddy[12241]: {"level":"info","ts":1641164133.8575718,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Jan 02 22:55:33 vinyl caddy[12241]: {"level":"info","ts":1641164133.8576267,"msg":"serving initial configuration"}
Jan 02 22:55:33 vinyl systemd[1]: Started Caddy.
Jan 02 22:55:33 vinyl caddy[12241]: {"level":"info","ts":1641164133.9303508,"logger":"tls","msg":"finished cleaning storage units"}

5. What I already tried:

I have shuffled the Cloudflare SSL/TLS Encryption Mode between Full and Flexible, tried to access both http and https addresses for hc2.klo.ink, served static files with file_server instead of respond on server #2, tried to switch the server roles by pointing the subdomains to server #2 and proxying from there to server #1 (after which I had the same problem for the server #1 application) and a lot of other things I don’t even remember after four hours.

I also tried pointing the hc2 subdomain straight to server #2 in order to separate the servers and have each handle their own subdomain by modifying the Caddyfiles accordingly. But then Caddy could not retrieve a certificate for hc2 at all when server #1 was already running and working well.

6. Links to relevant resources:

2

I’m not seeing evidence that a request even reaches Caddy in your logs :thinking:

I’m not sure I follow what’s going on here.

3

Well the ports should be open on both servers and when I tried it before, if I pointed a subdomain straight at server #2 without modifying the Caddyfile, the HTTP page would load properly without encryption. But if I changed the :80 in the Caddyfile to the actual subdomain, i.e. to HTTPS, it would no longer work and the Cloudflare page just told me that “The host cannot be reached”. I was expecting automatic certificate retrieval by Caddy and a translation of the HTTP content to HTTPS.

Are there caveats to using Caddy’s automatic HTTPS for different subdomains of the same domain? Each should be getting their individual certificate, right?

I can’t understand why the approach below didn’t work either (which is the reason why I ended up trying the solution described above):

  • both servers have their own, separate Caddyfiles which only contain one site with its subdomain and the reverse_proxy directive to its local content
  • the subdomains on Cloudflare point separately to the IPv6 addresses of the servers, hc.klo.ink to #1 and hc2.klo.ink to #2
  • only the first Caddy instance to go up works while according to the logs the other one stayed in a loop trying and failing to solve the certificate challenges - like the Caddy certificate from the other server was somehow completely blocking it
  • no wildcard domains in the Caddyfiles either
4

This topic was automatically closed after 30 days. New replies are no longer allowed.