Forward to nested Caddy instance if not defined

Help
1

1. Caddy version (caddy version):

v2.2.1 h1:Q62GWHMtztnvyRU+KPOpw6fNfeCD3SkwH7SfT1Tgt2c=

2. How I run Caddy:

Currently running Caddy on two Linux boxes (VPS and RPI).
Both have same Caddy version but different architecture (amd64, armv7).

a. System environment:

VPS
  • Ubuntu 20.04.1 LTS
  • apt 2.0.2ubuntu0.2
  • dpkg 1.19.7ubuntu3
  • systemd 245.4-4ubuntu3.3
Raspberry Pi
  • Raspbian Buster
  • apt 1.8.2.2
  • dpkg 1.19.7
  • systemd 241-7~deb10u5+rpi1

b. Command:

# from systemd service
caddy run --environ --config /etc/caddy/Caddyfile

c. Service/unit/compose file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

VPS
{
	email bryanjhv@gmail.com
	admin off
}

rapi.ga {
	root * /usr/share/caddy
	encode zstd gzip
	file_server
}

www.rapi.ga,
www.rapi.ga:80 {
	redir https://rapi.ga{uri}
}

:80 {
	reverse_proxy 10.10.10.21:80
}
:443 {
	reverse_proxy 10.10.10.21:443
}
Raspberry Pi
{
	email bryanjhv@gmail.com
	admin off
}

blog.rapi.ga {
	root * /usr/share/caddy
	encode zstd gzip
	file_server
}

3. The problem I’m having:

I’m trying to serve many websites using Caddy. I want to serve some of them straight away from my VPS, and the others from a Raspberry Pi. I’ve setup a VPN to have connectivity between the two machines, specifically Tinc VPN server and tested it’s working (for reference, the VPS IP is 10.10.10.12 and RPI IP is 10.10.10.21), I can ping between them, and also access a server (using Python’s http.server module) without trouble. I’ve checked the firewalls and I have ports 80/443 open in VPS (and 655 for Tinc), the RPI doesn’t use a firewall.

What I want to do is:

  • Serve websites defined in VPS’ Caddyfile using that server.
  • Serve anything else (pointing at the VPS public IP) using the RPI.
  • If neither server has that site defined, maybe show a 404 or do nothing. (it might guard against others pointing their DNS at my VPS)

For now, I see exactly two problems with the way I’m approaching:

  1. Any request, when redirected to HTTPS (automatic) is serving two Server: Caddy headers.
  2. Any request to some forwarded host is not responded at all, giving an internal error in handshake I guess.

4. Error messages and/or full log output:

RPI (journalctl --no-pager -u caddy)

-- Logs begin at Mon 2020-12-28 03:42:36 -05, end at Mon 2020-12-28 23:20:24 -05. --
Dec 28 22:19:21 router systemd[1]: Started Caddy.
Dec 28 22:19:22 router caddy[1113]: caddy.HomeDir=/var/lib/caddy
Dec 28 22:19:22 router caddy[1113]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Dec 28 22:19:22 router caddy[1113]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Dec 28 22:19:22 router caddy[1113]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Dec 28 22:19:22 router caddy[1113]: caddy.Version=v2.2.1
Dec 28 22:19:22 router caddy[1113]: runtime.GOOS=linux
Dec 28 22:19:22 router caddy[1113]: runtime.GOARCH=arm
Dec 28 22:19:22 router caddy[1113]: runtime.Compiler=gc
Dec 28 22:19:22 router caddy[1113]: runtime.NumCPU=4
Dec 28 22:19:22 router caddy[1113]: runtime.GOMAXPROCS=4
Dec 28 22:19:22 router caddy[1113]: runtime.Version=go1.15.2
Dec 28 22:19:22 router caddy[1113]: os.Getwd=/
Dec 28 22:19:22 router caddy[1113]: LANG=en_GB.UTF-8
Dec 28 22:19:22 router caddy[1113]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Dec 28 22:19:22 router caddy[1113]: HOME=/var/lib/caddy
Dec 28 22:19:22 router caddy[1113]: LOGNAME=caddy
Dec 28 22:19:22 router caddy[1113]: USER=caddy
Dec 28 22:19:22 router caddy[1113]: INVOCATION_ID=3edc82d1e2654d5c81141c1f7cf60e9f
Dec 28 22:19:22 router caddy[1113]: JOURNAL_STREAM=9:20948
Dec 28 22:19:22 router caddy[1113]: {"level":"info","ts":1609211962.5299518,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Dec 28 22:19:22 router caddy[1113]: {"level":"warn","ts":1609211962.5357702,"logger":"admin","msg":"admin endpoint disabled"}
Dec 28 22:19:22 router caddy[1113]: {"level":"info","ts":1609211962.536643,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x34bb4f0"}
Dec 28 22:19:22 router caddy[1113]: {"level":"info","ts":1609211962.538742,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Dec 28 22:19:22 router caddy[1113]: {"level":"info","ts":1609211962.538833,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Dec 28 22:19:22 router caddy[1113]: {"level":"info","ts":1609211962.54021,"logger":"tls","msg":"cleaned up storage units"}
Dec 28 22:19:22 router caddy[1113]: {"level":"info","ts":1609211962.5521178,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["blog.rapi.ga"]}
Dec 28 22:19:22 router caddy[1113]: {"level":"info","ts":1609211962.5530565,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Dec 28 22:19:22 router caddy[1113]: {"level":"info","ts":1609211962.553143,"msg":"serving initial configuration"}
Dec 28 22:19:22 router caddy[1113]: {"level":"info","ts":1609211962.5542479,"logger":"tls.obtain","msg":"acquiring lock","identifier":"blog.rapi.ga"}
Dec 28 22:19:22 router caddy[1113]: {"level":"info","ts":1609211962.5663404,"logger":"tls.obtain","msg":"lock acquired","identifier":"blog.rapi.ga"}
Dec 28 22:19:24 router caddy[1113]: {"level":"info","ts":1609211964.6543214,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["blog.rapi.ga"]}
Dec 28 22:19:24 router caddy[1113]: {"level":"info","ts":1609211964.654528,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["blog.rapi.ga"]}
Dec 28 22:19:25 router caddy[1113]: {"level":"info","ts":1609211965.3287256,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"blog.rapi.ga","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Dec 28 22:19:26 router caddy[1113]: {"level":"info","ts":1609211966.2297862,"logger":"tls.issuance.acme","msg":"served key authentication","identifier":"blog.rapi.ga","challenge":"http-01","remote":"10.10.10.12:47816"}
Dec 28 22:19:26 router caddy[1113]: {"level":"info","ts":1609211966.2302723,"logger":"tls.issuance.acme","msg":"served key authentication","identifier":"blog.rapi.ga","challenge":"http-01","remote":"10.10.10.12:47814"}
Dec 28 22:19:26 router caddy[1113]: {"level":"error","ts":1609211966.7330785,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"blog.rapi.ga","challenge_type":"http-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:dns","error":"During secondary validation: No valid IP addresses found for blog.rapi.ga"}
Dec 28 22:19:26 router caddy[1113]: {"level":"error","ts":1609211966.7332833,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"blog.rapi.ga","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:dns - During secondary validation: No valid IP addresses found for blog.rapi.ga","order":"https://acme-v02.api.letsencrypt.org/acme/order/107725454/7001219327","attempt":1,"max_attempts":3}
Dec 28 22:19:28 router caddy[1113]: {"level":"info","ts":1609211968.4469044,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"blog.rapi.ga","challenge_type":"tls-alpn-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Dec 28 22:19:29 router caddy[1113]: {"level":"error","ts":1609211969.282904,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"blog.rapi.ga","challenge_type":"tls-alpn-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:tls","error":"remote error: tls: internal error"}
Dec 28 22:19:29 router caddy[1113]: {"level":"error","ts":1609211969.2831862,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"blog.rapi.ga","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:tls - remote error: tls: internal error","order":"https://acme-v02.api.letsencrypt.org/acme/order/107725454/7001219875","attempt":2,"max_attempts":3}
Dec 28 22:19:31 router caddy[1113]: {"level":"error","ts":1609211971.2602496,"logger":"tls.obtain","msg":"will retry","error":"[blog.rapi.ga] Obtain: [blog.rapi.ga] solving challenges: blog.rapi.ga: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[dns-01]) (order=https://acme-v02.api.letsencrypt.org/acme/order/107725454/7001220336) (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":8.693832228,"max_duration":2592000}
Dec 28 22:20:33 router caddy[1113]: {"level":"info","ts":1609212033.7300854,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"blog.rapi.ga","challenge_type":"tls-alpn-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Dec 28 22:20:37 router caddy[1113]: {"level":"error","ts":1609212037.7104988,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"blog.rapi.ga","challenge_type":"tls-alpn-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:tls","error":"remote error: tls: internal error"}
Dec 28 22:20:37 router caddy[1113]: {"level":"error","ts":1609212037.7106655,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"blog.rapi.ga","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:tls - remote error: tls: internal error","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/17296823/209421235","attempt":1,"max_attempts":3}
Dec 28 22:20:39 router caddy[1113]: {"level":"info","ts":1609212039.299914,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"blog.rapi.ga","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Dec 28 22:20:40 router caddy[1113]: {"level":"error","ts":1609212040.1208243,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"blog.rapi.ga","challenge_type":"http-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:tls","error":"Fetching https://blog.rapi.ga/.well-known/acme-challenge/52nWamiQB32I9CwVqPQP0fgUVQ4eKDAMCBpn8ykYJUk: remote error: tls: internal error"}
Dec 28 22:20:40 router caddy[1113]: {"level":"error","ts":1609212040.1210163,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"blog.rapi.ga","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:tls - Fetching https://blog.rapi.ga/.well-known/acme-challenge/52nWamiQB32I9CwVqPQP0fgUVQ4eKDAMCBpn8ykYJUk: remote error: tls: internal error","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/17296823/209421273","attempt":2,"max_attempts":3}
Dec 28 22:20:41 router caddy[1113]: {"level":"error","ts":1609212041.9569454,"logger":"tls.obtain","msg":"will retry","error":"[blog.rapi.ga] Obtain: [blog.rapi.ga] solving challenges: blog.rapi.ga: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/17296823/209421289) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":2,"retrying_in":120,"elapsed":79.390530278,"max_duration":2592000}
Dec 28 22:22:43 router caddy[1113]: {"level":"info","ts":1609212163.8660378,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"blog.rapi.ga","challenge_type":"tls-alpn-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Dec 28 22:22:47 router caddy[1113]: {"level":"error","ts":1609212167.4703314,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"blog.rapi.ga","challenge_type":"tls-alpn-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:dns","error":"No valid IP addresses found for blog.rapi.ga"}
Dec 28 22:22:47 router caddy[1113]: {"level":"error","ts":1609212167.4706006,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"blog.rapi.ga","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:dns - No valid IP addresses found for blog.rapi.ga","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/17296823/209422018","attempt":1,"max_attempts":3}
Dec 28 22:22:48 router caddy[1113]: {"level":"info","ts":1609212168.970493,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"blog.rapi.ga","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Dec 28 22:22:49 router caddy[1113]: {"level":"error","ts":1609212169.7028062,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"blog.rapi.ga","challenge_type":"http-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:dns","error":"No valid IP addresses found for blog.rapi.ga"}
Dec 28 22:22:49 router caddy[1113]: {"level":"error","ts":1609212169.7030396,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"blog.rapi.ga","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:dns - No valid IP addresses found for blog.rapi.ga","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/17296823/209422051","attempt":2,"max_attempts":3}
Dec 28 22:22:51 router caddy[1113]: {"level":"error","ts":1609212171.4034579,"logger":"tls.obtain","msg":"will retry","error":"[blog.rapi.ga] Obtain: [blog.rapi.ga] solving challenges: blog.rapi.ga: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/17296823/209422070) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":3,"retrying_in":120,"elapsed":208.837042444,"max_duration":2592000}
Dec 28 22:24:52 router caddy[1113]: {"level":"info","ts":1609212292.8210285,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"blog.rapi.ga","challenge_type":"tls-alpn-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Dec 28 22:24:53 router caddy[1113]: {"level":"error","ts":1609212293.5286927,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"blog.rapi.ga","challenge_type":"tls-alpn-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:dns","error":"No valid IP addresses found for blog.rapi.ga"}
Dec 28 22:24:53 router caddy[1113]: {"level":"error","ts":1609212293.5288718,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"blog.rapi.ga","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:dns - No valid IP addresses found for blog.rapi.ga","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/17296823/209422835","attempt":1,"max_attempts":3}
Dec 28 22:24:55 router caddy[1113]: {"level":"info","ts":1609212295.0011096,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"blog.rapi.ga","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Dec 28 22:24:55 router caddy[1113]: {"level":"error","ts":1609212295.6888986,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"blog.rapi.ga","challenge_type":"http-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:dns","error":"No valid IP addresses found for blog.rapi.ga"}
Dec 28 22:24:55 router caddy[1113]: {"level":"error","ts":1609212295.6890812,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"blog.rapi.ga","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:dns - No valid IP addresses found for blog.rapi.ga","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/17296823/209422846","attempt":2,"max_attempts":3}
Dec 28 22:24:57 router caddy[1113]: {"level":"error","ts":1609212297.422211,"logger":"tls.obtain","msg":"will retry","error":"[blog.rapi.ga] Obtain: [blog.rapi.ga] solving challenges: blog.rapi.ga: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/17296823/209422849) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":4,"retrying_in":300,"elapsed":334.855793761,"max_duration":2592000}
Dec 28 22:29:59 router caddy[1113]: {"level":"info","ts":1609212599.140955,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"blog.rapi.ga","challenge_type":"tls-alpn-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Dec 28 22:29:59 router caddy[1113]: {"level":"error","ts":1609212599.946713,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"blog.rapi.ga","challenge_type":"tls-alpn-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:tls","error":"remote error: tls: internal error"}
Dec 28 22:29:59 router caddy[1113]: {"level":"error","ts":1609212599.9479592,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"blog.rapi.ga","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:tls - remote error: tls: internal error","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/17296823/209424635","attempt":1,"max_attempts":3}
Dec 28 22:30:01 router caddy[1113]: {"level":"info","ts":1609212601.5566218,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"blog.rapi.ga","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Dec 28 22:30:02 router caddy[1113]: {"level":"error","ts":1609212602.8869221,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"blog.rapi.ga","challenge_type":"http-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:tls","error":"Fetching https://blog.rapi.ga/.well-known/acme-challenge/XSwfEtAnc5XnnQDpkmE6YKFgta-aK1GwJnuWJJK7U88: remote error: tls: internal error"}
Dec 28 22:30:02 router caddy[1113]: {"level":"error","ts":1609212602.8898282,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"blog.rapi.ga","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:tls - Fetching https://blog.rapi.ga/.well-known/acme-challenge/XSwfEtAnc5XnnQDpkmE6YKFgta-aK1GwJnuWJJK7U88: remote error: tls: internal error","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/17296823/209424647","attempt":2,"max_attempts":3}
Dec 28 22:30:04 router caddy[1113]: {"level":"error","ts":1609212604.7523253,"logger":"tls.obtain","msg":"will retry","error":"[blog.rapi.ga] Obtain: [blog.rapi.ga] solving challenges: blog.rapi.ga: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/17296823/209424689) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":5,"retrying_in":600,"elapsed":642.185909989,"max_duration":2592000}
Dec 28 22:40:06 router caddy[1113]: {"level":"info","ts":1609213206.180286,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"blog.rapi.ga","challenge_type":"tls-alpn-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Dec 28 22:40:06 router caddy[1113]: {"level":"error","ts":1609213206.902358,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"blog.rapi.ga","challenge_type":"tls-alpn-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:tls","error":"remote error: tls: internal error"}
Dec 28 22:40:06 router caddy[1113]: {"level":"error","ts":1609213206.9038432,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"blog.rapi.ga","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:tls - remote error: tls: internal error","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/17296823/209428746","attempt":1,"max_attempts":3}
Dec 28 22:40:08 router caddy[1113]: {"level":"info","ts":1609213208.348413,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"blog.rapi.ga","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Dec 28 22:40:09 router caddy[1113]: {"level":"error","ts":1609213209.8673673,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"blog.rapi.ga","challenge_type":"http-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:tls","error":"Fetching https://blog.rapi.ga/.well-known/acme-challenge/JxyNLwfSxbbASWAL0akIobeX1zvCMJtgTTYV9W7bPNQ: remote error: tls: internal error"}
Dec 28 22:40:09 router caddy[1113]: {"level":"error","ts":1609213209.867592,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"blog.rapi.ga","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:tls - Fetching https://blog.rapi.ga/.well-known/acme-challenge/JxyNLwfSxbbASWAL0akIobeX1zvCMJtgTTYV9W7bPNQ: remote error: tls: internal error","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/17296823/209428755","attempt":2,"max_attempts":3}
Dec 28 22:40:11 router caddy[1113]: {"level":"error","ts":1609213211.565381,"logger":"tls.obtain","msg":"will retry","error":"[blog.rapi.ga] Obtain: [blog.rapi.ga] solving challenges: blog.rapi.ga: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/17296823/209428769) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":6,"retrying_in":1200,"elapsed":1248.998964158,"max_duration":2592000}
Dec 28 23:00:13 router caddy[1113]: {"level":"info","ts":1609214413.254681,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"blog.rapi.ga","challenge_type":"tls-alpn-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Dec 28 23:00:14 router caddy[1113]: {"level":"error","ts":1609214414.0136423,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"blog.rapi.ga","challenge_type":"tls-alpn-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:dns","error":"No valid IP addresses found for blog.rapi.ga"}
Dec 28 23:00:14 router caddy[1113]: {"level":"error","ts":1609214414.0138094,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"blog.rapi.ga","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:dns - No valid IP addresses found for blog.rapi.ga","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/17296823/209435245","attempt":1,"max_attempts":3}
Dec 28 23:00:15 router caddy[1113]: {"level":"info","ts":1609214415.5303009,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"blog.rapi.ga","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Dec 28 23:00:16 router caddy[1113]: {"level":"error","ts":1609214416.250094,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"blog.rapi.ga","challenge_type":"http-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:dns","error":"No valid IP addresses found for blog.rapi.ga"}
Dec 28 23:00:16 router caddy[1113]: {"level":"error","ts":1609214416.250276,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"blog.rapi.ga","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:dns - No valid IP addresses found for blog.rapi.ga","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/17296823/209435263","attempt":2,"max_attempts":3}
Dec 28 23:00:17 router caddy[1113]: {"level":"error","ts":1609214417.9022732,"logger":"tls.obtain","msg":"will retry","error":"[blog.rapi.ga] Obtain: [blog.rapi.ga] solving challenges: blog.rapi.ga: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/17296823/209435272) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":7,"retrying_in":1200,"elapsed":2455.335815921,"max_duration":2592000}
Dec 28 23:20:19 router caddy[1113]: {"level":"info","ts":1609215619.5627437,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"blog.rapi.ga","challenge_type":"tls-alpn-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Dec 28 23:20:20 router caddy[1113]: {"level":"error","ts":1609215620.2917356,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"blog.rapi.ga","challenge_type":"tls-alpn-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:dns","error":"No valid IP addresses found for blog.rapi.ga"}
Dec 28 23:20:20 router caddy[1113]: {"level":"error","ts":1609215620.2919166,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"blog.rapi.ga","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:dns - No valid IP addresses found for blog.rapi.ga","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/17296823/209441838","attempt":1,"max_attempts":3}
Dec 28 23:20:21 router caddy[1113]: {"level":"info","ts":1609215621.7386854,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"blog.rapi.ga","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Dec 28 23:20:22 router caddy[1113]: {"level":"error","ts":1609215622.44381,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"blog.rapi.ga","challenge_type":"http-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:dns","error":"No valid IP addresses found for blog.rapi.ga"}
Dec 28 23:20:22 router caddy[1113]: {"level":"error","ts":1609215622.4440396,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"blog.rapi.ga","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:dns - No valid IP addresses found for blog.rapi.ga","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/17296823/209441846","attempt":2,"max_attempts":3}
Dec 28 23:20:24 router caddy[1113]: {"level":"error","ts":1609215624.1239398,"logger":"tls.obtain","msg":"will retry","error":"[blog.rapi.ga] Obtain: [blog.rapi.ga] solving challenges: blog.rapi.ga: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/17296823/209441852) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":8,"retrying_in":1800,"elapsed":3661.557525334,"max_duration":2592000}

CURL (curl -Li blog.rapi.ga)

HTTP/1.1 308 Permanent Redirect
Content-Length: 0
Date: Tue, 29 Dec 2020 04:26:45 GMT
Location: https://blog.rapi.ga/
Server: Caddy
Server: Caddy

curl: (35) error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error

5. What I already tried:

At first I set-up IP forwarding on the VPS and then created some iptables rules to forward any traffic on ports 80/443 to the RPI (running Caddy), and it went smooth (that makes me think there’s no problem with the RPI serving or running Caddy, neither a problem with the VPN or something networking-related). Following is the config I used:

That doesn’t work to me because I want to be able to also serve sites from the VPS (it is faster), and give to the RPI everything else.

I read some posts where it says that might be a problem with SSL termination (something along the lines “the proxy should let the endpoint do the SSL termination so there are no SSL errors at all”).

6. Links to relevant resources:

This post says something about SSL termination:

I used this guide as reference to do the iptables-related stuff previously:

Thanks in advance for your great help!
I’m relatively new to Caddy.

2

First of all, be aware that if you turn off the admin endpoint with admin off, then you’re also turning off the ability to gracefully reload Caddy with sudo systemctl reload caddy, because the admin API is what’s used to trigger the reload. Do you have any specific reason for turning it off?

I’m not entirely sure I understand what you’re trying to achieve here with your setup. Seems like a strange approach to me.

There’s a few things going on here.

So when you request blog.rapi.ga, you’re hitting your VPS first, on port 80. That makes Caddy proxy the request to your RPI, also on port 80. That one sees the Host as blog.rapi.ga, so it matches that request. By default, Caddy has HTTP->HTTPS redirects enabled (unless overriden by a :80 site, which your VPS Caddy does, so it will not do any HTTP->HTTPS redirects on its own). So the RPI instance will respond with an HTTP 308 redirect response, and add the Server: Caddy header. The VPS instance then receives that response, and tacks on its own Server: Caddy header as well (which is expected and correct behaviour for this situation, no problem there).

Then, curl follows the redirect because you specified -L, and makes the request https://blog.rapi.ga/, which hits your VPS. But your VPS is not configured with a TLS certificate for that domain, so the TLS handshake fails.

Caddy can’t serve HTTPS without having a certificate that matches the domain on the request. It also can’t just blindly proxy the TCP traffic, because it’s an HTTP proxy, not a TCP proxy. (But if you really need to proxy raw TCP, you could use GitHub - mholt/caddy-l4: Layer 4 (TCP/UDP) app for Caddy but that’s definitely overcomplicating things here, so don’t do that :sweat_smile:)

If you were to have a certificate for blog.rapi.ga on your VPS, then it would be able to accept the connection, then proxy the request. But even if that happened, it would then try to proxy to 10.10.10.21:443 with the Host header set to blog.rapi.ga, which means it would try to make a new TLS connection to your RPI instance, requiring it to also have a certificate.

Problem is, the way ACME works, with the HTTP or ALPN challenges, only the first publicly accessible server would be able to manage a certificate for that domain. It wouldn’t be possible for some proxied server to also manage a certificate for that domain. It would be possible though, if you used the DNS challenge (requiring a plugin, and API credentials for your DNS provider).

So yeah. Like I said, there’s a lot going on here.

What are you actually trying to do? It doesn’t seem like a good idea. I would just put every site you actually want to serve on the VPS and call it a day. But I figure you probably have a good reason for trying to complicate things this much :sweat_smile:

3

I was aware of that after reading the documentation. I use systemctl restart caddy. The reason I’ve disabled that is because I somewhat care about security, and prefer a 1-sec downtime than a longer one. I mean, I make sure to review what I upload but some NPM/Composer/etc dependency could easily connect to the Caddy API which is unauthenticated as of now, not the fine-grained control I’d like to have.

Thank you so much for the explanation on what’s going on behind the scenes, I didn’t really understand the way reverse proxies would work with TLS enabled (I’ve worked just with HTTP targets and Nginx as HTTPS server). So, now I do know that a :443 request to the RPI would trigger another handshake.

From what I’ve read in the previously linked #wiki article, the duplicated certificates problem would be solved by using a storage adapter, which I could handle using SSHFS to mount RPI’s /var/lib/caddy/.local/share/certificates/... into the VPS, so no problem there, or a bind mount or something.

I also checked out L4’s code and purpose, and it seems like a solution to me, at least an easier one to manage than creating a bunch of sensible iptables rules. Might give it a try to check if it solves this problem.

I was thinking of using DNS challenge with Cloudflare or Linode/DigitalOcean, that’s the way I managed to work out some things before (to generate a wildcard certificate, mostly). But I’m guessing that two certificates for the same domain would be generated, both valid, so it would bring more problems to debug some things in case they go wrong.

Maybe something like Cloudflare’s Origin CAs would work here, if I’d want to use HTTPS anyway on the RPI, which I don’t think would be a good idea as of now. Using Caddy as just an application manager would achieve what I want if I’m going to be explicit about the domains the RPI will serve (defined in the VPS’ Caddyfile).

Well. What I’m trying to do is the following:

  • I want to use a VPS to run some services on, let’s say, domain-a.com. This VPS has a public IP, and now it’s using Nginx but want to switch to Caddy.
  • I also want to point domain-b.com to same VPS, but serve anything that’s not defined in the VPS using the RPI, there will be many more domains (so it would make me write them all and have two places to manage configuration instead of just one each time I want to create/delete an endpoint/domain).
  • The VPS is a cheap one, so it will have at most 20-25GB of storage, which could easily run out of resources for storing so much code for apps (MinIO, OwnCloud, some Node.js apps, etc).
  • I have that RPI so I’d want to use that for services which are to be accessed by just me from time to time (i.e. not crucial or in need to be fast/performant).

So, would the solution to this problem to just define the domains statically (not :80, :443), and don’t use HTTPS at all on the RPI?

4

What does this mean? What do you mean by “defined in the VPS”?

I have to admit I’m a little lost. (My expertise is helping people with Caddy; architecting systems sounds more like something you should hire a consultant for… would probably be more helpful than random Internet users like me trying to help, heh.)

5

By “defined in the VPS” I mean any domain listed explicitly in the VPS’ Caddyfile. For example, given the Caddyfile above, “defined” domains/URLs would be rapi.ga and www.rapi.ga (:443 and :80). Anything that’s different than that, would be forwarded to the RPI (that’s what I’m trying to do by creating rules for :80 and :443 alone).

But, like you said, this sounds more like a networking/architecting issue than something Caddy would solve (I’m gonna need something like Nginx’s stream {} blocks or your L4 module in order to forward raw TCP sockets, and the setup might be a lot more complex and far away from the purpose of any HTTP server).

For now I’ll try to keep working this out and see if I can come up with something, even if it isn’t a good idea (read somewhere that one of Caddy’s goal is to be clear/explicit about what’s going on in its configuration).
In case I get to achieve something, will be posting here. Gonna mark @francislavoie answer for now because his provided info oriented me a little more. Thanks, really!

P.S.: I don’t think you guys are “random Internet users”, because you created and maintain Caddy, which by the way I admire a lot, thank you for all of your hard work.

6

This topic was automatically closed after 30 days. New replies are no longer allowed.