1. The problem I’m having:
For domain say liveliteandwell.com I wrote a react.js + node.js frontend and backend. Backend starts a HTTPS server using a domain rp-tailscale.esco.ghaar on 8443 - which is not a valid domain on normal internet. Backend handles following paths: rp-tailscale.esco.ghaar/api/. I’m able to secure it (ala start HTTPS) using an internal root ca maintained via step-ca running on a VM on local network. Frontend runs securely (using same certificate) on port 3000. Direct access to rp-tailscale.esco.ghaar:3000 or rp-tailscale.esco.ghaar:8443 to verify functionality works like a charm.
Now comes caddy. On the same machine which runs backend+frontend, I installed Caddy. The idea was to have caddy host liveliteandwell.com (valid domain) for the rest of the world while fronting both the the front- and back-end to serve appropriate content.
Caddyfile has a very basic configuration comprising of one domain, followed by a single reverse_proxy directive (shown below) to test reverse proxying up to backend. But, it does not work.
When employing normal browser, postman or curl:
- URL:
https://liveliteandwell.com: 404 - URL:
https://liveliteandwell.com/api: 404
1 behaves as expected (because Caddyfile does not yet specify frontend configs). 2 should present a different msg (Route is valid). This tells me that backend is not being contacted by caddy. Backend logs confirm it. Since all of this happening in the same VM, there should be no need for traffic to leave the VM; even if it did, there is no routing or fw issue (tested extensively).
I’m sure its something basic, and being new to caddy its an RTFM problem. I’ve read local HTTPS TLS and other related wiki articles and posts. Trying suggested combinations of transport headers etc have not helped.
2. Error messages and/or full log output:
curl output for scenario#1 above
$ curl -vL https://liveliteandwell.com
* Trying 192.168.100.17:443...
* Connected to liveliteandwell.com (192.168.100.17) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/ssl/cert.pem
* CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256
* ALPN: server accepted h2
* Server certificate:
* subject: CN=liveliteandwell.com
* start date: Jun 17 00:50:29 2024 GMT
* expire date: Sep 15 00:50:28 2024 GMT
* subjectAltName: host "liveliteandwell.com" matched cert's "liveliteandwell.com"
* issuer: C=US; O=Let's Encrypt; CN=E6
* SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://liveliteandwell.com/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: liveliteandwell.com]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: liveliteandwell.com
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/2 404
< alt-svc: h3=":443"; ma=2592000
< server: Caddy
< content-length: 0
< date: Mon, 17 Jun 2024 03:55:47 GMT
<
* Connection #0 to host liveliteandwell.com left intact
caddy log for scenario#1
Jun 16 20:55:47 rp-tailscale caddy[14721]: {"level":"debug","ts":1718596547.7198305,"logger":"events","msg":"event","name":"tls_get_certificate","id":"a6deb7c6-673f-4d38-8527-bf1bc3784ac7","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49200,49196,49192,49188,49172,49162,159,107,57,52393,52392,52394,65413,196,136,129,157,61,53,192,132,49199,49195,49191,49187,49171,49161,158,103,51,190,69,156,60,47,186,65,49169,49159,5,4,49170,49160,22,10,255],"ServerName":"liveliteandwell.com","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2054,1537,1539,2053,1281,1283,2052,1025,1027,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"192.168.0.28","Port":54489,"Zone":""},"LocalAddr":{"IP":"192.168.100.17","Port":443,"Zone":""}}}}
Jun 16 20:55:47 rp-tailscale caddy[14721]: {"level":"debug","ts":1718596547.7199287,"logger":"tls.handshake","msg":"choosing certificate","identifier":"liveliteandwell.com","num_choices":1}
Jun 16 20:55:47 rp-tailscale caddy[14721]: {"level":"debug","ts":1718596547.7199898,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"liveliteandwell.com","subjects":["liveliteandwell.com"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"8e02fc2b7fd7fdf007881e7004fe00a66666501929f409bdc8344a0ab77e6baf"}
Jun 16 20:55:47 rp-tailscale caddy[14721]: {"level":"debug","ts":1718596547.720008,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"192.168.0.28","remote_port":"54489","subjects":["liveliteandwell.com"],"managed":true,"expiration":1726361429,"hash":"8e02fc2b7fd7fdf007881e7004fe00a66666501929f409bdc8344a0ab77e6baf"}
Jun 16 20:55:47 rp-tailscale caddy[14721]: {"level":"info","ts":1718596547.7375457,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"192.168.0.28","remote_port":"54489","client_ip":"192.168.0.28","proto":"HTTP/2.0","method":"GET","host":"liveliteandwell.com","uri":"/","headers":{"User-Agent":["curl/8.4.0"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"liveliteandwell.com"}},"bytes_read":0,"user_id":"","duration":0.000031043,"size":0,"status":404,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Content-Type":[]}}
curl log for scenario#2
$ curl -vL https://liveliteandwell.com/api
* Trying 192.168.100.17:443...
* Connected to liveliteandwell.com (192.168.100.17) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/ssl/cert.pem
* CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256
* ALPN: server accepted h2
* Server certificate:
* subject: CN=liveliteandwell.com
* start date: Jun 17 00:50:29 2024 GMT
* expire date: Sep 15 00:50:28 2024 GMT
* subjectAltName: host "liveliteandwell.com" matched cert's "liveliteandwell.com"
* issuer: C=US; O=Let's Encrypt; CN=E6
* SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://liveliteandwell.com/api
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: liveliteandwell.com]
* [HTTP/2] [1] [:path: /api]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
> GET /api HTTP/2
> Host: liveliteandwell.com
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/2 404
< alt-svc: h3=":443"; ma=2592000
< server: Caddy
< content-length: 0
< date: Mon, 17 Jun 2024 03:57:35 GMT
<
* Connection #0 to host liveliteandwell.com left intact
caddy log for scenario#2
Jun 16 20:57:35 rp-tailscale caddy[14721]: {"level":"debug","ts":1718596655.7855647,"logger":"events","msg":"event","name":"tls_get_certificate","id":"73f1da52-a9d2-43e7-9d82-8fb909437a36","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49200,49196,49192,49188,49172,49162,159,107,57,52393,52392,52394,65413,196,136,129,157,61,53,192,132,49199,49195,49191,49187,49171,49161,158,103,51,190,69,156,60,47,186,65,49169,49159,5,4,49170,49160,22,10,255],"ServerName":"liveliteandwell.com","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2054,1537,1539,2053,1281,1283,2052,1025,1027,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"192.168.0.28","Port":54523,"Zone":""},"LocalAddr":{"IP":"192.168.100.17","Port":443,"Zone":""}}}}
Jun 16 20:57:35 rp-tailscale caddy[14721]: {"level":"debug","ts":1718596655.785646,"logger":"tls.handshake","msg":"choosing certificate","identifier":"liveliteandwell.com","num_choices":1}
Jun 16 20:57:35 rp-tailscale caddy[14721]: {"level":"debug","ts":1718596655.785678,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"liveliteandwell.com","subjects":["liveliteandwell.com"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"8e02fc2b7fd7fdf007881e7004fe00a66666501929f409bdc8344a0ab77e6baf"}
Jun 16 20:57:35 rp-tailscale caddy[14721]: {"level":"debug","ts":1718596655.7856965,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"192.168.0.28","remote_port":"54523","subjects":["liveliteandwell.com"],"managed":true,"expiration":1726361429,"hash":"8e02fc2b7fd7fdf007881e7004fe00a66666501929f409bdc8344a0ab77e6baf"}
Jun 16 20:57:35 rp-tailscale caddy[14721]: {"level":"info","ts":1718596655.8055096,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"192.168.0.28","remote_port":"54523","client_ip":"192.168.0.28","proto":"HTTP/2.0","method":"GET","host":"liveliteandwell.com","uri":"/api","headers":{"User-Agent":["curl/8.4.0"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"liveliteandwell.com"}},"bytes_read":0,"user_id":"","duration":0.000025911,"size":0,"status":404,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Content-Type":[]}}
internal domain certificate validity confirmation
$ sudo certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
Certificate Name: rp-tailscale.esco.ghaar
Serial Number: <redacted.. but its valid>
Key Type: RSA
Domains: rp-tailscale.esco.ghaar
Expiry Date: <redacted.. but its valid>
Certificate Path: /etc/letsencrypt/live/rp-tailscale.esco.ghaar/fullchain.pem
Private Key Path: /etc/letsencrypt/live/rp-tailscale.esco.ghaar/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
curl log for backend access directly via internal domain (rp-tailscale.esco.ghaar) for scenario#1
$ curl -vkL https://rp-tailscale.esco.ghaar:8443
* Trying 192.168.100.17:8443...
* Connected to rp-tailscale.esco.ghaar (192.168.100.17) port 8443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted http/1.1
* Server certificate:
* subject: [NONE]
* start date: Jan 29 16:44:18 2024 GMT
* expire date: Jan 26 16:45:18 2034 GMT
* issuer: O=Escoghaar; CN=Escoghaar Intermediate CA
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/1.1
> GET / HTTP/1.1
> Host: rp-tailscale.esco.ghaar:8443
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Access-Control-Allow-Origin: *
< Content-Security-Policy: default-src 'self';base-uri 'self';font-src 'self' https: data:;form-action 'self';frame-ancestors 'self';img-src 'self' data:;object-src 'none';script-src 'self';script-src-attr 'none';style-src 'self' https: 'unsafe-inline';upgrade-insecure-requests
< Cross-Origin-Opener-Policy: same-origin
< Cross-Origin-Resource-Policy: same-origin
< Origin-Agent-Cluster: ?1
< Referrer-Policy: no-referrer
< Strict-Transport-Security: max-age=15552000; includeSubDomains
< X-Content-Type-Options: nosniff
< X-DNS-Prefetch-Control: off
< X-Download-Options: noopen
< X-Frame-Options: SAMEORIGIN
< X-Permitted-Cross-Domain-Policies: none
< X-XSS-Protection: 0
< Content-Type: application/json; charset=utf-8
< Content-Length: 26
< ETag: W/"1a-8zibdJXT1RIHjViPh/9pjQz5GqI"
< Date: Mon, 17 Jun 2024 04:00:56 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
<
* Connection #0 to host rp-tailscale.esco.ghaar left intact
{"msg":"Route not found."}
curl log for backend access directly via internal domain (rp-tailscale.esco.ghaar) for scenario#2
$ curl -vkL https://rp-tailscale.esco.ghaar:8443/api
* Trying 192.168.100.17:8443...
* Connected to rp-tailscale.esco.ghaar (192.168.100.17) port 8443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted http/1.1
* Server certificate:
* subject: [NONE]
* start date: Jan 29 16:44:18 2024 GMT
* expire date: Jan 26 16:45:18 2034 GMT
* issuer: O=Escoghaar; CN=Escoghaar Intermediate CA
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/1.1
> GET /api HTTP/1.1
> Host: rp-tailscale.esco.ghaar:8443
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Access-Control-Allow-Origin: *
< Content-Security-Policy: default-src 'self';base-uri 'self';font-src 'self' https: data:;form-action 'self';frame-ancestors 'self';img-src 'self' data:;object-src 'none';script-src 'self';script-src-attr 'none';style-src 'self' https: 'unsafe-inline';upgrade-insecure-requests
< Cross-Origin-Opener-Policy: same-origin
< Cross-Origin-Resource-Policy: same-origin
< Origin-Agent-Cluster: ?1
< Referrer-Policy: no-referrer
< Strict-Transport-Security: max-age=15552000; includeSubDomains
< X-Content-Type-Options: nosniff
< X-DNS-Prefetch-Control: off
< X-Download-Options: noopen
< X-Frame-Options: SAMEORIGIN
< X-Permitted-Cross-Domain-Policies: none
< X-XSS-Protection: 0
< Content-Type: application/json; charset=utf-8
< Content-Length: 24
< ETag: W/"18-rnYwPv+5WQ64AtZltSy3r5DUfsk"
< Date: Mon, 17 Jun 2024 04:05:19 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
<
* Connection #0 to host rp-tailscale.esco.ghaar left intact
{"msg":"A valid route."}
3. Caddy version:
v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=
4. How I installed and ran Caddy:
Using official instructions for ubuntu
a. System environment:
OS: Linux rp-tailscale 5.15.0-97-generic #107-Ubuntu SMP Wed Feb 7 13:26:48 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
systemd: yes, caddy daemon runs fine
docker: installed, but caddy is not running on it
b. Command:
Standard systemd caddy service as setup during installation
$ cat /etc/systemd/system/multi-user.target.wants/caddy.service
# caddy.service
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
c. Service/unit/compose file:
yes, please see above
d. My complete Caddy config:
{
debug
storage file_system {
root /opt/caddy
}
}
liveliteandwell.com {
log
route {
reverse_proxy /api/* rp-tailscale.esco.ghaar:8443
respond 404
}
handle_errors {
respond "{err.status_code} {err.status_text}"
}
}
