1. Caddy version (caddy version):
v2.4.1 h1:kAJ0JB5Xk5gPdTH/27S5cyoMGqD5lBAe9yZ8zTjVJa0=
2. How I run Caddy:
Run caddy as a systemd service using stock service file except changed to be used by non-root systemd using custom user:
[Service]
Type=notify
ExecStart=/srv/popple/caddy/caddy run --environ --config /srv/popple/caddy/caddy.yaml --adapter yaml
ExecReload=/srv/popple/caddy/caddy reload --config /srv/popple/caddy/caddy.yaml --adapter yaml
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
a. System environment:
Ubuntu 20.04 using basic user (no root or sudo) that I use to run rootless Docker containers. All serving is done by and from this rootless user.
b. Command:
systemctl --user start caddy
d. My complete Caddyfile or JSON config:
My json file as yaml:
apps:
layer4:
servers:
mdath:
listen:
- ":443"
routes:
- match:
- tls:
sni:
- "*.e1cmyhhndp0ep.cdn.network"
handle:
- handler: "proxy"
proxy_protocol: "v2"
upstreams:
- dial:
- "localhost:4433"
- handle:
- handler: "tls"
- match:
- http: []
handle:
- handler: proxy
upstreams:
- dial:
- localhost:80
http:
servers:
blog.me:
listen:
- ":80"
routes:
- match:
- host:
- "localhost"
- "127.0.0.1"
handle:
- handler: file_server
root: /srv/popple/blog.me
my_blog:
listen: ["localhost:5733"]
routes:
- handle:
- handler: "reverse_proxy"
upstreams:
- dial: "localhost:2368"
tls:
certificates:
automate:
- "localhost"
automation:
policies:
- issuers:
- module: internal
subjects:
- "localhost"
- issuers:
- module: acme
email: <email>
ca: https://acme-staging-v02.api.letsencrypt.org/directory
logging:
logs:
default:
level: "DEBUG"
3. The problem I’m having:
I want to use layer4 app to proxy some tls traffic to a backend unterminated while getting all the normal caddy automatic HTTPS behavior for a variety of other http servers.
Since I can’t use caddyfile and layer4 at the same time I’ve been studying the json configs and forums to try to get it to work using only localhost for now.
My current solution is to terminate in layer4 and proxy to http but the proxying isn’t working and it seems like I won’t get auto https features like always redirect (since output thinks I’m only listening to port 80)
4. Error messages and/or full log output:
Startup output:
Jun 06 01:15:21 localhost caddy[1238114]: {"level":"info","ts":1622909721.1721902,"msg":"using provided configuration","config_file":"/srv/popple/caddy/caddy.yaml","config_adapter":"yaml"}
Jun 06 01:15:21 localhost caddy[1238114]: {"level":"info","ts":1622909721.173868,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Jun 06 01:15:21 localhost caddy[1238114]: {"level":"info","ts":1622909721.1747122,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0001c50a0"}
Jun 06 01:15:21 localhost caddy[1238114]: {"level":"info","ts":1622909721.181795,"logger":"http","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"blog.me","http_port":80}
Jun 06 01:15:21 localhost caddy[1238114]: {"level":"warn","ts":1622909721.1829004,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [localhost]: no OCSP server specified in certificate"}
Jun 06 01:15:21 localhost caddy[1238114]: {"level":"debug","ts":1622909721.183083,"logger":"http","msg":"starting server loop","address":"[::]:80","http3":false,"tls":false}
Jun 06 01:15:21 localhost caddy[1238114]: {"level":"info","ts":1622909721.1830945,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/srv/popple/.local/share/caddy"}
Jun 06 01:15:21 localhost caddy[1238114]: {"level":"debug","ts":1622909721.1832669,"logger":"http","msg":"starting server loop","address":"127.0.0.1:5733","http3":false,"tls":false}
Jun 06 01:15:21 localhost caddy[1238114]: {"level":"debug","ts":1622909721.183423,"logger":"layer4","msg":"listening","address":"tcp/[::]:443"}
Jun 06 01:15:21 localhost caddy[1238114]: {"level":"info","ts":1622909721.1842442,"logger":"tls","msg":"finished cleaning storage units"}
Jun 06 01:15:21 localhost caddy[1238114]: {"level":"warn","ts":1622909721.207919,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
Jun 06 01:15:21 localhost caddy[1238114]: 2021/06/06 01:15:21 not NSS security databases found
Jun 06 01:15:21 localhost caddy[1238114]: 2021/06/06 01:15:21 define JAVA_HOME environment variable to use the Java trust
Jun 06 01:15:21 localhost sudo[1238122]: pam_unix(sudo:auth): Couldn't open /etc/securetty: No such file or directory
Jun 06 01:15:21 localhost sudo[1238122]: pam_unix(sudo:auth): conversation failed
Jun 06 01:15:21 localhost sudo[1238122]: pam_unix(sudo:auth): auth could not identify password for [popple]
Jun 06 01:15:21 localhost caddy[1238114]: {"level":"error","ts":1622909721.2141056,"logger":"pki.ca.local","msg":"failed to install root certificate","error":"failed to execute sudo: exit status 1","certificate_file":"storage:pki/authorities/local/root.crt"}
Jun 06 01:15:21 localhost caddy[1238114]: {"level":"info","ts":1622909721.2145255,"msg":"autosaved config (load with --resume flag)","file":"/srv/popple/.config/caddy/autosave.json"}
Jun 06 01:15:21 localhost caddy[1238114]: {"level":"info","ts":1622909721.2148037,"msg":"serving initial configuration"}
Jun 06 01:15:21 localhost systemd[871310]: Started Caddy.
-- Subject: A start job for unit UNIT has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit UNIT has finished successfully.
This is normal except for the lack of sudo causing error when trying to create root certificate locally. I ran caddy trust from a sudo account and it says it’s trusted but I guess there’s a problem since curl also doesn’t work without a -k argument to trust invalid certs. I don’t think that’s related to my problem but I want to be complete so mentioning it.
curl -v http://localhost returns the expected html contents.
curl -kv https://localhost terminates tls and appears to proxy to localhost:80 but no content is returned and curl returns an error.
caddy logs:
Jun 06 01:20:31 localhost caddy[1238114]: {"level":"debug","ts":1622910031.5418913,"logger":"layer4.handlers.tls","msg":"terminated TLS","server_name":"localhost"}
Jun 06 01:20:31 localhost caddy[1238114]: {"level":"debug","ts":1622910031.5425372,"logger":"layer4.handlers.proxy","msg":"dial upstream","address":"localhost:80"}
curl output:
curl -kv https://localhost
* Trying ::1:443...
* TCP_NODELAY set
* Connected to localhost (::1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: [NONE]
* start date: Jun 5 08:29:53 2021 GMT
* expire date: Jun 5 20:29:53 2021 GMT
* issuer: CN=Caddy Local Authority - ECC Intermediate
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55c036351e10)
> GET / HTTP/2
> Host: localhost
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* http2 error: Remote peer returned unexpected data while we expected SETTINGS frame. Perhaps, peer does not support HTTP/2 properly.
* OpenSSL SSL_write: Broken pipe, errno 32
* Failed sending HTTP2 data
* Connection #0 to host localhost left intact
curl: (55) OpenSSL SSL_write: Broken pipe, errno 32
5. What I already tried:
Since layer4 is binding :443 I assume I need to terminate my https requests there always since there’s an error if I try to listen to :443 in the http app as well.
I’ve tried various combinations of localhost vs 127.0.0.1 and moving logic between matchers adn listen blocks but nothing worked and it was hard to know if I was going in the right direction.
6. Links to relevant resources:
GitHub - mholt/caddy-l4: Layer 4 (TCP/UDP) app for Caddy has various examples but none using layer4 with http app together.
