1. The problem I’m having:
I am currently using Caddy to serve a bunch of sites with automatic HTTPS and it has been working perfectly for this use-case.
However, I now need Caddy to serve another site that manages its own certificates by passing through all requests to that site (including ACME challenges). After doing some research, I found the caddy-l4 repository and I understand that it should be able to do what I want. I converted my Caddyfile to JSON and tried to add in the layer4 configuration by hand but it does not seem to work.
Here’s what I’ve tried to do:
- I left my configuration for the
httpapp as is. - I added a
listener_wrapperand setlayer4as a wrapper. - Inside the wrapper, I matched the SNIs of the site I want to pass through and proxied it to the service handling the site.
After doing this, when trying to load a site that is managed by Caddy’s http app, I run into the following:
$ curl -vvL https://abc.xyz.org
* Trying 143.215.130.109:443...
* TCP_NODELAY set
* Connected to abc.xyz.org (143.215.130.109) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=abc.xyz.org
* start date: Dec 18 23:15:19 2023 GMT
* expire date: Mar 17 23:15:18 2024 GMT
* subjectAltName: host "abc.xyz.org" matched cert's "abc.xyz.org"
* issuer: C=US; O=Let's Encrypt; CN=R3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55b751f820c0)
> GET / HTTP/2
> Host: abc.xyz.org
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* http2 error: Remote peer returned unexpected data while we expected SETTINGS frame. Perhaps, peer does not support HTTP/2 properly.
* OpenSSL SSL_write: Connection reset by peer, errno 104
* Failed sending HTTP2 data
* Connection #0 to host abc.xyz.org left intact
curl: (55) OpenSSL SSL_write: Connection reset by peer, errno 104
I am very new to Caddy’s JSON config and I’m sure there are some gaps in my understanding. I am also aware that there are older topics on this forum that address this but I am not sure if they are still relevant. I would also appreciate a solution that prevents me from writing more JSON than I need to xD.
2. Caddy version:
v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=
3. How I installed and ran Caddy:
I installed Caddy using xcaddy -
xcaddy build --with github.com/mholt/caddy-l4
a. System environment:
Ubuntu 22.04.3 LTS
systemd 249
b. Command:
N/A - I run Caddy using the systemd service, see below.
c. Service/unit/compose file:
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/config.json
ExecReload=/usr/bin/caddy reload --config /etc/caddy/config.json --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
d. My complete Caddy config:
{
"logging": {
"sink": {
"writer": {
"filename": "/var/log/caddy/sink.log",
"output": "file"
}
},
"logs": {
"default": {
"exclude": [
"http.log.access.log0",
"http.log.access.log1",
"http.log.access.log2",
"http.log.access.log3",
"http.log.access.log4"
],
"writer": {
"filename": "/var/log/caddy/default.log",
"output": "file"
},
"level": "DEBUG"
},
"log0": {
"writer": {
"filename": "/var/log/caddy/def.log",
"output": "file",
"roll_keep": 1000,
"roll_keep_days": 900
},
"include": [
"http.log.access.log0"
]
},
"log1": {
"writer": {
"filename": "/var/log/caddy/abc.log",
"output": "file",
"roll_keep": 1000,
"roll_keep_days": 900
},
"include": [
"http.log.access.log1"
]
},
"log2": {
"writer": {
"filename": "/var/log/caddy/coolsite.log",
"output": "file",
"roll_keep": 100,
"roll_keep_days": 90
},
"include": [
"http.log.access.log2"
]
},
"log3": {
"writer": {
"filename": "/var/log/caddy/files.log",
"output": "file",
"roll_keep": 100,
"roll_keep_days": 90
},
"include": [
"http.log.access.log3"
]
},
"log4": {
"writer": {
"filename": "/var/log/caddy/website.log",
"output": "file"
},
"include": [
"http.log.access.log4"
]
}
}
},
"apps": {
"http": {
"servers": {
"srv443": {
"listen": [
":443",
":853"
],
"routes": [
{
"match": [
{
"host": [
"abc.xyz.org",
"abc.xyz.fail"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"handler": "reverse_proxy",
"upstreams": [
{
"dial": "localhost:1111"
}
]
}
]
}
]
}
],
"terminal": true
},
{
"match": [
{
"host": [
"def.xyz.org",
"def.xyz.fail"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"group": "group5",
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"handler": "reverse_proxy",
"upstreams": [
{
"dial": "localhost:5003"
}
]
}
]
}
]
}
],
"match": [
{
"path": [
"/randompath*"
]
}
]
},
{
"group": "group5",
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"handler": "reverse_proxy",
"upstreams": [
{
"dial": "localhost:5002"
}
]
}
]
}
]
}
]
}
]
}
],
"terminal": true
},
{
"match": [
{
"host": [
"files.xyz.org",
"files.xyz.fail"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"handler": "vars",
"root": "/opt/xyz-site-hosted-files"
},
{
"handler": "file_server",
"hide": [
"/etc/caddy/Caddyfile"
]
}
]
}
]
}
],
"terminal": true
},
{
"match": [
{
"host": [
"www.xyz.fail"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"handler": "static_response",
"headers": {
"Location": [
"https://xyz.fail{http.request.uri}"
]
},
"status_code": 301
}
]
}
]
}
],
"terminal": true
},
{
"match": [
{
"host": [
"coolsite.dev"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"handler": "reverse_proxy",
"upstreams": [
{
"dial": "localhost:7732"
}
]
}
]
}
]
}
],
"terminal": true
},
{
"match": [
{
"host": [
"xyz.fail"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"handler": "vars",
"root": "/opt/xyz-site"
},
{
"handler": "file_server",
"hide": [
"/etc/caddy/Caddyfile"
]
}
]
}
]
}
],
"terminal": true
}
],
"listener_wrappers": [
{
"wrapper": "layer4",
"routes": [
{
"match": [
{
"tls": {
"sni": [
"coolsite-backend.io",
"admin.coolsite-backend.io",
"www.coolsite-backend.io",
"mfa-test.coolsite-backend.io",
"mdm.coolsite-backend.io",
"ns.coolsite-backend.io"
]
}
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"handler": "proxy",
"upstreams": [
{
"dial": [
"localhost:8443"
]
}
]
}
]
}
]
}
]
}
]
}
],
"logs": {
"logger_names": {
"abc.xyz.fail": "log1",
"abc.xyz.org": "log1",
"coolsite.dev": "log2",
"files.xyz.fail": "log3",
"files.xyz.org": "log3",
"xyz.fail": "log4",
"def.xyz.fail": "log0",
"def.xyz.org": "log0"
},
"skip_hosts": [
"www.xyz.fail"
]
}
}
}
}
}
}
