1. The problem I’m having:
I get an error when routing Caddy+Layer4 plugin for both TLS and STARTTLS.
I have 4 services (2 SMTP TLS/STARTTLS 25 + 587) and 2 (IMAP TLS/STARTTLS 110 + 143) that I can’t route to the correct TLS certificates.
I can either route TLS and not STARTTLS or I can route STARTTLS and not TLS.
SMTP 25 TLS
openssl s_client -connect mail.landingdev.xyz:25 -servername mail.landingdev.xyz -showcerts
SMTP 25 STARTTLS
openssl s_client -connect mail.landingdev.xyz:25 -servername mail.landingdev.xyz -starttls smtp -showcerts
SMTP 587 TLS
openssl s_client -connect mail.landingdev.xyz:587 -servername mail.landingdev.xyz -showcerts
SMTP 587 STARTTLS
openssl s_client -connect mail.landingdev.xyz:587 -servername mail.landingdev.xyz -starttls smtp -showcerts
IMAP 143 TLS
openssl s_client -connect mail.landingdev.xyz:143 -servername mail.landingdev.xyz -showcerts
IMAP 143 STARTTLS
openssl s_client -connect mail.landingdev.xyz:143 -servername mail.landingdev.xyz -starttls imap -showcerts
IMAP 110 TLS
openssl s_client -connect mail.landingdev.xyz:110 -servername mail.landingdev.xyz -showcerts
IMAP 110 STARTTLS
openssl s_client -connect mail.landingdev.xyz:110 -servername mail.landingdev.xyz -starttls pop3 -showcerts
2. Error messages and/or full log output:
SMTP 25 with TLS does not hand out the correct TLS Certificates
🔴 openssl s_client -connect mail.landingdev.xyz:25 -servername mail.landingdev.xyz -showcerts
# openssl s_client -connect mail.landingdev.xyz:25 -servername mail.landingdev.xyz -showcerts
CONNECTED(00000003)
40B79AC54F7F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../ssl/record/ssl3_record.c:354:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 5 bytes and written 321 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
SMTP 25 with STARTTLS with the correct TLS Certificates
openssl s_client -connect mail.landingdev.xyz:25 -servername mail.landingdev.xyz -starttls smtp
openssl s_client -connect mail.landingdev.xyz:25 -servername mail.landingdev.xyz -starttls smtp -showcerts
CONNECTED(00000003)
depth=0 CN = rcgen self signed cert
verify error:num=18:self-signed certificate
verify return:1
depth=0 CN = rcgen self signed cert
verify return:1
---
Certificate chain
0 s:CN = rcgen self signed cert
i:CN = rcgen self signed cert
a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256
v:NotBefore: Jan 1 00:00:00 1975 GMT; NotAfter: Jan 1 00:00:00 4096 GMT
-----BEGIN CERTIFICATE-----
MIIBXzCCAQSgAwIBAgIUHPdGX/ymogT/2lMLPo6LYJwrGLQwCgYIKoZIzj0EAwIw
ITEfMB0GA1UEAwwWcmNnZW4gc2VsZiBzaWduZWQgY2VydDAgFw03NTAxMDEwMDAw
MDBaGA80MDk2MDEwMTAwMDAwMFowITEfMB0GA1UEAwwWcmNnZW4gc2VsZiBzaWdu
ZWQgY2VydDBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABINhFWt6rTEnwuy1Eaj4
/ViTYy+B1iHUWLGWeQjFIAh90efxQGAez3QYrJnipMbi43al1TgQrmQr+C5hx1qt
oW2jGDAWMBQGA1UdEQQNMAuCCWxvY2FsaG9zdDAKBggqhkjOPQQDAgNJADBGAiEA
hpow5JWQnSc2b0DLWVKdWhHIlhAEzzCxr6KXenM9NqQCIQDuJWe8ksqbjqkKUMF2
f55kFAgncZ/8re7AFW6W09AnUQ==
-----END CERTIFICATE-----
---
Server certificate
subject=CN = rcgen self signed cert
issuer=CN = rcgen self signed cert
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1029 bytes and written 434 bytes
Verification error: self-signed certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 18 (self-signed certificate)
---
250 8BITMIME
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID: 0DEE6DA6BCED9BF15314450F5A9852FC32C9C1EF0E2E51FA75ADE7E966EACC75
Session-ID-ctx:
Resumption PSK: BCD50EECADA46AD6FAF154AE7B91963A16D383F62F95E180BA0B96B003201192847307DDB013AAAD467194A99A93E70D
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 86400 (seconds)
TLS session ticket:
0000 - 57 3e ed 59 ff 09 8b a9-08 02 a6 29 27 c8 71 4d W>.Y.......)'.qM
0010 - 39 1a 1a 72 c0 ce 14 b2-25 b8 e1 df b8 d8 50 b2 9..r....%.....P.
Start Time: 1725393862
Timeout : 7200 (sec)
Verify return code: 18 (self-signed certificate)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID: 995DA60F1F10C139AC0F16838D627B14FE34D0CDAC69C45E6C13367181F443D9
Session-ID-ctx:
Resumption PSK: 5F17FFD9E329CA6C1756E04A0258F3FBE404440282E5980E9A72D0D5C700A81049E8E708F4ED02043DAC0252E3BF922D
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 86400 (seconds)
TLS session ticket:
0000 - ea c0 e2 be ab d5 90 a9-be 44 ef 77 85 52 cf 5b .........D.w.R.[
0010 - 6c 1c 9f d9 fa df c7 7f-18 41 7c f5 99 24 5c 81 l........A|..$\.
Start Time: 1725393862
Timeout : 7200 (sec)
Verify return code: 18 (self-signed certificate)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID: 5B329A906E044EFC8D59FE71D9322CB2978A3D183B4C3206D4658A9D979728EA
Session-ID-ctx:
Resumption PSK: 6561243A67453B20E1FE685718A8F0F02ED0E6164DDDC8E8F30FB740D4AC5A0BA7E6A1C67510EC1244A5945CE0B2159F
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 86400 (seconds)
TLS session ticket:
0000 - 6e f2 13 64 50 09 75 a7-cd 7d 70 bf d4 9a 70 bf n..dP.u..}p...p.
0010 - d1 26 d3 20 33 2c b1 0a-b1 e6 89 9c 2a ac 07 18 .&. 3,......*...
Start Time: 1725393862
Timeout : 7200 (sec)
Verify return code: 18 (self-signed certificate)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID: 20217E7A4A2A4805D2721100989743534B26B3B71057D7B0D486759DC17617B2
Session-ID-ctx:
Resumption PSK: 7D834BAAD901A13904418ED3625D85A0DFD0CDCE4B8A265EEC30618916C3F9C347C4F80098F5CC6C8884C85696B2E8E0
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 86400 (seconds)
TLS session ticket:
0000 - d4 3a bf 50 96 48 e3 2f-92 cd 23 0d 6c 48 ad c9 .:.P.H./..#.lH..
0010 - eb dd 85 b0 40 3c 40 06-73 a3 c5 55 d9 a9 75 c5 ....@<@.s..U..u.
Start Time: 1725393862
Timeout : 7200 (sec)
Verify return code: 18 (self-signed certificate)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
3. Caddy version:
xcaddy | v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=
4. How I installed and ran Caddy:
I’m using Caddy with the Layer4 and YAML plugins inside of Docker
FROM ubuntu:latest
RUN apt -y update
RUN apt -y upgrade
RUN apt -y install \
golang \
git
RUN go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest
RUN /root/go/bin/xcaddy \
build \
--with github.com/abiosoft/caddy-json-schema \
--with github.com/mholt/caddy-l4 \
--with github.com/lucaslorentz/caddy-docker-proxy/v2@v2.9.1 \
--with github.com/abiosoft/caddy-yaml
CMD ["/bin/bash", "-c", "bash"]
I copy the caddy
binary out of my image with:
docker build -t cad/cad:234 -f caddy-l4-docker .
docker run --rm --entrypoint cat cad/cad:234 /caddy > ./caddy
And make another container with just the caddy
binary which is just an unnecessary extra step.
FROM ubuntu:latest
RUN apt -y update
RUN apt -y upgrade
COPY caddy /usr/local/bin/
CMD ["/bin/bash", "-c", "/usr/local/bin/caddy"]
# CMD ["/bin/bash", "-c", "bash"]
a. System environment:
My VPS server
uname -a
Linux DESKTOP-3LUEUH3 5.15.153.1-microsoft-standard-WSL2 #1 SMP Fri Mar 29 23:14:13 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
# docker version
Client: Docker Engine - Community
Version: 27.2.0
API version: 1.47
Go version: go1.21.13
Git commit: 3ab4256
Built: Tue Aug 27 14:15:15 2024
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 27.2.0
API version: 1.47 (minimum version 1.24)
Go version: go1.21.13
Git commit: 3ab5c7d
Built: Tue Aug 27 14:15:15 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.21
GitCommit: 472731909fa34bd7bc9c087e4c27943f9835f111
runc:
Version: 1.1.13
GitCommit: v1.1.13-0-g58aa920
docker-init:
Version: 0.19.0
GitCommit: de40ad0
root@server:~/xcad-stal#
b. Command:
network.sh
docker network create \
--driver=bridge \
--subnet=192.168.10.0/24 \
--gateway=192.168.10.1 \
caddy-bridge-net
c. Service/unit/compose file:
compose.yml
networks:
caddy-bridge-net:
name: "caddy-bridge-net"
external: true
volumes:
caddy_data: {}
caddy_config: {}
services: #
xcaddy:
image: xcad/xcad:234
build:
context: .
dockerfile: caddy-binary-docker
command:
- /bin/sh
- -c
- |
caddy version
caddy run --config /etc/caddy/caddy.yml --adapter yaml
container_name: xcaddy
hostname: xcaddy
domainname: landingdev.xyz
networks:
caddy-bridge-net:
ipv4_address: 192.168.10.10
ports:
- "25:25"
- "110:110"
- "143:143"
- "443:443"
- "465:465"
- "587:587"
- "993:993"
- "995:995"
- "4190:4190"
- "8080:8080"
volumes:
- ./caddy.yml:/etc/caddy/caddy.yml:ro
- ./caddy_data:/data
- ./caddy_config:/config
- ./certificates/landingdev.xyz.crt:/certs/server.crt:ro
- ./certificates/landingdev.xyz.key:/certs/server.key:ro
- ./certificates/landingdev.xyz.pem:/certs/server.pem:ro
restart: no
stalwart: # mail server
depends_on:
- xcaddy
image: stalwartlabs/mail-server:latest
container_name: mail
domainname: landingdev.xyz
hostname: mail.landingdev.xyz
# ports:
# - "25:25"
# - "110:110"
# - "143:143"
# - "443:443"
# - "465:465"
# - "587:587"
# - "993:993"
# - "995:995"
# - "4190:4190"
# - "8080:8080"
networks:
caddy-bridge-net:
ipv4_address: 192.168.10.100
volumes:
- "./opt/stalwart-mail:/opt/stalwart-mail"
d. My complete Caddy config:
caddy.yml
logging:
logs:
default:
level: DEBUG
apps:
tls:
certificates:
load_files:
- certificate: "/certs/server.crt"
key: "/certs/server.key"
automation:
policies:
- subjects: ["landingdev.xyz", "*.landingdev.xyz"]
issuers:
- module: internal
layer4:
servers:
mail_8080:
listen: [":8080"]
routes:
- match:
- tls: {}
- handle:
- handler: tls
connection_policies:
- alpn: ["http/1.1", "http/2"]
- handler: proxy
upstreams:
- dial: ["mail.landingdev.xyz:8080"]
smtp_25:
listen: [":25"]
routes:
- handle:
- handler: proxy
upstreams:
- dial: ["mail.landingdev.xyz:25"]
smtp_465:
listen: [":465"]
routes:
- handle:
- handler: proxy
upstreams:
- dial: ["mail.landingdev.xyz:465"]
smtp_587:
listen: [":587"]
routes:
- handle:
- handler: proxy
upstreams:
- dial: ["mail.landingdev.xyz:587"]
imap_143:
listen: [":143"]
routes:
- handle:
- handler: tls
connection_policies:
- alpn: ["http/1.1", "http/2"]
- handler: proxy
upstreams:
- dial: ["mail.landingdev.xyz:143"]
imap_993:
listen: [":993"]
routes:
- match:
- tls: {}
- handle:
- handler: tls
connection_policies:
- alpn: ["http/1.1", "http/2"]
- handler: proxy
upstreams:
- dial: ["mail.landingdev.xyz:993"]
pop3_110:
listen: [":110"]
routes:
- handle:
- handler: proxy
upstreams:
- dial: ["mail.landingdev.xyz:110"]
pop3_995:
listen: [":995"]
routes:
- handle:
- handler: proxy
upstreams:
- dial: ["mail.landingdev.xyz:995"]
sieve_4190:
listen: [":4190"]
routes:
- match:
- tls: {}
- handle:
- handler: tls
connection_policies:
- alpn: ["http/1.1", "http/2"]
- handler: proxy
upstreams:
- dial: ["mail.landingdev.xyz:4190"]
jmap_8080_443:
listen: [":443"]
routes:
- match:
- tls: {}
- handle:
- handler: tls
connection_policies:
- alpn: ["http/1.1", "http/2"]
- handler: proxy
upstreams:
- dial: ["mail.landingdev.xyz:8080"]
5. Links to relevant resources:
caddyserver config adapters documentation
https://caddyserver.com/docs/config-adapters
caddy-docker-proxy
https://github.com/lucaslorentz/caddy-docker-proxy
caddy-yaml
https://github.com/abiosoft/caddy-yaml
caddy-l4
https://github.com/mholt/caddy-l4
caddy-json-schema
https://github.com/abiosoft/caddy-json-schema