1. The problem I’m having:
I running Caddy as a reverse proxy to serve Matrix Synapse Server. The configuration is the bare minimum and worked well.
Now I ran into a problem with the Element iOS client. The networking stopped (randomly) working and the client seems to endless loading. I digged into the problem and debugged the Element for iOS and received this logs:
2023-04-29 12:13:37.341322+0200 Element[76705:4662274] [boringssl] boringssl_session_handshake_incomplete(88) [C12.1.2.1:2][0x123220370] SSL library error
2023-04-29 12:13:37.345932+0200 Element[76705:4662274] [boringssl] boringssl_session_handshake_error_print(43) [C12.1.2.1:2][0x123220370] Error: 4850737128:error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:/Library/Caches/com.apple.xbs/Sources/boringssl_Sim/ssl/handshake.cc:419:
2023-04-29 12:13:37.346013+0200 Element[76705:4662274] [boringssl] nw_protocol_boringssl_handshake_negotiate_proceed(771) [C12.1.2.1:2][0x123220370] handshake failed at state 12288: not completed
2023-04-29 12:13:37.346285+0200 Element[76705:4662274] [boringssl] boringssl_context_handle_fatal_alert(1991) [C12.1.2.2:2][0x14c5b73d0] write alert, level: fatal, description: certificate unknown
2023-04-29 12:13:37.346336+0200 Element[76705:4662274] [boringssl] boringssl_context_error_print(1981) [C12.1.2.2:2][0x14c5b73d0] Error: 4850737128:error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:/Library/Caches/com.apple.xbs/Sources/boringssl_Sim/ssl/handshake.cc:419:
2023-04-29 12:13:37.346460+0200 Element[76705:4662274] [boringssl] boringssl_session_handshake_incomplete(88) [C12.1.2.2:2][0x14c5b73d0] SSL library error
2023-04-29 12:13:37.346510+0200 Element[76705:4662274] [boringssl] boringssl_session_handshake_error_print(43) [C12.1.2.2:2][0x14c5b73d0] Error: 4850737128:error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:/Library/Caches/com.apple.xbs/Sources/boringssl_Sim/ssl/handshake.cc:419:
2023-04-29 12:13:37.346561+0200 Element[76705:4662274] [boringssl] nw_protocol_boringssl_handshake_negotiate_proceed(771) [C12.1.2.2:2][0x14c5b73d0] handshake failed at state 12288: not completed
Why I ask this here you might ask and I also tried different networks, reinstall the App and at the end I used them in an airgapped environment to reproduce this scenario. But the solution was to restart Caddy. So I tried to find out why iOS was not able to do the handshake and found a document which outlines the requirements for TLS on iOS: Apple's Certificate Transparency policy - Apple Support
There is also an Stackoverflow post existing targeting this issue: nginx - Secure websocket connection fails on iOS due to BoringSSL certificate verification failure - Stack Overflow
It could be related with OCSP Stapling and Signed Certificate Timestamps. Therefore I seek for a Caddy Pro which can help me to dig deeper and maybe shed some light in this section and how Caddy works in that section.
2. Error messages and/or full log output:
There is no log present because of not enabled debug and the problem is before the connection is established.
3. Caddy version:
v2.6.4
4. How I installed and ran Caddy:
a. System environment:
OS: Debian Bullseye
ARCH: amd64
Service file:
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs
After=network.target network-online.target
Requires=network-online.target
[Service]
User=caddy
Group=caddy
LimitNOFILE=1048576
LimitNPROC=512
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
Restart=on-failure
StartLimitInterval=600
TimeoutStopSec=5s
# Service security
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=yes
PrivateTmp=yes
PrivateDevices=yes
DevicePolicy=closed
ProtectSystem=full
ProtectControlGroups=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
RestrictNamespaces=yes
RestrictRealtime=yes
MemoryDenyWriteExecute=yes
LockPersonality=yes
ProtectHome=true
CapabilityBoundingSet=~CAP_SYS_ADMIN CAP_SYS_PTRACE CAP_SETUID CAP_SETGID CAP_SETPCAP CAP_CHOWN CAP_FSETID CAP_SETFCAP CAP_NET_ADMIN CAP_SYS_TIME
ProtectClock=true
ProtectKernelLogs=true
ProtectProc=invisible
[Install]
WantedBy=multi-user.target
My complete Caddy config:
email admin@example.org
matrix.example.org {
reverse_proxy http://matrix1:80 {
header_down -Server
}
}
matrix.example.org:8448 {
reverse_proxy http://matrix1:8448 {
header_down -Server
}
}