TLS Cipher mismatch on iOS only when using websocket

Help
1

Hi,

I’m having trouble connecting to websockets on iOS to a server running caddy while “plain https” works just fine. After spending some time debugging it, it seems to be due to a TLS cipher mismatch that happens only with websockets. More details below.

1. Caddy version (caddy version):

v2.4.6 h1:HGkGICFGvyrodcqOOclHKfvJC0qTU7vny/7FhYp9hNw=

2. How I run Caddy:

a. System environment:

Fedora 35

b. Command:

# As root
cd /etc/caddy
caddy run

the problem is the same when using a systemd service (i use both depending if i’m actively trying to debug or not)

c. Service/unit/compose file:

N/A

d. My complete Caddyfile or JSON config:

{
    debug
}

weechat.partou.se {
    reverse_proxy /weechat localhost:8001
}

3. The problem I’m having:

I have trouble starting a websocket connection between my caddy server and my iOS device, both using Lith (a native weechat relay client) or using Safari and Glowing bear (a browser weechat relay client) due to a cipher mismatch.
The websocket endpoint is https://weechat.partou.se/weechat
Nevertheless, by doing a “plain HTTP” request with Safari on https://weechat.partou.se/ the TLS negotiation works without problem.

4. Error messages and/or full log output:

On the server side:

2022/02/22 06:22:12.808	DEBUG	http.stdlib	http: TLS handshake error from 94.109.184.121:28624: tls: no cipher suite supported by both client and server

On the client side:

Conection failed: SSLHandshake failed: -9824

5. What I already tried:

Activating all ciphers on the caddy side

I tried to activate all the cipher suites that i could find in the documentation but it did not help

{
    debug
}

weechat.partou.se {
    reverse_proxy /weechat localhost:8001
    tls {
        ciphers TLS_RSA_WITH_3DES_EDE_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_AES_128_GCM_SHA256 TLS_RSA_WITH_AES_256_GCM_SHA384 TLS_AES_128_GCM_SHA256 TLS_AES_256_GCM_SHA384 TLS_CHACHA20_POLY1305_SHA256 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
    }
}

tcpdump on the server side

In last resort, here is the the “Client Hello” frame of a tcpdump i did on my server (tcpdump port 443 -w websocket.pcap) visualized in wireshark.

I did this both on with the “plain HTTPS” and the websocket.

I can provide the raw pcap if needed but i did not find a way to upload them on the forum.

We clearly see that when doing a websocket connection, the client supports less ciphers. But it still supports TLS_RSA_WITH_3DES_EDE_CBC_SHA that is also in Caddy so i don’t understand why the negotiation fails.

Websocket

We can see that the client accepts 17 cipher suites (and the connection then fails)

wireshark output (17 suites) 
Frame 3: 233 bytes on wire (1864 bits), 233 bytes captured (1864 bits)
Ethernet II, Src: Cisco_e7:6a:a7 (b0:c5:3c:e7:6a:a7), Dst: VMware_05:16:57 (00:50:56:05:16:57)
Internet Protocol Version 4, Src: 109.122.178.45, Dst: 178.32.41.106
Transmission Control Protocol, Src Port: 53483, Dst Port: 443, Seq: 1, Ack: 1, Len: 167
Transport Layer Security
    TLSv1.2 Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 162
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Length: 158
            Version: TLS 1.2 (0x0303)
            Random: 62148444e4c7329975919223874f2cd6ed38401c9331debd54b84e7851949b0f
            Session ID Length: 0
            Cipher Suites Length: 34
            Cipher Suites (17 suites)
                Cipher Suite: TLS_EMPTY_RENEGOTIATION_INFO_SCSV (0x00ff)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 (0xc024)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 (0xc023)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027)
                Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 (0x006b)
                Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (0x0067)
                Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
                Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)
                Cipher Suite: TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA (0x0016)
                Cipher Suite: TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d)
                Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA256 (0x003d)
                Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA256 (0x003c)
                Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
                Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
                Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)
            Compression Methods Length: 1
            Compression Methods (1 method)
            Extensions Length: 83
            Extension: server_name (len=22)
            Extension: supported_groups (len=8)
            Extension: ec_point_formats (len=2)
            Extension: signature_algorithms (len=18)
            Extension: status_request (len=5)
            Extension: signed_certificate_timestamp (len=0)
            Extension: extended_master_secret (len=0)
            [JA3 Fullstring: 771,255-49188-49187-49200-49192-49191-107-103-57-51-22-157-61-60-53-47-10,0-10-11-13-5-18-23,23-24-25,0]
            [JA3: 70623e36f44974e030ad5615c304b592]
Plain HTTPS

We can see that the client accepts 27 cipher suites (and the connection then succeeds)

wireshark output 
Frame 4: 583 bytes on wire (4664 bits), 583 bytes captured (4664 bits)
Ethernet II, Src: Cisco_e7:6a:a7 (b0:c5:3c:e7:6a:a7), Dst: VMware_05:16:57 (00:50:56:05:16:57)
Internet Protocol Version 4, Src: 109.122.178.45, Dst: 178.32.41.106
Transmission Control Protocol, Src Port: 49496, Dst Port: 443, Seq: 1, Ack: 1, Len: 517
Transport Layer Security
    TLSv1.3 Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 512
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Length: 508
            Version: TLS 1.2 (0x0303)
            Random: bb4748f644cbe9b421d27390050c5a1d207b8b42378771a0ed2bfe1cc07e2447
            Session ID Length: 32
            Session ID: 89ce72717f57277cb7e0689217f4b8f6bed88ad128b5faae59594f32b4ceb620
            Cipher Suites Length: 54
            Cipher Suites (27 suites)
                Cipher Suite: Reserved (GREASE) (0x2a2a)
                Cipher Suite: TLS_AES_128_GCM_SHA256 (0x1301)
                Cipher Suite: TLS_AES_256_GCM_SHA384 (0x1302)
                Cipher Suite: TLS_CHACHA20_POLY1305_SHA256 (0x1303)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca9)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
                Cipher Suite: TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca8)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 (0xc024)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 (0xc023)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
                Cipher Suite: TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d)
                Cipher Suite: TLS_RSA_WITH_AES_128_GCM_SHA256 (0x009c)
                Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA256 (0x003d)
                Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA256 (0x003c)
                Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
                Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA (0xc008)
                Cipher Suite: TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA (0xc012)
                Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)
            Compression Methods Length: 1
            Compression Methods (1 method)
            Extensions Length: 381
            Extension: Reserved (GREASE) (len=0)
            Extension: server_name (len=22)
            Extension: extended_master_secret (len=0)
            Extension: renegotiation_info (len=1)
            Extension: supported_groups (len=12)
            Extension: ec_point_formats (len=2)
            Extension: application_layer_protocol_negotiation (len=14)
            Extension: status_request (len=5)
            Extension: signature_algorithms (len=24)
            Extension: signed_certificate_timestamp (len=0)
            Extension: key_share (len=43)
            Extension: psk_key_exchange_modes (len=2)
            Extension: supported_versions (len=11)
            Extension: compress_certificate (len=3)
            Extension: Reserved (GREASE) (len=1)
            Extension: padding (len=177)
            [JA3 Fullstring: 771,10794-4865-4866-4867-49196-49195-52393-49200-49199-52392-49188-49187-49162-49161-49192-49191-49172-49171-157-156-61-60-53-47-49160-49170-10,39578-0-23-65281-10-11-16-5-13-18-51-45-43-27-10794-21,39578-29-23-24-25,0]
            [JA3: 742e9bf0c7a47c03cbfec3ce8e427841]
2

Please upgrade to v2.4.6!

See SSL Server Test: weechat.partou.se (Powered by Qualys SSL Labs), your certificate was revoked. See here for an explanation:

Upgrading to v2.4.6 should cause Caddy to renew the certificate, or you can force a renewal by deleting the cert from Caddy’s storage and reloading Caddy.

You didn’t say exactly how you’re running Caddy, so it’s unclear to me where your storage location is. But if you’re running it as a systemd service (you should) then it’ll be in /var/lib/caddy/.local/share/caddy.

1 Like
3

Hum, it does look like i had 2 problems at the same time. Indeed, some browsers were complaining about the certificate revocation (but not on iOS).

I did the upgrade, the certificates were not regenerated but i was able to deleted them by hand and request them again.

Nevertheless, the problem is still happening with my iOS device: negotiation is not happening and it does not even reach the certificate validation step.

I updated my post above to precise how i’m running caddy and with the new version.

4

I’m not sure why you would ever need to run it outside of systemd, that’ll just cause your site to serve a different set of certificates (because the storage location will be different between systemd vs not). You can find your logs when running as a systemd service by running the command outlined here in the docs:

Which version of iOS is this on? Is it a relatively old device? The device might be too old to connect to servers with modern TLS.

See SSL Server Test: weechat.partou.se (Powered by Qualys SSL Labs), you probably need iOS 9 or higher.

5

that’ll just cause your site to serve a different set of certificates (because the storage location will be different between systemd vs not

Oh, i did not know that. Thank you !

Which version of iOS is this on? Is it a relatively old device? The device might be too old to connect to servers with modern TLS.

It’s iOS 15.3.1, the newest version, running on an iPhone SE 2020. To be clear, it does work when loading a web page, just not when trying to establish a websocket connection.

6

Looking at the 17 cipher suites you wrote, I actually don’t see any overlap with Caddy’s. Of note, Caddy will use an EC256 private key, meaning all the RSA suites won’t get used (because Caddy isn’t using an RSA key).

You could configure Caddy to generate an RSA key instead and have a cert issued with that (you’ll need to wipe out your storage again to force it to renew with the RSA key).

You can set this global option to do it:

key_type rsa2048

RSA is less efficient, will produce a bigger TLS response (because the keys are bigger, so bigger certificates), but I think it should have better compatibility for now.

I’d suggest it’s an issue with the way those apps were written that is causing the problem otherwise.

1 Like
7

You could configure Caddy to generate an RSA key instead and have a cert issued with that (you’ll need to wipe out your storage again to force it to renew with the RSA key).

Awesome, this does work ! Thanks a lot !

I’d suggest it’s an issue with the way those apps were written that is causing the problem otherwise.

For me, the problem is solved and i’m happy (thank you :heart:) , but i do think this is a compatibility problem with iOS as a whole, not the way the app is written as the problem also happens with Safari

This is the result of a page load over https with Safari (it works)

image
image1920×1200 87.8 KB

While this is the result of the websocket connection : (imgur link as i may not embed 2 images in a single post) https://imgur.com/a/dz5lTUk

1 Like
8

I would find that surprising because we would’ve heard more complaints about it otherwise.

Glad you have a solution for now though :+1:

9

This topic was automatically closed after 30 days. New replies are no longer allowed.