`tls: bad record MAC` when attempting to establish a websocket connection over iOS

1. The problem I’m having:

I am attempting to establish a websocket connection via an iOS client through Caddy.

The iOS client is built with Unreal 4.27.

Normal HTTPS requests work on iOS.
Both HTTPS requests and websocket connections work on Windows via a built version of the game.
Websockets can connect to the server that Caddy is routing to directly by bypassing Caddy (non-wss, no cert on the server).
If I do not enable WSS, and allow for insecure websockets, I can connect through Caddy.

Since Caddy uses LetsEncrypt as a CA, this should work without needing to bake in a cert on the client app as far as I know (am I wrong here?).

Is there something I’m doing wrong here, or should this be working? Thanks!

2. Error messages and/or full log output:

Client error log message:

LogWebSockets: Warning: Lws(Error): SSL error: unable to get local issuer certificate (preverify_ok=0;err=20;depth=1) 

Entire Caddy log:

2024/05/20 07:00:42.702 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2024/05/20 07:00:42.705 INFO    http.auto_https enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2024/05/20 07:00:42.705 INFO    http.auto_https enabling automatic HTTP->HTTPS redirects        {"server_name": "srv1"}
2024/05/20 07:00:42.705 INFO    http.auto_https enabling automatic HTTP->HTTPS redirects        {"server_name": "srv2"}
2024/05/20 07:00:42.705 INFO    http.auto_https enabling automatic HTTP->HTTPS redirects        {"server_name": "srv3"}
2024/05/20 07:00:42.705 INFO    http.auto_https enabling automatic HTTP->HTTPS redirects        {"server_name": "srv4"}
2024/05/20 07:00:42.705 INFO    http.auto_https enabling automatic HTTP->HTTPS redirects        {"server_name": "srv5"}
2024/05/20 07:00:42.709 INFO    http    enabling HTTP/3 listener        {"addr": ":10100"}
2024/05/20 07:00:42.709 INFO    http.log        server running  {"name": "srv1", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:00:42.709 INFO    http    enabling HTTP/3 listener        {"addr": ":10200"}
2024/05/20 07:00:42.709 INFO    http.log        server running  {"name": "srv2", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:00:42.709 INFO    http    enabling HTTP/3 listener        {"addr": ":11000"}
2024/05/20 07:00:42.709 INFO    http.log        server running  {"name": "srv3", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:00:42.709 INFO    http    enabling HTTP/3 listener        {"addr": ":11100"}
2024/05/20 07:00:42.709 INFO    http.log        server running  {"name": "srv4", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:00:42.709 INFO    http    enabling HTTP/3 listener        {"addr": ":11200"}
2024/05/20 07:00:42.710 INFO    http.log        server running  {"name": "srv5", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:00:42.710 INFO    http.log        server running  {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:00:42.710 INFO    http    enabling HTTP/3 listener        {"addr": ":10000"}
2024/05/20 07:00:42.710 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0x4000496c80"}
2024/05/20 07:00:42.711 INFO    http.log        server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:00:42.711 INFO    http    enabling automatic TLS certificate management   {"domains": ["my.env.internal.domain.net"]}
2024/05/20 07:00:42.711 INFO    autosaved config (load with --resume flag)      {"file": "/root/.config/caddy/autosave.json"}
2024/05/20 07:00:42.711 INFO    serving initial configuration
2024/05/20 07:00:42.711 WARN    tls     unable to get instance ID; storage clean stamps will be incomplete      {"error": "open /root/.local/share/caddy/instance.uuid: no such file or directory"}
2024/05/20 07:00:42.711 INFO    tls.obtain      acquiring lock  {"identifier": "my.env.internal.domain.net"}
2024/05/20 07:00:42.713 INFO    tls     cleaning storage unit   {"storage": "FileStorage:/root/.local/share/caddy"}
2024/05/20 07:00:42.713 INFO    tls     finished cleaning storage units
2024/05/20 07:00:42.713 INFO    tls.obtain      lock acquired   {"identifier": "my.env.internal.domain.net"}
2024/05/20 07:00:42.713 INFO    tls.obtain      obtaining certificate   {"identifier": "my.env.internal.domain.net"}
2024/05/20 07:00:42.723 INFO    admin.api       received request        {"method": "GET", "host": "localhost:2019", "uri": "/pki/ca/local", "remote_ip": "127.0.0.1", "remote_port": "37166", "headers": {"Accept-Encoding":["gzip"],"Origin":["http://localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
2024/05/20 07:00:43.078 INFO    http    waiting on internal rate limiter        {"identifiers": ["my.env.internal.domain.net"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2024/05/20 07:00:43.078 INFO    http    done waiting on internal rate limiter   {"identifiers": ["my.env.internal.domain.net"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2024/05/20 07:00:43.403 INFO    http.acme_client        trying to solve challenge       {"identifier": "my.env.internal.domain.net", "challenge_type": "http-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2024/05/20 07:00:43.550 INFO    http    served key authentication       {"identifier": "my.env.internal.domain.net", "challenge": "http-01", "remote": "3.129.25.162:48210", "distributed": false}
2024/05/20 07:00:43.736 INFO    http    served key authentication       {"identifier": "my.env.internal.domain.net", "challenge": "http-01", "remote": "23.178.112.107:45455", "distributed": false}
2024/05/20 07:00:43.746 INFO    http    served key authentication       {"identifier": "my.env.internal.domain.net", "challenge": "http-01", "remote": "16.170.255.201:30194", "distributed": false}
2024/05/20 07:00:43.759 INFO    http    served key authentication       {"identifier": "my.env.internal.domain.net", "challenge": "http-01", "remote": "18.237.3.70:30254", "distributed": false}
2024/05/20 07:00:43.986 INFO    http    served key authentication       {"identifier": "my.env.internal.domain.net", "challenge": "http-01", "remote": "13.212.231.193:16712", "distributed": false}
2024/05/20 07:00:44.094 INFO    http.acme_client        authorization finalized {"identifier": "my.env.internal.domain.net", "authz_status": "valid"}
2024/05/20 07:00:44.094 INFO    http.acme_client        validations succeeded; finalizing order {"order": "https://acme-v02.api.letsencrypt.org/acme/order/1735754592/270951627302"}
2024/05/20 07:00:44.951 INFO    http.acme_client        successfully downloaded available certificate chains    {"count": 2, "first_url": "https://acme-v02.api.letsencrypt.org/acme/cert/0306bc40dbe9f9c0a9438f86261d9909bf9b"}
2024/05/20 07:00:44.951 INFO    tls.obtain      certificate obtained successfully       {"identifier": "my.env.internal.domain.net"}
2024/05/20 07:00:44.951 INFO    tls.obtain      releasing lock  {"identifier": "my.env.internal.domain.net"}
2024/05/20 07:15:16.127 INFO    shutting down apps, then terminating    {"signal": "SIGTERM"}
2024/05/20 07:15:16.127 WARN    exiting; byeee!! 👋     {"signal": "SIGTERM"}
2024/05/20 07:15:16.127 INFO    http    servers shutting down with eternal grace period
2024/05/20 07:15:16.127 INFO    admin   stopped previous server {"address": "localhost:2019"}
2024/05/20 07:15:16.128 INFO    shutdown complete       {"signal": "SIGTERM", "exit_code": 0}
2024/05/20 07:15:16.446 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["//127.0.0.1:2019", "//localhost:2019", "//[::1]:2019"]}
2024/05/20 07:15:16.447 INFO    http.auto_https enabling automatic HTTP->HTTPS redirects        {"server_name": "srv1"}
2024/05/20 07:15:16.447 INFO    http.auto_https enabling automatic HTTP->HTTPS redirects        {"server_name": "srv2"}
2024/05/20 07:15:16.447 INFO    http.auto_https enabling automatic HTTP->HTTPS redirects        {"server_name": "srv3"}
2024/05/20 07:15:16.447 INFO    http.auto_https enabling automatic HTTP->HTTPS redirects        {"server_name": "srv4"}
2024/05/20 07:15:16.447 INFO    http.auto_https enabling automatic HTTP->HTTPS redirects        {"server_name": "srv5"}
2024/05/20 07:15:16.447 INFO    http.auto_https enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2024/05/20 07:15:16.447 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0x40004e2880"}
2024/05/20 07:15:16.447 DEBUG   http.auto_https adjusted config {"tls": {"automation":{"policies":[{}]}}, "http": {"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":10000"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"rate_limit","rate_limits":{"vin_diesel":{"key":"static","max_events":5,"window":1000000000}}},{"handler":"reverse_proxy","upstreams":[{"dial":"172.31.91.222:10000"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}},"srv1":{"listen":[":10100"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"172.31.91.222:10100"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}},"srv2":{"listen":[":10200"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"rate_limit","rate_limits":{"operator_bouncer":{"key":"static","max_events":100,"window":60000000000}}},{"handler":"reverse_proxy","upstreams":[{"dial":"172.31.91.222:10200"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}},"srv3":{"listen":[":11000"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"rate_limit","rate_limits":{"vin_diesel":{"key":"static","max_events":5,"window":1000000000}}},{"handler":"reverse_proxy","upstreams":[{"dial":"172.31.91.222:11000"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}},"srv4":{"listen":[":11100"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"172.31.91.222:10100"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}},"srv5":{"listen":[":11200"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"rate_limit","rate_limits":{"operator_bouncer":{"key":"static","max_events":100,"window":60000000000}}},{"handler":"reverse_proxy","upstreams":[{"dial":"172.31.91.222:11200"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}}
2024/05/20 07:15:16.448 INFO    http    enabling HTTP/3 listener        {"addr": ":10000"}
2024/05/20 07:15:16.448 DEBUG   http    starting server loop    {"address": "[::]:10000", "tls": true, "http3": true}
2024/05/20 07:15:16.448 INFO    http.log        server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:15:16.448 INFO    http    enabling HTTP/3 listener        {"addr": ":10100"}
2024/05/20 07:15:16.449 DEBUG   http    starting server loop    {"address": "[::]:10100", "tls": true, "http3": true}
2024/05/20 07:15:16.449 INFO    http.log        server running  {"name": "srv1", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:15:16.449 INFO    http    enabling HTTP/3 listener        {"addr": ":10200"}
2024/05/20 07:15:16.449 DEBUG   http    starting server loop    {"address": "[::]:10200", "tls": true, "http3": true}
2024/05/20 07:15:16.449 INFO    http.log        server running  {"name": "srv2", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:15:16.449 INFO    http    enabling HTTP/3 listener        {"addr": ":11000"}
2024/05/20 07:15:16.449 DEBUG   http    starting server loop    {"address": "[::]:11000", "tls": true, "http3": true}
2024/05/20 07:15:16.449 INFO    http.log        server running  {"name": "srv3", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:15:16.449 INFO    http    enabling HTTP/3 listener        {"addr": ":11100"}
2024/05/20 07:15:16.449 DEBUG   http    starting server loop    {"address": "[::]:11100", "tls": true, "http3": true}
2024/05/20 07:15:16.449 INFO    http.log        server running  {"name": "srv4", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:15:16.449 INFO    http    enabling HTTP/3 listener        {"addr": ":11200"}
2024/05/20 07:15:16.449 DEBUG   http    starting server loop    {"address": "[::]:11200", "tls": true, "http3": true}
2024/05/20 07:15:16.449 INFO    http.log        server running  {"name": "srv5", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:15:16.449 DEBUG   http    starting server loop    {"address": "[::]:80", "tls": false, "http3": false}
2024/05/20 07:15:16.449 INFO    http.log        server running  {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:15:16.449 INFO    http    enabling automatic TLS certificate management   {"domains": ["my.env.internal.domain.net"]}
2024/05/20 07:15:16.449 DEBUG   tls     loading managed certificate     {"domain": "my.env.internal.domain.net", "expiration": "2024/08/18 06:00:44.000", "issuer_key": "acme-v02.api.letsencrypt.org-directory", "storage": "FileStorage:/root/.local/share/caddy"}
2024/05/20 07:15:16.450 DEBUG   tls.cache       added certificate to cache      {"subjects": ["my.env.internal.domain.net"], "expiration": "2024/08/18 06:00:44.000", "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427", "cache_size": 1, "cache_capacity": 10000}
2024/05/20 07:15:16.450 DEBUG   events  event   {"name": "cached_managed_cert", "id": "98f1da84-8498-46ed-acd8-5f2c318fa605", "origin": "tls", "data": {"sans":["my.env.internal.domain.net"]}}
2024/05/20 07:15:16.450 INFO    autosaved config (load with --resume flag)      {"file": "/root/.config/caddy/autosave.json"}
2024/05/20 07:15:16.450 INFO    serving initial configuration
2024/05/20 07:15:16.450 WARN    tls     storage cleaning happened too recently; skipping for now        {"storage": "FileStorage:/root/.local/share/caddy", "instance": "65008f7c-e084-4c9d-b076-a99dd2cca441", "try_again": "2024/05/21 07:15:16.450", "try_again_in": 86399.99999959}
2024/05/20 07:15:16.450 INFO    tls     finished cleaning storage units
2024/05/20 07:16:03.603 DEBUG   events  event   {"name": "tls_get_certificate", "id": "ca7dbef5-7c42-4a0f-9b1c-8e62714b5f1a", "origin": "tls", "data": {"client_hello":{"CipherSuites":[47802,4865,4866,4867,49196,49195,52393,49200,49199,52392,49162,49161,49172,49171,157,156,53,47,49160,49170,10],"ServerName":"my.env.internal.domain.net","SupportedCurves":[56026,29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,515,2053,2053,1281,2054,1537,513],"SupportedProtos":["http/1.1"],"SupportedVersions":[10794,772,771,770,769],"RemoteAddr":{"IP":"104.50.3.130","Port":58381,"Zone":""},"LocalAddr":{"IP":"172.31.87.3","Port":10000,"Zone":""}}}}
2024/05/20 07:16:03.604 DEBUG   tls.handshake   choosing certificate    {"identifier": "my.env.internal.domain.net", "num_choices": 1}
2024/05/20 07:16:03.604 DEBUG   tls.handshake   default certificate selection results   {"identifier": "my.env.internal.domain.net", "subjects": ["my.env.internal.domain.net"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:03.604 DEBUG   tls.handshake   matched certificate in cache    {"remote_ip": "104.50.3.130", "remote_port": "58381", "subjects": ["my.env.internal.domain.net"], "managed": true, "expiration": "2024/08/18 06:00:44.000", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:03.711 DEBUG   http.handlers.reverse_proxy     selected upstream       {"dial": "172.31.91.222:10000", "total_upstreams": 1}
2024/05/20 07:16:03.846 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "172.31.91.222:10000", "duration": 0.134716008, "request": {"remote_ip": "104.50.3.130", "remote_port": "58381", "client_ip": "104.50.3.130", "proto": "HTTP/1.1", "method": "GET", "host": "my.env.internal.domain.net:10000", "uri": "/v1/sdk-core", "headers": {"Content-Type": ["application/json"], "Accept": ["application/json"], "User-Agent": ["X-UnrealEngine-Agent"], "Accept-Language": ["en-US,en;q=0.9"], "X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["my.env.internal.domain.net:10000"], "Accept-Encoding": ["gzip, deflate, br"], "X-Forwarded-For": ["104.50.3.130"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "http/1.1", "server_name": "my.env.internal.domain.net"}}, "headers": {"Vary": ["Origin"], "Content-Length": ["17"], "Content-Type": ["application/json; charset=UTF-8"]}, "status": 200}
2024/05/20 07:16:03.962 DEBUG   http.handlers.reverse_proxy     selected upstream       {"dial": "172.31.91.222:10000", "total_upstreams": 1}
2024/05/20 07:16:04.035 DEBUG   events  event   {"name": "tls_get_certificate", "id": "c2ca1a0b-ac9a-46e4-acbc-242dd92e8e56", "origin": "tls", "data": {"client_hello":{"CipherSuites":[31354,4865,4866,4867,49196,49195,52393,49200,49199,52392,49162,49161,49172,49171,157,156,53,47,49160,49170,10],"ServerName":"my.env.internal.domain.net","SupportedCurves":[35466,29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,515,2053,2053,1281,2054,1537,513],"SupportedProtos":["http/1.1"],"SupportedVersions":[51914,772,771,770,769],"RemoteAddr":{"IP":"104.50.3.130","Port":58382,"Zone":""},"LocalAddr":{"IP":"172.31.87.3","Port":10000,"Zone":""}}}}
2024/05/20 07:16:04.035 DEBUG   tls.handshake   choosing certificate    {"identifier": "my.env.internal.domain.net", "num_choices": 1}
2024/05/20 07:16:04.035 DEBUG   tls.handshake   default certificate selection results   {"identifier": "my.env.internal.domain.net", "subjects": ["my.env.internal.domain.net"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:04.035 DEBUG   tls.handshake   matched certificate in cache    {"remote_ip": "104.50.3.130", "remote_port": "58382", "subjects": ["my.env.internal.domain.net"], "managed": true, "expiration": "2024/08/18 06:00:44.000", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:04.099 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "172.31.91.222:10000", "duration": 0.136561194, "request": {"remote_ip": "104.50.3.130", "remote_port": "58381", "client_ip": "104.50.3.130", "proto": "HTTP/1.1", "method": "GET", "host": "my.env.internal.domain.net:10000", "uri": "/v1/info", "headers": {"Accept-Encoding": ["gzip, deflate, br"], "X-Forwarded-For": ["104.50.3.130"], "X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["my.env.internal.domain.net:10000"], "Content-Type": ["application/json"], "Accept": ["application/json"], "User-Agent": ["X-UnrealEngine-Agent"], "Accept-Language": ["en-US,en;q=0.9"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "http/1.1", "server_name": "my.env.internal.domain.net"}}, "headers": {"Vary": ["Origin"], "Content-Length": ["528"], "Content-Type": ["application/json; charset=UTF-8"]}, "status": 200}
2024/05/20 07:16:04.111 DEBUG   http.handlers.reverse_proxy     selected upstream       {"dial": "172.31.91.222:10000", "total_upstreams": 1}
2024/05/20 07:16:04.114 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "172.31.91.222:10000", "duration": 0.002919186, "request": {"remote_ip": "104.50.3.130", "remote_port": "58382", "client_ip": "104.50.3.130", "proto": "HTTP/1.1", "method": "GET", "host": "my.env.internal.domain.net:10000", "uri": "/v1/types", "headers": {"X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["my.env.internal.domain.net:10000"], "X-Forwarded-For": ["104.50.3.130"], "Content-Type": ["application/json"], "Accept": ["application/json"], "User-Agent": ["X-UnrealEngine-Agent"], "Accept-Language": ["en-US,en;q=0.9"], "Accept-Encoding": ["gzip, deflate, br"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "http/1.1", "server_name": "my.env.internal.domain.net"}}, "headers": {"Vary": ["Origin"], "Content-Length": ["9984"], "Content-Type": ["application/json; charset=UTF-8"]}, "status": 200}
2024/05/20 07:16:04.227 DEBUG   http.handlers.reverse_proxy     selected upstream       {"dial": "172.31.91.222:10000", "total_upstreams": 1}
2024/05/20 07:16:04.305 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "172.31.91.222:10000", "duration": 0.077972769, "request": {"remote_ip": "104.50.3.130", "remote_port": "58381", "client_ip": "104.50.3.130", "proto": "HTTP/1.1", "method": "GET", "host": "my.env.internal.domain.net:10000", "uri": "/v1/loginqueue/getinqueuev1", "headers": {"Authorization": [], "Accept-Language": ["en-US,en;q=0.9"], "Content-Type": ["application/json"], "X-Forwarded-For": ["104.50.3.130"], "Accept": ["application/json"], "X-Forwarded-Host": ["my.env.internal.domain.net:10000"], "User-Agent": ["X-UnrealEngine-Agent"], "Accept-Encoding": ["gzip, deflate, br"], "X-Forwarded-Proto": ["https"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "http/1.1", "server_name": "my.env.internal.domain.net"}}, "headers": {"Vary": ["Origin"], "Content-Length": ["659"], "Content-Type": ["application/json; charset=UTF-8"]}, "status": 200}
2024/05/20 07:16:04.473 DEBUG   events  event   {"name": "tls_get_certificate", "id": "e9a1b317-78fc-4151-9b43-88a7614989d5", "origin": "tls", "data": {"client_hello":{"CipherSuites":[39578,4865,4866,4867,49196,49195,52393,49200,49199,52392,49162,49161,49172,49171,157,156,53,47,49160,49170,10],"ServerName":"my.env.internal.domain.net","SupportedCurves":[2570,29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,515,2053,2053,1281,2054,1537,513],"SupportedProtos":["http/1.1"],"SupportedVersions":[10794,772,771,770,769],"RemoteAddr":{"IP":"104.50.3.130","Port":58383,"Zone":""},"LocalAddr":{"IP":"172.31.87.3","Port":11000,"Zone":""}}}}
2024/05/20 07:16:04.473 DEBUG   tls.handshake   choosing certificate    {"identifier": "my.env.internal.domain.net", "num_choices": 1}
2024/05/20 07:16:04.473 DEBUG   tls.handshake   default certificate selection results   {"identifier": "my.env.internal.domain.net", "subjects": ["my.env.internal.domain.net"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:04.473 DEBUG   tls.handshake   matched certificate in cache    {"remote_ip": "104.50.3.130", "remote_port": "58383", "subjects": ["my.env.internal.domain.net"], "managed": true, "expiration": "2024/08/18 06:00:44.000", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:04.554 DEBUG   http.handlers.reverse_proxy     selected upstream       {"dial": "172.31.91.222:11000", "total_upstreams": 1}
2024/05/20 07:16:04.853 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "172.31.91.222:11000", "duration": 0.298466029, "request": {"remote_ip": "104.50.3.130", "remote_port": "58383", "client_ip": "104.50.3.130", "proto": "HTTP/1.1", "method": "POST", "host": "my.env.internal.domain.net:11000", "uri": "/v1/account/authenticateorcreatev2", "headers": {"X-Forwarded-For": ["104.50.3.130"], "Accept": ["application/json"], "Authorization": [], "Accept-Language": ["en-US,en;q=0.9"], "X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["my.env.internal.domain.net:11000"], "Content-Length": ["1003"], "User-Agent": ["X-UnrealEngine-Agent"], "Content-Type": ["application/json"], "Accept-Encoding": ["gzip, deflate, br"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "http/1.1", "server_name": "my.env.internal.domain.net"}}, "headers": {"Content-Type": ["application/json; charset=UTF-8"], "Vary": ["Origin"], "Content-Length": ["1907"]}, "status": 200}
2024/05/20 07:16:05.127 DEBUG   events  event   {"name": "tls_get_certificate", "id": "cdf2ae70-798c-4a67-9683-f4b1946884bd", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"my.env.internal.domain.net","SupportedCurves":[29,23,30,25,24],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,515,769,513,770,514,1026,1282,1538],"SupportedProtos":["http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"104.50.3.130","Port":49413,"Zone":""},"LocalAddr":{"IP":"172.31.87.3","Port":11000,"Zone":""}}}}
2024/05/20 07:16:05.127 DEBUG   tls.handshake   choosing certificate    {"identifier": "my.env.internal.domain.net", "num_choices": 1}
2024/05/20 07:16:05.127 DEBUG   tls.handshake   default certificate selection results   {"identifier": "my.env.internal.domain.net", "subjects": ["my.env.internal.domain.net"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:05.127 DEBUG   tls.handshake   matched certificate in cache    {"remote_ip": "104.50.3.130", "remote_port": "49413", "subjects": ["my.env.internal.domain.net"], "managed": true, "expiration": "2024/08/18 06:00:44.000", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:05.128 DEBUG   events  event   {"name": "tls_get_certificate", "id": "60ea49ed-0643-4ae7-ac97-a823777efd62", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"my.env.internal.domain.net","SupportedCurves":[29,23,30,25,24],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,515,769,513,770,514,1026,1282,1538],"SupportedProtos":["http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"104.50.3.130","Port":49414,"Zone":""},"LocalAddr":{"IP":"172.31.87.3","Port":10000,"Zone":""}}}}
2024/05/20 07:16:05.128 DEBUG   tls.handshake   choosing certificate    {"identifier": "my.env.internal.domain.net", "num_choices": 1}
2024/05/20 07:16:05.128 DEBUG   tls.handshake   default certificate selection results   {"identifier": "my.env.internal.domain.net", "subjects": ["my.env.internal.domain.net"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:05.128 DEBUG   tls.handshake   matched certificate in cache    {"remote_ip": "104.50.3.130", "remote_port": "49414", "subjects": ["my.env.internal.domain.net"], "managed": true, "expiration": "2024/08/18 06:00:44.000", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:05.230 DEBUG   http.stdlib     http: TLS handshake error from 104.50.3.130:49413: local error: tls: bad record MAC
2024/05/20 07:16:05.230 DEBUG   http.stdlib     http: TLS handshake error from 104.50.3.130:49414: local error: tls: bad record MAC
2024/05/20 07:16:06.410 DEBUG   events  event   {"name": "tls_get_certificate", "id": "b40e428f-f861-431f-a388-d14777defd24", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"my.env.internal.domain.net","SupportedCurves":[29,23,30,25,24],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,515,769,513,770,514,1026,1282,1538],"SupportedProtos":["http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"104.50.3.130","Port":49415,"Zone":""},"LocalAddr":{"IP":"172.31.87.3","Port":11000,"Zone":""}}}}
2024/05/20 07:16:06.410 DEBUG   tls.handshake   choosing certificate    {"identifier": "my.env.internal.domain.net", "num_choices": 1}
2024/05/20 07:16:06.410 DEBUG   tls.handshake   default certificate selection results   {"identifier": "my.env.internal.domain.net", "subjects": ["my.env.internal.domain.net"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:06.410 DEBUG   tls.handshake   matched certificate in cache    {"remote_ip": "104.50.3.130", "remote_port": "49415", "subjects": ["my.env.internal.domain.net"], "managed": true, "expiration": "2024/08/18 06:00:44.000", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:06.411 DEBUG   events  event   {"name": "tls_get_certificate", "id": "b25ca310-a9b3-48d5-b8cb-f0cb2adae281", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"my.env.internal.domain.net","SupportedCurves":[29,23,30,25,24],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,515,769,513,770,514,1026,1282,1538],"SupportedProtos":["http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"104.50.3.130","Port":49416,"Zone":""},"LocalAddr":{"IP":"172.31.87.3","Port":10000,"Zone":""}}}}
2024/05/20 07:16:06.411 DEBUG   tls.handshake   choosing certificate    {"identifier": "my.env.internal.domain.net", "num_choices": 1}
2024/05/20 07:16:06.411 DEBUG   tls.handshake   default certificate selection results   {"identifier": "my.env.internal.domain.net", "subjects": ["my.env.internal.domain.net"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:06.411 DEBUG   tls.handshake   matched certificate in cache    {"remote_ip": "104.50.3.130", "remote_port": "49416", "subjects": ["my.env.internal.domain.net"], "managed": true, "expiration": "2024/08/18 06:00:44.000", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:06.515 DEBUG   http.stdlib     http: TLS handshake error from 104.50.3.130:49415: local error: tls: bad record MAC
2024/05/20 07:16:06.516 DEBUG   http.stdlib     http: TLS handshake error from 104.50.3.130:49416: local error: tls: bad record MAC
2024/05/20 07:16:07.870 DEBUG   events  event   {"name": "tls_get_certificate", "id": "63cb48cd-23ca-4737-8cb4-755d2aceee93", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"my.env.internal.domain.net","SupportedCurves":[29,23,30,25,24],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,515,769,513,770,514,1026,1282,1538],"SupportedProtos":["http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"104.50.3.130","Port":49417,"Zone":""},"LocalAddr":{"IP":"172.31.87.3","Port":11000,"Zone":""}}}}
2024/05/20 07:16:07.870 DEBUG   tls.handshake   choosing certificate    {"identifier": "my.env.internal.domain.net", "num_choices": 1}
2024/05/20 07:16:07.870 DEBUG   tls.handshake   default certificate selection results   {"identifier": "my.env.internal.domain.net", "subjects": ["my.env.internal.domain.net"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:07.870 DEBUG   tls.handshake   matched certificate in cache    {"remote_ip": "104.50.3.130", "remote_port": "49417", "subjects": ["my.env.internal.domain.net"], "managed": true, "expiration": "2024/08/18 06:00:44.000", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:08.057 DEBUG   http.stdlib     http: TLS handshake error from 104.50.3.130:49417: local error: tls: bad record MAC
2024/05/20 07:16:08.131 DEBUG   events  event   {"name": "tls_get_certificate", "id": "5afa7ded-92d3-4401-ace9-9b383f38fa7c", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"my.env.internal.domain.net","SupportedCurves":[29,23,30,25,24],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,515,769,513,770,514,1026,1282,1538],"SupportedProtos":["http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"104.50.3.130","Port":49418,"Zone":""},"LocalAddr":{"IP":"172.31.87.3","Port":10000,"Zone":""}}}}
2024/05/20 07:16:08.131 DEBUG   tls.handshake   choosing certificate    {"identifier": "my.env.internal.domain.net", "num_choices": 1}
2024/05/20 07:16:08.131 DEBUG   tls.handshake   default certificate selection results   {"identifier": "my.env.internal.domain.net", "subjects": ["my.env.internal.domain.net"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:08.131 DEBUG   tls.handshake   matched certificate in cache    {"remote_ip": "104.50.3.130", "remote_port": "49418", "subjects": ["my.env.internal.domain.net"], "managed": true, "expiration": "2024/08/18 06:00:44.000", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:08.238 DEBUG   http.stdlib     http: TLS handshake error from 104.50.3.130:49418: local error: tls: bad record MAC

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

sudo dnf update -y
sudo yum install nss-tools

curl -L -o caddy https://dc0eql1zxfug.cloudfront.net/caddy-with-rate-limiting-AL2023 # Download our version of Caddy, bundled with caddy-rate-limit built for AL2023
chmod +x caddy
sudo mv caddy /usr/bin/

sudo mkdir /etc/caddy
echo "${caddyfileAsText}" | sudo tee /etc/caddy/Caddyfile

echo "${caddyServiceFileAsText}" | sudo tee /etc/systemd/system/caddy.service
sudo systemctl enable caddy.service
sudo systemctl start caddy.service

sudo caddy trust

a. System environment:

Amazon Linux 2023 6.1.90-99.173.amzn2023.aarch64

b. Command:

sudo systemctl start caddy.service

c. Service/unit/compose file:

caddy.service:

[Unit]
Description=Caddy
After=network-online.target

[Service]
User=root
Group=root
ExecStart=/usr/bin/caddy run --config /etc/caddy/Caddyfile
Restart=on-failure
RestartSec=15s # Delay restart by 15 seconds

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

I’m sorry I’m truly unable to share the full domain name at this time, but can share the current IP.

{
    order rate_limit before basicauth

    log {
        format console
        level debug
        output file /home/caddy/logs/caddy.log {
            roll_uncompressed
            roll_keep 30
            roll_keep_for 30d
        }
    }
}

# Common rate_limit block
(player_rate_limit_settings) {
    # Docs: https://github.com/mholt/caddy-ratelimit
    rate_limit {
        zone vin_diesel {
            key    static # Only one rate limiter
            events 5
            window 1s
        }
    }
}

# Common rate_limit block
(operator_rate_limit_settings) {
    # Docs: https://github.com/mholt/caddy-ratelimit
    rate_limit {
        zone operator_bouncer {
            key    static # Only one rate limiter
            events 100
            window 1m
        }
    }
}

# Route traffic from ports 10000, 11000, 10200, and 11200 to the IP address 172.31.91.222
my.env.internal.domain.net:10000 {
    import player_rate_limit_settings
    reverse_proxy 172.31.91.222:10000
}

my.env.internal.domain.net:11000 {
    import player_rate_limit_settings
    reverse_proxy 172.31.91.222:11000
}

my.env.internal.domain.net:10200 {
    import operator_rate_limit_settings
    reverse_proxy 172.31.91.222:10200
}

my.env.internal.domain.net:11200 {
    import operator_rate_limit_settings
    reverse_proxy 172.31.91.222:11200
}

my.env.internal.domain.net:10100 {
    reverse_proxy 172.31.91.222:10100
}

my.env.internal.domain.net:11100 {
    reverse_proxy 172.31.91.222:10100
}

5. Links to relevant resources:

https://docs.unrealengine.com/4.27/en-US/API/Runtime/WebSockets/

I managed to fix it by adding:

[/Script/Engine.NetworkSettings]
n.VerifyPeer=true

to my Engine.ini file.

We do deploy pre-prod builds through TestFairy - I wonder if bypassing the iOS App Store somehow was the culprit, since all apps on the App Store should trust LetsEncrypt as a valid CA.

By doing this, you’re disabling all security. HTTPS is now meaningless. It means someone could man-in-the-middle the connection, intercepting the connection, reading or manipulating it, and using their own certificate to re-encrypt the connection, and you would have no way to tell that it was tampered with.

I’m not sure why you’re getting that error (must be somekind of client TLS misconfiguration, I dunno) but turning off verification is absolutely not the right thing to do.

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.