1. The problem I’m having:
I am attempting to establish a websocket connection via an iOS client through Caddy.
The iOS client is built with Unreal 4.27.
Normal HTTPS requests work on iOS.
Both HTTPS requests and websocket connections work on Windows via a built version of the game.
Websockets can connect to the server that Caddy is routing to directly by bypassing Caddy (non-wss, no cert on the server).
If I do not enable WSS, and allow for insecure websockets, I can connect through Caddy.
Since Caddy uses LetsEncrypt as a CA, this should work without needing to bake in a cert on the client app as far as I know (am I wrong here?).
Is there something I’m doing wrong here, or should this be working? Thanks!
2. Error messages and/or full log output:
Client error log message:
LogWebSockets: Warning: Lws(Error): SSL error: unable to get local issuer certificate (preverify_ok=0;err=20;depth=1)
Entire Caddy log:
2024/05/20 07:00:42.702 INFO admin admin endpoint started {"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2024/05/20 07:00:42.705 INFO http.auto_https enabling automatic HTTP->HTTPS redirects {"server_name": "srv0"}
2024/05/20 07:00:42.705 INFO http.auto_https enabling automatic HTTP->HTTPS redirects {"server_name": "srv1"}
2024/05/20 07:00:42.705 INFO http.auto_https enabling automatic HTTP->HTTPS redirects {"server_name": "srv2"}
2024/05/20 07:00:42.705 INFO http.auto_https enabling automatic HTTP->HTTPS redirects {"server_name": "srv3"}
2024/05/20 07:00:42.705 INFO http.auto_https enabling automatic HTTP->HTTPS redirects {"server_name": "srv4"}
2024/05/20 07:00:42.705 INFO http.auto_https enabling automatic HTTP->HTTPS redirects {"server_name": "srv5"}
2024/05/20 07:00:42.709 INFO http enabling HTTP/3 listener {"addr": ":10100"}
2024/05/20 07:00:42.709 INFO http.log server running {"name": "srv1", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:00:42.709 INFO http enabling HTTP/3 listener {"addr": ":10200"}
2024/05/20 07:00:42.709 INFO http.log server running {"name": "srv2", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:00:42.709 INFO http enabling HTTP/3 listener {"addr": ":11000"}
2024/05/20 07:00:42.709 INFO http.log server running {"name": "srv3", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:00:42.709 INFO http enabling HTTP/3 listener {"addr": ":11100"}
2024/05/20 07:00:42.709 INFO http.log server running {"name": "srv4", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:00:42.709 INFO http enabling HTTP/3 listener {"addr": ":11200"}
2024/05/20 07:00:42.710 INFO http.log server running {"name": "srv5", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:00:42.710 INFO http.log server running {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:00:42.710 INFO http enabling HTTP/3 listener {"addr": ":10000"}
2024/05/20 07:00:42.710 INFO tls.cache.maintenance started background certificate maintenance {"cache": "0x4000496c80"}
2024/05/20 07:00:42.711 INFO http.log server running {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:00:42.711 INFO http enabling automatic TLS certificate management {"domains": ["my.env.internal.domain.net"]}
2024/05/20 07:00:42.711 INFO autosaved config (load with --resume flag) {"file": "/root/.config/caddy/autosave.json"}
2024/05/20 07:00:42.711 INFO serving initial configuration
2024/05/20 07:00:42.711 WARN tls unable to get instance ID; storage clean stamps will be incomplete {"error": "open /root/.local/share/caddy/instance.uuid: no such file or directory"}
2024/05/20 07:00:42.711 INFO tls.obtain acquiring lock {"identifier": "my.env.internal.domain.net"}
2024/05/20 07:00:42.713 INFO tls cleaning storage unit {"storage": "FileStorage:/root/.local/share/caddy"}
2024/05/20 07:00:42.713 INFO tls finished cleaning storage units
2024/05/20 07:00:42.713 INFO tls.obtain lock acquired {"identifier": "my.env.internal.domain.net"}
2024/05/20 07:00:42.713 INFO tls.obtain obtaining certificate {"identifier": "my.env.internal.domain.net"}
2024/05/20 07:00:42.723 INFO admin.api received request {"method": "GET", "host": "localhost:2019", "uri": "/pki/ca/local", "remote_ip": "127.0.0.1", "remote_port": "37166", "headers": {"Accept-Encoding":["gzip"],"Origin":["http://localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
2024/05/20 07:00:43.078 INFO http waiting on internal rate limiter {"identifiers": ["my.env.internal.domain.net"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2024/05/20 07:00:43.078 INFO http done waiting on internal rate limiter {"identifiers": ["my.env.internal.domain.net"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2024/05/20 07:00:43.403 INFO http.acme_client trying to solve challenge {"identifier": "my.env.internal.domain.net", "challenge_type": "http-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2024/05/20 07:00:43.550 INFO http served key authentication {"identifier": "my.env.internal.domain.net", "challenge": "http-01", "remote": "3.129.25.162:48210", "distributed": false}
2024/05/20 07:00:43.736 INFO http served key authentication {"identifier": "my.env.internal.domain.net", "challenge": "http-01", "remote": "23.178.112.107:45455", "distributed": false}
2024/05/20 07:00:43.746 INFO http served key authentication {"identifier": "my.env.internal.domain.net", "challenge": "http-01", "remote": "16.170.255.201:30194", "distributed": false}
2024/05/20 07:00:43.759 INFO http served key authentication {"identifier": "my.env.internal.domain.net", "challenge": "http-01", "remote": "18.237.3.70:30254", "distributed": false}
2024/05/20 07:00:43.986 INFO http served key authentication {"identifier": "my.env.internal.domain.net", "challenge": "http-01", "remote": "13.212.231.193:16712", "distributed": false}
2024/05/20 07:00:44.094 INFO http.acme_client authorization finalized {"identifier": "my.env.internal.domain.net", "authz_status": "valid"}
2024/05/20 07:00:44.094 INFO http.acme_client validations succeeded; finalizing order {"order": "https://acme-v02.api.letsencrypt.org/acme/order/1735754592/270951627302"}
2024/05/20 07:00:44.951 INFO http.acme_client successfully downloaded available certificate chains {"count": 2, "first_url": "https://acme-v02.api.letsencrypt.org/acme/cert/0306bc40dbe9f9c0a9438f86261d9909bf9b"}
2024/05/20 07:00:44.951 INFO tls.obtain certificate obtained successfully {"identifier": "my.env.internal.domain.net"}
2024/05/20 07:00:44.951 INFO tls.obtain releasing lock {"identifier": "my.env.internal.domain.net"}
2024/05/20 07:15:16.127 INFO shutting down apps, then terminating {"signal": "SIGTERM"}
2024/05/20 07:15:16.127 WARN exiting; byeee!! 👋 {"signal": "SIGTERM"}
2024/05/20 07:15:16.127 INFO http servers shutting down with eternal grace period
2024/05/20 07:15:16.127 INFO admin stopped previous server {"address": "localhost:2019"}
2024/05/20 07:15:16.128 INFO shutdown complete {"signal": "SIGTERM", "exit_code": 0}
2024/05/20 07:15:16.446 INFO admin admin endpoint started {"address": "localhost:2019", "enforce_origin": false, "origins": ["//127.0.0.1:2019", "//localhost:2019", "//[::1]:2019"]}
2024/05/20 07:15:16.447 INFO http.auto_https enabling automatic HTTP->HTTPS redirects {"server_name": "srv1"}
2024/05/20 07:15:16.447 INFO http.auto_https enabling automatic HTTP->HTTPS redirects {"server_name": "srv2"}
2024/05/20 07:15:16.447 INFO http.auto_https enabling automatic HTTP->HTTPS redirects {"server_name": "srv3"}
2024/05/20 07:15:16.447 INFO http.auto_https enabling automatic HTTP->HTTPS redirects {"server_name": "srv4"}
2024/05/20 07:15:16.447 INFO http.auto_https enabling automatic HTTP->HTTPS redirects {"server_name": "srv5"}
2024/05/20 07:15:16.447 INFO http.auto_https enabling automatic HTTP->HTTPS redirects {"server_name": "srv0"}
2024/05/20 07:15:16.447 INFO tls.cache.maintenance started background certificate maintenance {"cache": "0x40004e2880"}
2024/05/20 07:15:16.447 DEBUG http.auto_https adjusted config {"tls": {"automation":{"policies":[{}]}}, "http": {"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":10000"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"rate_limit","rate_limits":{"vin_diesel":{"key":"static","max_events":5,"window":1000000000}}},{"handler":"reverse_proxy","upstreams":[{"dial":"172.31.91.222:10000"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}},"srv1":{"listen":[":10100"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"172.31.91.222:10100"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}},"srv2":{"listen":[":10200"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"rate_limit","rate_limits":{"operator_bouncer":{"key":"static","max_events":100,"window":60000000000}}},{"handler":"reverse_proxy","upstreams":[{"dial":"172.31.91.222:10200"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}},"srv3":{"listen":[":11000"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"rate_limit","rate_limits":{"vin_diesel":{"key":"static","max_events":5,"window":1000000000}}},{"handler":"reverse_proxy","upstreams":[{"dial":"172.31.91.222:11000"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}},"srv4":{"listen":[":11100"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"172.31.91.222:10100"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}},"srv5":{"listen":[":11200"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"rate_limit","rate_limits":{"operator_bouncer":{"key":"static","max_events":100,"window":60000000000}}},{"handler":"reverse_proxy","upstreams":[{"dial":"172.31.91.222:11200"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}}
2024/05/20 07:15:16.448 INFO http enabling HTTP/3 listener {"addr": ":10000"}
2024/05/20 07:15:16.448 DEBUG http starting server loop {"address": "[::]:10000", "tls": true, "http3": true}
2024/05/20 07:15:16.448 INFO http.log server running {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:15:16.448 INFO http enabling HTTP/3 listener {"addr": ":10100"}
2024/05/20 07:15:16.449 DEBUG http starting server loop {"address": "[::]:10100", "tls": true, "http3": true}
2024/05/20 07:15:16.449 INFO http.log server running {"name": "srv1", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:15:16.449 INFO http enabling HTTP/3 listener {"addr": ":10200"}
2024/05/20 07:15:16.449 DEBUG http starting server loop {"address": "[::]:10200", "tls": true, "http3": true}
2024/05/20 07:15:16.449 INFO http.log server running {"name": "srv2", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:15:16.449 INFO http enabling HTTP/3 listener {"addr": ":11000"}
2024/05/20 07:15:16.449 DEBUG http starting server loop {"address": "[::]:11000", "tls": true, "http3": true}
2024/05/20 07:15:16.449 INFO http.log server running {"name": "srv3", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:15:16.449 INFO http enabling HTTP/3 listener {"addr": ":11100"}
2024/05/20 07:15:16.449 DEBUG http starting server loop {"address": "[::]:11100", "tls": true, "http3": true}
2024/05/20 07:15:16.449 INFO http.log server running {"name": "srv4", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:15:16.449 INFO http enabling HTTP/3 listener {"addr": ":11200"}
2024/05/20 07:15:16.449 DEBUG http starting server loop {"address": "[::]:11200", "tls": true, "http3": true}
2024/05/20 07:15:16.449 INFO http.log server running {"name": "srv5", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:15:16.449 DEBUG http starting server loop {"address": "[::]:80", "tls": false, "http3": false}
2024/05/20 07:15:16.449 INFO http.log server running {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2024/05/20 07:15:16.449 INFO http enabling automatic TLS certificate management {"domains": ["my.env.internal.domain.net"]}
2024/05/20 07:15:16.449 DEBUG tls loading managed certificate {"domain": "my.env.internal.domain.net", "expiration": "2024/08/18 06:00:44.000", "issuer_key": "acme-v02.api.letsencrypt.org-directory", "storage": "FileStorage:/root/.local/share/caddy"}
2024/05/20 07:15:16.450 DEBUG tls.cache added certificate to cache {"subjects": ["my.env.internal.domain.net"], "expiration": "2024/08/18 06:00:44.000", "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427", "cache_size": 1, "cache_capacity": 10000}
2024/05/20 07:15:16.450 DEBUG events event {"name": "cached_managed_cert", "id": "98f1da84-8498-46ed-acd8-5f2c318fa605", "origin": "tls", "data": {"sans":["my.env.internal.domain.net"]}}
2024/05/20 07:15:16.450 INFO autosaved config (load with --resume flag) {"file": "/root/.config/caddy/autosave.json"}
2024/05/20 07:15:16.450 INFO serving initial configuration
2024/05/20 07:15:16.450 WARN tls storage cleaning happened too recently; skipping for now {"storage": "FileStorage:/root/.local/share/caddy", "instance": "65008f7c-e084-4c9d-b076-a99dd2cca441", "try_again": "2024/05/21 07:15:16.450", "try_again_in": 86399.99999959}
2024/05/20 07:15:16.450 INFO tls finished cleaning storage units
2024/05/20 07:16:03.603 DEBUG events event {"name": "tls_get_certificate", "id": "ca7dbef5-7c42-4a0f-9b1c-8e62714b5f1a", "origin": "tls", "data": {"client_hello":{"CipherSuites":[47802,4865,4866,4867,49196,49195,52393,49200,49199,52392,49162,49161,49172,49171,157,156,53,47,49160,49170,10],"ServerName":"my.env.internal.domain.net","SupportedCurves":[56026,29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,515,2053,2053,1281,2054,1537,513],"SupportedProtos":["http/1.1"],"SupportedVersions":[10794,772,771,770,769],"RemoteAddr":{"IP":"104.50.3.130","Port":58381,"Zone":""},"LocalAddr":{"IP":"172.31.87.3","Port":10000,"Zone":""}}}}
2024/05/20 07:16:03.604 DEBUG tls.handshake choosing certificate {"identifier": "my.env.internal.domain.net", "num_choices": 1}
2024/05/20 07:16:03.604 DEBUG tls.handshake default certificate selection results {"identifier": "my.env.internal.domain.net", "subjects": ["my.env.internal.domain.net"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:03.604 DEBUG tls.handshake matched certificate in cache {"remote_ip": "104.50.3.130", "remote_port": "58381", "subjects": ["my.env.internal.domain.net"], "managed": true, "expiration": "2024/08/18 06:00:44.000", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:03.711 DEBUG http.handlers.reverse_proxy selected upstream {"dial": "172.31.91.222:10000", "total_upstreams": 1}
2024/05/20 07:16:03.846 DEBUG http.handlers.reverse_proxy upstream roundtrip {"upstream": "172.31.91.222:10000", "duration": 0.134716008, "request": {"remote_ip": "104.50.3.130", "remote_port": "58381", "client_ip": "104.50.3.130", "proto": "HTTP/1.1", "method": "GET", "host": "my.env.internal.domain.net:10000", "uri": "/v1/sdk-core", "headers": {"Content-Type": ["application/json"], "Accept": ["application/json"], "User-Agent": ["X-UnrealEngine-Agent"], "Accept-Language": ["en-US,en;q=0.9"], "X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["my.env.internal.domain.net:10000"], "Accept-Encoding": ["gzip, deflate, br"], "X-Forwarded-For": ["104.50.3.130"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "http/1.1", "server_name": "my.env.internal.domain.net"}}, "headers": {"Vary": ["Origin"], "Content-Length": ["17"], "Content-Type": ["application/json; charset=UTF-8"]}, "status": 200}
2024/05/20 07:16:03.962 DEBUG http.handlers.reverse_proxy selected upstream {"dial": "172.31.91.222:10000", "total_upstreams": 1}
2024/05/20 07:16:04.035 DEBUG events event {"name": "tls_get_certificate", "id": "c2ca1a0b-ac9a-46e4-acbc-242dd92e8e56", "origin": "tls", "data": {"client_hello":{"CipherSuites":[31354,4865,4866,4867,49196,49195,52393,49200,49199,52392,49162,49161,49172,49171,157,156,53,47,49160,49170,10],"ServerName":"my.env.internal.domain.net","SupportedCurves":[35466,29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,515,2053,2053,1281,2054,1537,513],"SupportedProtos":["http/1.1"],"SupportedVersions":[51914,772,771,770,769],"RemoteAddr":{"IP":"104.50.3.130","Port":58382,"Zone":""},"LocalAddr":{"IP":"172.31.87.3","Port":10000,"Zone":""}}}}
2024/05/20 07:16:04.035 DEBUG tls.handshake choosing certificate {"identifier": "my.env.internal.domain.net", "num_choices": 1}
2024/05/20 07:16:04.035 DEBUG tls.handshake default certificate selection results {"identifier": "my.env.internal.domain.net", "subjects": ["my.env.internal.domain.net"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:04.035 DEBUG tls.handshake matched certificate in cache {"remote_ip": "104.50.3.130", "remote_port": "58382", "subjects": ["my.env.internal.domain.net"], "managed": true, "expiration": "2024/08/18 06:00:44.000", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:04.099 DEBUG http.handlers.reverse_proxy upstream roundtrip {"upstream": "172.31.91.222:10000", "duration": 0.136561194, "request": {"remote_ip": "104.50.3.130", "remote_port": "58381", "client_ip": "104.50.3.130", "proto": "HTTP/1.1", "method": "GET", "host": "my.env.internal.domain.net:10000", "uri": "/v1/info", "headers": {"Accept-Encoding": ["gzip, deflate, br"], "X-Forwarded-For": ["104.50.3.130"], "X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["my.env.internal.domain.net:10000"], "Content-Type": ["application/json"], "Accept": ["application/json"], "User-Agent": ["X-UnrealEngine-Agent"], "Accept-Language": ["en-US,en;q=0.9"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "http/1.1", "server_name": "my.env.internal.domain.net"}}, "headers": {"Vary": ["Origin"], "Content-Length": ["528"], "Content-Type": ["application/json; charset=UTF-8"]}, "status": 200}
2024/05/20 07:16:04.111 DEBUG http.handlers.reverse_proxy selected upstream {"dial": "172.31.91.222:10000", "total_upstreams": 1}
2024/05/20 07:16:04.114 DEBUG http.handlers.reverse_proxy upstream roundtrip {"upstream": "172.31.91.222:10000", "duration": 0.002919186, "request": {"remote_ip": "104.50.3.130", "remote_port": "58382", "client_ip": "104.50.3.130", "proto": "HTTP/1.1", "method": "GET", "host": "my.env.internal.domain.net:10000", "uri": "/v1/types", "headers": {"X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["my.env.internal.domain.net:10000"], "X-Forwarded-For": ["104.50.3.130"], "Content-Type": ["application/json"], "Accept": ["application/json"], "User-Agent": ["X-UnrealEngine-Agent"], "Accept-Language": ["en-US,en;q=0.9"], "Accept-Encoding": ["gzip, deflate, br"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "http/1.1", "server_name": "my.env.internal.domain.net"}}, "headers": {"Vary": ["Origin"], "Content-Length": ["9984"], "Content-Type": ["application/json; charset=UTF-8"]}, "status": 200}
2024/05/20 07:16:04.227 DEBUG http.handlers.reverse_proxy selected upstream {"dial": "172.31.91.222:10000", "total_upstreams": 1}
2024/05/20 07:16:04.305 DEBUG http.handlers.reverse_proxy upstream roundtrip {"upstream": "172.31.91.222:10000", "duration": 0.077972769, "request": {"remote_ip": "104.50.3.130", "remote_port": "58381", "client_ip": "104.50.3.130", "proto": "HTTP/1.1", "method": "GET", "host": "my.env.internal.domain.net:10000", "uri": "/v1/loginqueue/getinqueuev1", "headers": {"Authorization": [], "Accept-Language": ["en-US,en;q=0.9"], "Content-Type": ["application/json"], "X-Forwarded-For": ["104.50.3.130"], "Accept": ["application/json"], "X-Forwarded-Host": ["my.env.internal.domain.net:10000"], "User-Agent": ["X-UnrealEngine-Agent"], "Accept-Encoding": ["gzip, deflate, br"], "X-Forwarded-Proto": ["https"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "http/1.1", "server_name": "my.env.internal.domain.net"}}, "headers": {"Vary": ["Origin"], "Content-Length": ["659"], "Content-Type": ["application/json; charset=UTF-8"]}, "status": 200}
2024/05/20 07:16:04.473 DEBUG events event {"name": "tls_get_certificate", "id": "e9a1b317-78fc-4151-9b43-88a7614989d5", "origin": "tls", "data": {"client_hello":{"CipherSuites":[39578,4865,4866,4867,49196,49195,52393,49200,49199,52392,49162,49161,49172,49171,157,156,53,47,49160,49170,10],"ServerName":"my.env.internal.domain.net","SupportedCurves":[2570,29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,515,2053,2053,1281,2054,1537,513],"SupportedProtos":["http/1.1"],"SupportedVersions":[10794,772,771,770,769],"RemoteAddr":{"IP":"104.50.3.130","Port":58383,"Zone":""},"LocalAddr":{"IP":"172.31.87.3","Port":11000,"Zone":""}}}}
2024/05/20 07:16:04.473 DEBUG tls.handshake choosing certificate {"identifier": "my.env.internal.domain.net", "num_choices": 1}
2024/05/20 07:16:04.473 DEBUG tls.handshake default certificate selection results {"identifier": "my.env.internal.domain.net", "subjects": ["my.env.internal.domain.net"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:04.473 DEBUG tls.handshake matched certificate in cache {"remote_ip": "104.50.3.130", "remote_port": "58383", "subjects": ["my.env.internal.domain.net"], "managed": true, "expiration": "2024/08/18 06:00:44.000", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:04.554 DEBUG http.handlers.reverse_proxy selected upstream {"dial": "172.31.91.222:11000", "total_upstreams": 1}
2024/05/20 07:16:04.853 DEBUG http.handlers.reverse_proxy upstream roundtrip {"upstream": "172.31.91.222:11000", "duration": 0.298466029, "request": {"remote_ip": "104.50.3.130", "remote_port": "58383", "client_ip": "104.50.3.130", "proto": "HTTP/1.1", "method": "POST", "host": "my.env.internal.domain.net:11000", "uri": "/v1/account/authenticateorcreatev2", "headers": {"X-Forwarded-For": ["104.50.3.130"], "Accept": ["application/json"], "Authorization": [], "Accept-Language": ["en-US,en;q=0.9"], "X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["my.env.internal.domain.net:11000"], "Content-Length": ["1003"], "User-Agent": ["X-UnrealEngine-Agent"], "Content-Type": ["application/json"], "Accept-Encoding": ["gzip, deflate, br"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "http/1.1", "server_name": "my.env.internal.domain.net"}}, "headers": {"Content-Type": ["application/json; charset=UTF-8"], "Vary": ["Origin"], "Content-Length": ["1907"]}, "status": 200}
2024/05/20 07:16:05.127 DEBUG events event {"name": "tls_get_certificate", "id": "cdf2ae70-798c-4a67-9683-f4b1946884bd", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"my.env.internal.domain.net","SupportedCurves":[29,23,30,25,24],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,515,769,513,770,514,1026,1282,1538],"SupportedProtos":["http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"104.50.3.130","Port":49413,"Zone":""},"LocalAddr":{"IP":"172.31.87.3","Port":11000,"Zone":""}}}}
2024/05/20 07:16:05.127 DEBUG tls.handshake choosing certificate {"identifier": "my.env.internal.domain.net", "num_choices": 1}
2024/05/20 07:16:05.127 DEBUG tls.handshake default certificate selection results {"identifier": "my.env.internal.domain.net", "subjects": ["my.env.internal.domain.net"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:05.127 DEBUG tls.handshake matched certificate in cache {"remote_ip": "104.50.3.130", "remote_port": "49413", "subjects": ["my.env.internal.domain.net"], "managed": true, "expiration": "2024/08/18 06:00:44.000", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:05.128 DEBUG events event {"name": "tls_get_certificate", "id": "60ea49ed-0643-4ae7-ac97-a823777efd62", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"my.env.internal.domain.net","SupportedCurves":[29,23,30,25,24],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,515,769,513,770,514,1026,1282,1538],"SupportedProtos":["http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"104.50.3.130","Port":49414,"Zone":""},"LocalAddr":{"IP":"172.31.87.3","Port":10000,"Zone":""}}}}
2024/05/20 07:16:05.128 DEBUG tls.handshake choosing certificate {"identifier": "my.env.internal.domain.net", "num_choices": 1}
2024/05/20 07:16:05.128 DEBUG tls.handshake default certificate selection results {"identifier": "my.env.internal.domain.net", "subjects": ["my.env.internal.domain.net"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:05.128 DEBUG tls.handshake matched certificate in cache {"remote_ip": "104.50.3.130", "remote_port": "49414", "subjects": ["my.env.internal.domain.net"], "managed": true, "expiration": "2024/08/18 06:00:44.000", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:05.230 DEBUG http.stdlib http: TLS handshake error from 104.50.3.130:49413: local error: tls: bad record MAC
2024/05/20 07:16:05.230 DEBUG http.stdlib http: TLS handshake error from 104.50.3.130:49414: local error: tls: bad record MAC
2024/05/20 07:16:06.410 DEBUG events event {"name": "tls_get_certificate", "id": "b40e428f-f861-431f-a388-d14777defd24", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"my.env.internal.domain.net","SupportedCurves":[29,23,30,25,24],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,515,769,513,770,514,1026,1282,1538],"SupportedProtos":["http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"104.50.3.130","Port":49415,"Zone":""},"LocalAddr":{"IP":"172.31.87.3","Port":11000,"Zone":""}}}}
2024/05/20 07:16:06.410 DEBUG tls.handshake choosing certificate {"identifier": "my.env.internal.domain.net", "num_choices": 1}
2024/05/20 07:16:06.410 DEBUG tls.handshake default certificate selection results {"identifier": "my.env.internal.domain.net", "subjects": ["my.env.internal.domain.net"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:06.410 DEBUG tls.handshake matched certificate in cache {"remote_ip": "104.50.3.130", "remote_port": "49415", "subjects": ["my.env.internal.domain.net"], "managed": true, "expiration": "2024/08/18 06:00:44.000", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:06.411 DEBUG events event {"name": "tls_get_certificate", "id": "b25ca310-a9b3-48d5-b8cb-f0cb2adae281", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"my.env.internal.domain.net","SupportedCurves":[29,23,30,25,24],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,515,769,513,770,514,1026,1282,1538],"SupportedProtos":["http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"104.50.3.130","Port":49416,"Zone":""},"LocalAddr":{"IP":"172.31.87.3","Port":10000,"Zone":""}}}}
2024/05/20 07:16:06.411 DEBUG tls.handshake choosing certificate {"identifier": "my.env.internal.domain.net", "num_choices": 1}
2024/05/20 07:16:06.411 DEBUG tls.handshake default certificate selection results {"identifier": "my.env.internal.domain.net", "subjects": ["my.env.internal.domain.net"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:06.411 DEBUG tls.handshake matched certificate in cache {"remote_ip": "104.50.3.130", "remote_port": "49416", "subjects": ["my.env.internal.domain.net"], "managed": true, "expiration": "2024/08/18 06:00:44.000", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:06.515 DEBUG http.stdlib http: TLS handshake error from 104.50.3.130:49415: local error: tls: bad record MAC
2024/05/20 07:16:06.516 DEBUG http.stdlib http: TLS handshake error from 104.50.3.130:49416: local error: tls: bad record MAC
2024/05/20 07:16:07.870 DEBUG events event {"name": "tls_get_certificate", "id": "63cb48cd-23ca-4737-8cb4-755d2aceee93", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"my.env.internal.domain.net","SupportedCurves":[29,23,30,25,24],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,515,769,513,770,514,1026,1282,1538],"SupportedProtos":["http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"104.50.3.130","Port":49417,"Zone":""},"LocalAddr":{"IP":"172.31.87.3","Port":11000,"Zone":""}}}}
2024/05/20 07:16:07.870 DEBUG tls.handshake choosing certificate {"identifier": "my.env.internal.domain.net", "num_choices": 1}
2024/05/20 07:16:07.870 DEBUG tls.handshake default certificate selection results {"identifier": "my.env.internal.domain.net", "subjects": ["my.env.internal.domain.net"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:07.870 DEBUG tls.handshake matched certificate in cache {"remote_ip": "104.50.3.130", "remote_port": "49417", "subjects": ["my.env.internal.domain.net"], "managed": true, "expiration": "2024/08/18 06:00:44.000", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:08.057 DEBUG http.stdlib http: TLS handshake error from 104.50.3.130:49417: local error: tls: bad record MAC
2024/05/20 07:16:08.131 DEBUG events event {"name": "tls_get_certificate", "id": "5afa7ded-92d3-4401-ace9-9b383f38fa7c", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"my.env.internal.domain.net","SupportedCurves":[29,23,30,25,24],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,515,769,513,770,514,1026,1282,1538],"SupportedProtos":["http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"104.50.3.130","Port":49418,"Zone":""},"LocalAddr":{"IP":"172.31.87.3","Port":10000,"Zone":""}}}}
2024/05/20 07:16:08.131 DEBUG tls.handshake choosing certificate {"identifier": "my.env.internal.domain.net", "num_choices": 1}
2024/05/20 07:16:08.131 DEBUG tls.handshake default certificate selection results {"identifier": "my.env.internal.domain.net", "subjects": ["my.env.internal.domain.net"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:08.131 DEBUG tls.handshake matched certificate in cache {"remote_ip": "104.50.3.130", "remote_port": "49418", "subjects": ["my.env.internal.domain.net"], "managed": true, "expiration": "2024/08/18 06:00:44.000", "hash": "6f02679e1f96fb057580537487b62c0e4d0d3d44f8e3c53f38e66ecd02a3f427"}
2024/05/20 07:16:08.238 DEBUG http.stdlib http: TLS handshake error from 104.50.3.130:49418: local error: tls: bad record MAC
3. Caddy version:
v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=
4. How I installed and ran Caddy:
sudo dnf update -y
sudo yum install nss-tools
curl -L -o caddy https://dc0eql1zxfug.cloudfront.net/caddy-with-rate-limiting-AL2023 # Download our version of Caddy, bundled with caddy-rate-limit built for AL2023
chmod +x caddy
sudo mv caddy /usr/bin/
sudo mkdir /etc/caddy
echo "${caddyfileAsText}" | sudo tee /etc/caddy/Caddyfile
echo "${caddyServiceFileAsText}" | sudo tee /etc/systemd/system/caddy.service
sudo systemctl enable caddy.service
sudo systemctl start caddy.service
sudo caddy trust
a. System environment:
Amazon Linux 2023 6.1.90-99.173.amzn2023.aarch64
b. Command:
sudo systemctl start caddy.service
c. Service/unit/compose file:
caddy.service:
[Unit]
Description=Caddy
After=network-online.target
[Service]
User=root
Group=root
ExecStart=/usr/bin/caddy run --config /etc/caddy/Caddyfile
Restart=on-failure
RestartSec=15s # Delay restart by 15 seconds
[Install]
WantedBy=multi-user.target
d. My complete Caddy config:
I’m sorry I’m truly unable to share the full domain name at this time, but can share the current IP.
{
order rate_limit before basicauth
log {
format console
level debug
output file /home/caddy/logs/caddy.log {
roll_uncompressed
roll_keep 30
roll_keep_for 30d
}
}
}
# Common rate_limit block
(player_rate_limit_settings) {
# Docs: https://github.com/mholt/caddy-ratelimit
rate_limit {
zone vin_diesel {
key static # Only one rate limiter
events 5
window 1s
}
}
}
# Common rate_limit block
(operator_rate_limit_settings) {
# Docs: https://github.com/mholt/caddy-ratelimit
rate_limit {
zone operator_bouncer {
key static # Only one rate limiter
events 100
window 1m
}
}
}
# Route traffic from ports 10000, 11000, 10200, and 11200 to the IP address 172.31.91.222
my.env.internal.domain.net:10000 {
import player_rate_limit_settings
reverse_proxy 172.31.91.222:10000
}
my.env.internal.domain.net:11000 {
import player_rate_limit_settings
reverse_proxy 172.31.91.222:11000
}
my.env.internal.domain.net:10200 {
import operator_rate_limit_settings
reverse_proxy 172.31.91.222:10200
}
my.env.internal.domain.net:11200 {
import operator_rate_limit_settings
reverse_proxy 172.31.91.222:11200
}
my.env.internal.domain.net:10100 {
reverse_proxy 172.31.91.222:10100
}
my.env.internal.domain.net:11100 {
reverse_proxy 172.31.91.222:10100
}
5. Links to relevant resources:
https://docs.unrealengine.com/4.27/en-US/API/Runtime/WebSockets/