I think I found the issue. I needed these lines in the haproxy config.
# Wait for a client hello for at most 5 seconds
tcp-request inspect-delay 5s
tcp-request content accept if { req.ssl_hello_type 1 }
Thank you for your help. Sorry for requesting it without a log.