Access to OPNSense throws a 400 BAD REQUEST after upgrading Caddy from 2.8.4 to 2.9

Help
1

1. The problem I’m having:

Hey all, first post here! So I have a bunch of self-hosted services that I access through a Caddy LXC that’s setup with Let’s Encrypt and CrowdSec using OPNsense LAPI. After upgrading from 2.84 ⇒ 2.9 all services are accessible except for my OPNSense instance (v. 24.7.11) which throws a 400 bad request error. After scouring the Caddy community for a few days to find a solution, I sort of gave up and just restored the Caddy Instance back to 2.8.4 from backup thus regaining access to OPNSense again.

Anyone know off the top the issue I am seeing? Possibly underlying changes in 2.9 and/or incompatibilities with OPNsense?

2. Error messages and/or full log output:

Jan 10 07:47:31 caddy caddy[2465]: {"level":"debug","ts":1736513251.9370177,"logger":"crowdsec","msg":"finished processing 29748 new decisions","instance_id":"7772a16f"}
Jan 10 07:47:46 caddy caddy[2465]: {"level":"debug","ts":1736513266.8469832,"logger":"events","msg":"event","name":"tls_get_certificate","id":"858a71d5-7bb1-42ec-bbc9-4ad996ec7083","origin":"tls","data":{"client_hello":{"CipherSuites":[4865,4867,4866],"ServerName":"opensense.home.janetix.top","SupportedCurves":[29,23,24,25],"SupportedPoints":null,"SignatureSchemes":[1027,1283,1539,515,2052,2053,2054,1025,1281,1537,513],"SupportedProtos":["h3"],"SupportedVersions":[772],"RemoteAddr":{"IP":"192.168.20.178","Port":50258,"Zone":""},"LocalAddr":{"IP":"192.168.10.101","Port":443,"Zone":""}}}}
Jan 10 07:47:46 caddy caddy[2465]: {"level":"debug","ts":1736513266.8473003,"logger":"tls.handshake","msg":"choosing certificate","identifier":"opensense.home.janetix.top","num_choices":1}
Jan 10 07:47:46 caddy caddy[2465]: {"level":"debug","ts":1736513266.847317,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"opensense.home.janetix.top","subjects":["opensense.home.janetix.top"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"99d5cfd330c0833870f89be548a0588decd11c9079c49955c1f63a7006c9dd89"}
Jan 10 07:47:46 caddy caddy[2465]: {"level":"debug","ts":1736513266.847332,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"192.168.20.178","remote_port":"50258","subjects":["opensense.home.janetix.top"],"managed":true,"expiration":1740765835,"hash":"99d5cfd330c0833870f89be548a0588decd11c9079c
 ESCOD
772a16f"}
Jan 10 07:47:31 caddy caddy[2465]: {"level":"debug","ts":1736513251.9370177,"logger":"crowdsec","msg":"finished processing 29748 new decisions","instance_id":"7772a16f"}
Jan 10 07:47:46 caddy caddy[2465]: {"level":"debug","ts":1736513266.8469832,"logger":"events","msg":"event","name":"tls_get_certificate","id":"858a71d5-7bb1-42ec-bbc9-4ad996ec7083","origin":"tls","data":{"client_hello":{"CipherSuites":[4865,4867,4866],"ServerName":"opensense.home.janetix.top","SupportedCurves":[29,23,24,25],"SupportedPoints":null,"SignatureSchemes":[1027,1283,1539,515,2052,2053,2054,1025,1281,1537,513],"SupportedProtos":["h3"],"SupportedVersions":[772],"RemoteAddr":{"IP":"192.168.20.178","Port":50258,"Zone":""},"LocalAddr":{"IP":"192.168.10.101","Port":443,"Zone":""}}}}
Jan 10 07:47:46 caddy caddy[2465]: {"level":"debug","ts":1736513266.8473003,"logger":"tls.handshake","msg":"choosing certificate","identifier":"opensense.home.janetix.top","num_choices":1}
Jan 10 07:47:46 caddy caddy[2465]: {"level":"debug","ts":1736513266.847317,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"opensense.home.janetix.top","subjects":["opensense.home.janetix.top"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"99d5cfd330c0833870f89be548a0588decd11c9079c49955c1f63a7006c9dd89"}
Jan 10 07:47:46 caddy caddy[2465]: {"level":"debug","ts":1736513266.847332,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"192.168.20.178","remote_port":"50258","subjects":["opensense.home.janetix.top"],"managed":true,"expiration":1740765835,"hash":"99d5cfd330c0833870f89be548a0588decd11c9079c
49955c1f63a7006c9dd89"}
Jan 10 07:47:46 caddy caddy[2465]: {"level":"debug","ts":1736513266.8595457,"log
ger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.1.1
:443","total_upstreams":1}
Jan 10 07:47:46 caddy caddy[2465]: {"level":"debug","ts":1736513266.8754885,"log
ger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.16
8.1.1:443","duration":0.015860287,"request":{"remote_ip":"192.168.20.178","remot
e_port":"50258","client_ip":"192.168.20.178","proto":"HTTP/3.0","method":"GET","
host":"opensense.home.janetix.top","uri":"/","headers":{"Upgrade-Insecure-Reques
ts":["1"],"Priority":["u=0, i"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Proto
":["https"],"X-Forwarded-Host":["opensense.home.janetix.top"],"Sec-Fetch-Dest":[
"document"],"Alt-Used":["opensense.home.janetix.top"],"Sec-Gpc":["1"],"Sec-Fetch
-User":["?1"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Accept-Language":["
en-US,en;q=0.5"],"Sec-Fetch-Site":["none"],"User-Agent":["Mozilla/5.0 (Macintosh
; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0"],"Accept":["text
/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"Dnt":["1"],"X-For
warded-For":["192.168.20.178"]},"tls":{"resumed":false,"version":772,"cipher_sui
te":4867,"proto":"h3","server_name":"opensense.home.janetix.top"}},"headers":{"C
ontent-Type":["text/html"],"Content-Length":["162"],"Date":["Fri, 10 Jan 2025 12
:47:46 GMT"],"Server":["OPNsense"]},"status":400}

3. Caddy version:

v2.9.1 h1:OEYiZ7DbCzAWVb6TNEkjRcSCRGHVoZsJinoDR/n9oaY=

4. How I installed and ran Caddy:

Installed Caddy LXC via ProxMox helper script

a. System environment:

b. Command:

systemctl start caddy

c. Service/unit/compose file:

d. My complete Caddy config:

Below is the directive for OPNSense.  NOTE:  I have 14 other services configured
just like the OPNSense directlive below.  All 14 servies were still accessible after
upgrading to 2.9.

{
        acme_dns cloudflare <my data here>

        crowdsec {
                api_key <my API key here>
                api_url http://192.168.1.1:8080/
        }
}
opnsense.my.domain.com {
        reverse_proxy https://192.168.1.1 {
                transport http {
                        tls
                        tls_insecure_skip_verify
                }
        }

5. Links to relevant resources:

2

For testing I used the Caddy plugin on the OPNsense with the caddy binary manually replaced to v2.9.0 h1:rteY8N18LsQn+2KVk6R10Vg/AlNsID1N/Ek9JLjm2yE=

Maybe there is a difference in the environment when it runs in Docker. In my case I saw no error and the website opened up.

example.com {
	reverse_proxy https://172.16.0.254:4444 {
		transport http {
			tls_insecure_skip_verify
		}
	}
}
3

I had this exact issue. I was able to work around it by adding versions 1.1 1.2 to the caddy config

opnsense.example.com {
	tls /ssl/fullchain.pem /ssl/privkey.pem
    reverse_proxy https://1.2.3.4:443 {
		transport http {
			tls
			tls_insecure_skip_verify
			versions 1.1 1.2
		}
	}
}

Some incompatibility with Opnsense and HTML3 maybe?

1 Like
4

Thanks man. That was it!

I also run a nginx reverse proxy as more of a fallback w/ Let’s Encrypt and the same OPNsense directive as the Caddyfile I posted up. Oddly, it has no issues accessing Opnsense, but as you said, maybe it’s a compat. issue. I’ll comment out the declared versions in my CaddyFile on the next OPNsense update to see if things revert to normal!

Thanks again!