Haproxy as frontend, Caddy HTTPS as backend not working

1. The problem I’m having:

I have Haproxy running as frontend on my pfSense-box handling reverse proxying duties. It works fine for my other services with an unencrypted HTTP backend connection between Haproxy and the services, but I’ve got one backend that I need to use HTTPS for. This service is using Caddy to provide HTTPS, ie. the setup is Haproxy<->Caddy<->actual underlying service.

Alas, I just can’t get things working and at this point I have no idea, if the issue is my Haproxy configuration or Caddy. I’m just getting a HTTP 308 reply with no content, if I try to connect via an external IP, ie. through Haproxy, but it works just fine from within LAN directly to Caddy. Something between Haproxy and Caddy refuses to work right.

curl -vL produces the following (I cut a part off, since it just keeps on repeating):

ita@ubuvm:~$ curl -vL https://myserver.mydomain
* Host myserver.mydomain:443 was resolved.
* IPv6: (none)
* IPv4: myexternalip
*   Trying myexternalip:443...
* Connected to myserver.mydomain (myexternalip) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=myserver.mydomain
*  start date: May  5 07:23:59 2024 GMT
*  expire date: Aug  3 07:23:58 2024 GMT
*  subjectAltName: host "myserver.mydomain" matched cert's "myserver.mydomain"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://myserver.mydomain/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: myserver.mydomain]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: myserver.mydomain
> User-Agent: curl/8.5.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 308 
< location: https://myserver.mydomain/
< server: Caddy
< date: Sun, 05 May 2024 10:50:03 GMT
< content-length: 0
< 
* Connection #0 to host myserver.mydomain left intact
* Issue another request to this URL: 'https://myserver.mydomain/'
* Found bundle for host: 0x5bdc86085df0 [can multiplex]
* Re-using existing connection with host myserver.mydomain
* [HTTP/2] [3] OPENED stream for https://myserver.mydomain/
* [HTTP/2] [3] [:method: GET]
* [HTTP/2] [3] [:scheme: https]
* [HTTP/2] [3] [:authority: myserver.mydomain]
* [HTTP/2] [3] [:path: /]
* [HTTP/2] [3] [user-agent: curl/8.5.0]
* [HTTP/2] [3] [accept: */*]
> GET / HTTP/2
> Host: myserver.mydomain
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/2 308 
< location: https://myserver.mydomain/
< server: Caddy
< date: Sun, 05 May 2024 10:50:03 GMT
< content-length: 0
< 
* Connection #0 to host myserver.mydomain left intact
* Issue another request to this URL: 'https://myserver.mydomain/'
* Found bundle for host: 0x5bdc86085df0 [can multiplex]
* Re-using existing connection with host myserver.mydomain
* [HTTP/2] [5] OPENED stream for https://myserver.mydomain/
* [HTTP/2] [5] [:method: GET]
* [HTTP/2] [5] [:scheme: https]
* [HTTP/2] [5] [:authority: myserver.mydomain]
* [HTTP/2] [5] [:path: /]
* [HTTP/2] [5] [user-agent: curl/8.5.0]
* [HTTP/2] [5] [accept: */*]
> GET / HTTP/2
> Host: myserver.mydomain
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/2 308 
< location: https://myserver.mydomain/
< server: Caddy
< date: Sun, 05 May 2024 10:50:03 GMT
< content-length: 0
< 
* Connection #0 to host myserver.mydomain left intact
* Issue another request to this URL: 'https://myserver.mydomain/'
* Found bundle for host: 0x5bdc86085df0 [can multiplex]
* Re-using existing connection with host myserver.mydomain
* [HTTP/2] [7] OPENED stream for https://myserver.mydomain/
* [HTTP/2] [7] [:method: GET]
* [HTTP/2] [7] [:scheme: https]
* [HTTP/2] [7] [:authority: myserver.mydomain]
* [HTTP/2] [7] [:path: /]
* [HTTP/2] [7] [user-agent: curl/8.5.0]
* [HTTP/2] [7] [accept: */*]
> GET / HTTP/2
> Host: myserver.mydomain
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/2 308 
< location: https://myserver.mydomain/
< server: Caddy
< date: Sun, 05 May 2024 10:50:03 GMT
< content-length: 0
<

2. Error messages and/or full log output:

Access log shows the following:

{"level":"info","ts":1714906204.38699,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"192.168.1.1","remote_port":"62540","client_ip":"192.168.1.1","proto":"HTTP/1.1","method":"GET","host":"myserver.mydomain","uri":"/","headers":{"User-Agent":["curl/8.5.0"],"Accept":["*/*"],"X-Forwarded-Proto":["https"],"X-Forwarded-For":["85.76.105.149"]}},"bytes_read":0,"user_id":"","duration":0.000141567,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://myserver.mydomain/"],"Content-Type":[]}}

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

Not really relevant, but I downloaded the binary from your build service with Porkbun DNS support built-in.

a. System environment:

Linux AMD64 Ubuntu 24.04, with Caddy running via Docker.

d. My complete Caddy config:

{
        email {$EMAIL}
        acme_dns porkbun {
                api_key {$PORKBUN_API_KEY}
                api_secret_key {$PORKBUN_API_SECRET_KEY}
        }
        servers {
                trusted_proxies static 192.168.1.0/24
        }
}

(admin_redir) {
        @admin {
                path /admin*
                not remote_ip private_ranges
        }
        redir @admin /
}

{$DOMAIN}:443 {
        log {
                level INFO
                output file {$LOG_FILE} {
                        roll_size 10MB
                        roll_keep 10
                }
        }

        encode gzip

        header / {
                X-Robots-Tag "noindex, nofollow"
                -X-Powered-By
                -Last-Modified
        }

        import admin_redir

        reverse_proxy vaultwarden:80 {
                header_up X-Real-IP {remote_host}
        }
}

5. Links to relevant resources:

I’ll also leave my Haproxy config here, but I have no idea if it’s any useful here or not:

# Automaticaly generated, dont edit manually.
# Generated on: 2024-05-05 12:59
global
	maxconn			1000
	log			/var/run/log	local0	notice
	stats socket /tmp/haproxy.socket level admin  expose-fd listeners
	uid			80
	gid			80
	nbthread			3
	hard-stop-after		15m
	chroot				/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	server-state-file /tmp/haproxy_server_state

frontend https_frontend
	bind			myexternalip:443 name myexternalip:443   ssl crt-list /var/etc/haproxy/https_frontend.crt_list  
	mode			http
	log			global
	option			dontlognull
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	acl			oneservice_acl	var(txn.txnhost) -m str -i oneservice.mydomain
	acl			myserver_acl	var(txn.txnhost) -m str -i myserver.mydomain
	acl			geofi_acl	src -f /var/etc/haproxy/ipalias_geofi.lst
	acl			robots_acl	var(txn.txnpath) -m end -i /robots.txt
	acl			aclcrt_https_frontend	var(txn.txnhost) -m reg -i ^myserver\.mydomain(:([0-9]){1,5})?$
	acl			aclcrt_https_frontend	var(txn.txnhost) -m reg -i ^oneservice\.mydomain(:([0-9]){1,5})?$
	http-request set-var(txn.txnhost) hdr(host)
	http-request set-var(txn.txnpath) path
	http-request return status 200 content-type "text/plain" file "/var/robots.txt" hdr "cache-control" "no-cache"  if  robots_acl aclcrt_https_frontend
	use_backend oneservice_backend_ipvANY  if  oneservice_acl aclcrt_https_frontend
	use_backend myserver_backend_ipvANY  if  myserver_acl aclcrt_https_frontend

frontend http_redirect
	bind			myexternalip:80 name myexternalip:80   
	mode			http
	log			global
	option			dontlognull
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	acl			robots_acl	var(txn.txnpath) -m end -i /robots.txt
	acl			no_robots_acl	var(txn.txnpath) -m end -i /robots.txt
	http-request set-var(txn.txnpath) path
	http-request return status 200 content-type "text/plain" file "/var/robots.txt" hdr "cache-control" "no-cache"  if  robots_acl 
	http-request redirect scheme https  if  !no_robots_acl 

backend oneservice_backend_ipvANY
	mode			http
	id			100
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	load-server-state-from-file	global
	server			oneservice 192.168.1.15:8080 id 101  

backend myserver_backend_ipvANY
	mode			http
	id			106
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	load-server-state-from-file	global
	server			myserver_backend 192.168.1.86:443 id 105 ssl  verify none

Your proxy is making HTTP requests to Caddy, not HTTPS requests, so Caddy serves an HTTP->HTTPS redirect.

1 Like

I actually just found out that Haproxy on pfSense is broken in several ways and it may just be that why it’s not working and not necessarily anything to do with my settings. I suppose you’re not familiar with Haproxy, but the config file does show that the backend was configured to use HTTPS, not HTTP – it just seems to have insisted on using HTTP anyways and ignoring some of the settings.