1. The problem I’m having:
I have Haproxy running as frontend on my pfSense-box handling reverse proxying duties. It works fine for my other services with an unencrypted HTTP backend connection between Haproxy and the services, but I’ve got one backend that I need to use HTTPS for. This service is using Caddy to provide HTTPS, ie. the setup is Haproxy<->Caddy<->actual underlying service.
Alas, I just can’t get things working and at this point I have no idea, if the issue is my Haproxy configuration or Caddy. I’m just getting a HTTP 308 reply with no content, if I try to connect via an external IP, ie. through Haproxy, but it works just fine from within LAN directly to Caddy. Something between Haproxy and Caddy refuses to work right.
curl -vL produces the following (I cut a part off, since it just keeps on repeating):
ita@ubuvm:~$ curl -vL https://myserver.mydomain
* Host myserver.mydomain:443 was resolved.
* IPv6: (none)
* IPv4: myexternalip
* Trying myexternalip:443...
* Connected to myserver.mydomain (myexternalip) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
* subject: CN=myserver.mydomain
* start date: May 5 07:23:59 2024 GMT
* expire date: Aug 3 07:23:58 2024 GMT
* subjectAltName: host "myserver.mydomain" matched cert's "myserver.mydomain"
* issuer: C=US; O=Let's Encrypt; CN=R3
* SSL certificate verify ok.
* Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://myserver.mydomain/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: myserver.mydomain]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: myserver.mydomain
> User-Agent: curl/8.5.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 308
< location: https://myserver.mydomain/
< server: Caddy
< date: Sun, 05 May 2024 10:50:03 GMT
< content-length: 0
<
* Connection #0 to host myserver.mydomain left intact
* Issue another request to this URL: 'https://myserver.mydomain/'
* Found bundle for host: 0x5bdc86085df0 [can multiplex]
* Re-using existing connection with host myserver.mydomain
* [HTTP/2] [3] OPENED stream for https://myserver.mydomain/
* [HTTP/2] [3] [:method: GET]
* [HTTP/2] [3] [:scheme: https]
* [HTTP/2] [3] [:authority: myserver.mydomain]
* [HTTP/2] [3] [:path: /]
* [HTTP/2] [3] [user-agent: curl/8.5.0]
* [HTTP/2] [3] [accept: */*]
> GET / HTTP/2
> Host: myserver.mydomain
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/2 308
< location: https://myserver.mydomain/
< server: Caddy
< date: Sun, 05 May 2024 10:50:03 GMT
< content-length: 0
<
* Connection #0 to host myserver.mydomain left intact
* Issue another request to this URL: 'https://myserver.mydomain/'
* Found bundle for host: 0x5bdc86085df0 [can multiplex]
* Re-using existing connection with host myserver.mydomain
* [HTTP/2] [5] OPENED stream for https://myserver.mydomain/
* [HTTP/2] [5] [:method: GET]
* [HTTP/2] [5] [:scheme: https]
* [HTTP/2] [5] [:authority: myserver.mydomain]
* [HTTP/2] [5] [:path: /]
* [HTTP/2] [5] [user-agent: curl/8.5.0]
* [HTTP/2] [5] [accept: */*]
> GET / HTTP/2
> Host: myserver.mydomain
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/2 308
< location: https://myserver.mydomain/
< server: Caddy
< date: Sun, 05 May 2024 10:50:03 GMT
< content-length: 0
<
* Connection #0 to host myserver.mydomain left intact
* Issue another request to this URL: 'https://myserver.mydomain/'
* Found bundle for host: 0x5bdc86085df0 [can multiplex]
* Re-using existing connection with host myserver.mydomain
* [HTTP/2] [7] OPENED stream for https://myserver.mydomain/
* [HTTP/2] [7] [:method: GET]
* [HTTP/2] [7] [:scheme: https]
* [HTTP/2] [7] [:authority: myserver.mydomain]
* [HTTP/2] [7] [:path: /]
* [HTTP/2] [7] [user-agent: curl/8.5.0]
* [HTTP/2] [7] [accept: */*]
> GET / HTTP/2
> Host: myserver.mydomain
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/2 308
< location: https://myserver.mydomain/
< server: Caddy
< date: Sun, 05 May 2024 10:50:03 GMT
< content-length: 0
<
2. Error messages and/or full log output:
Access log shows the following:
{"level":"info","ts":1714906204.38699,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"192.168.1.1","remote_port":"62540","client_ip":"192.168.1.1","proto":"HTTP/1.1","method":"GET","host":"myserver.mydomain","uri":"/","headers":{"User-Agent":["curl/8.5.0"],"Accept":["*/*"],"X-Forwarded-Proto":["https"],"X-Forwarded-For":["85.76.105.149"]}},"bytes_read":0,"user_id":"","duration":0.000141567,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://myserver.mydomain/"],"Content-Type":[]}}
3. Caddy version:
v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=
4. How I installed and ran Caddy:
Not really relevant, but I downloaded the binary from your build service with Porkbun DNS support built-in.
a. System environment:
Linux AMD64 Ubuntu 24.04, with Caddy running via Docker.
d. My complete Caddy config:
{
email {$EMAIL}
acme_dns porkbun {
api_key {$PORKBUN_API_KEY}
api_secret_key {$PORKBUN_API_SECRET_KEY}
}
servers {
trusted_proxies static 192.168.1.0/24
}
}
(admin_redir) {
@admin {
path /admin*
not remote_ip private_ranges
}
redir @admin /
}
{$DOMAIN}:443 {
log {
level INFO
output file {$LOG_FILE} {
roll_size 10MB
roll_keep 10
}
}
encode gzip
header / {
X-Robots-Tag "noindex, nofollow"
-X-Powered-By
-Last-Modified
}
import admin_redir
reverse_proxy vaultwarden:80 {
header_up X-Real-IP {remote_host}
}
}
5. Links to relevant resources:
I’ll also leave my Haproxy config here, but I have no idea if it’s any useful here or not:
# Automaticaly generated, dont edit manually.
# Generated on: 2024-05-05 12:59
global
maxconn 1000
log /var/run/log local0 notice
stats socket /tmp/haproxy.socket level admin expose-fd listeners
uid 80
gid 80
nbthread 3
hard-stop-after 15m
chroot /tmp/haproxy_chroot
daemon
tune.ssl.default-dh-param 2048
server-state-file /tmp/haproxy_server_state
frontend https_frontend
bind myexternalip:443 name myexternalip:443 ssl crt-list /var/etc/haproxy/https_frontend.crt_list
mode http
log global
option dontlognull
option http-keep-alive
option forwardfor
acl https ssl_fc
http-request set-header X-Forwarded-Proto http if !https
http-request set-header X-Forwarded-Proto https if https
timeout client 30000
acl oneservice_acl var(txn.txnhost) -m str -i oneservice.mydomain
acl myserver_acl var(txn.txnhost) -m str -i myserver.mydomain
acl geofi_acl src -f /var/etc/haproxy/ipalias_geofi.lst
acl robots_acl var(txn.txnpath) -m end -i /robots.txt
acl aclcrt_https_frontend var(txn.txnhost) -m reg -i ^myserver\.mydomain(:([0-9]){1,5})?$
acl aclcrt_https_frontend var(txn.txnhost) -m reg -i ^oneservice\.mydomain(:([0-9]){1,5})?$
http-request set-var(txn.txnhost) hdr(host)
http-request set-var(txn.txnpath) path
http-request return status 200 content-type "text/plain" file "/var/robots.txt" hdr "cache-control" "no-cache" if robots_acl aclcrt_https_frontend
use_backend oneservice_backend_ipvANY if oneservice_acl aclcrt_https_frontend
use_backend myserver_backend_ipvANY if myserver_acl aclcrt_https_frontend
frontend http_redirect
bind myexternalip:80 name myexternalip:80
mode http
log global
option dontlognull
option http-keep-alive
option forwardfor
acl https ssl_fc
http-request set-header X-Forwarded-Proto http if !https
http-request set-header X-Forwarded-Proto https if https
timeout client 30000
acl robots_acl var(txn.txnpath) -m end -i /robots.txt
acl no_robots_acl var(txn.txnpath) -m end -i /robots.txt
http-request set-var(txn.txnpath) path
http-request return status 200 content-type "text/plain" file "/var/robots.txt" hdr "cache-control" "no-cache" if robots_acl
http-request redirect scheme https if !no_robots_acl
backend oneservice_backend_ipvANY
mode http
id 100
log global
timeout connect 30000
timeout server 30000
retries 3
load-server-state-from-file global
server oneservice 192.168.1.15:8080 id 101
backend myserver_backend_ipvANY
mode http
id 106
log global
timeout connect 30000
timeout server 30000
retries 3
load-server-state-from-file global
server myserver_backend 192.168.1.86:443 id 105 ssl verify none