Configuring minio site replication with caddy reverse proxy returns error 400 bad request

Help
1

1. The problem I’m having:

I’m trying to set site replication with mc admin replicate add <alias1> <alias2> --insecure, where alias1 and alias2 is the caddy load balancer. alias1 has 2 node minio-1 and minio-2, alias 2 has 2 node minio-3 and minio-4. This command is run in minio-1, with caddy reverse proxy running in Running the replicate returns error 400 bad request and remote target is offline, but running

mc admin info alias1 --insecure
mc admin replicate info alias1 insecure
mc ping alias1 --insecure

mc admin info alias2 --insecure
mc admin replicate info alias2 insecure
mc ping alias2 --insecure

returns ok. When trying to site replicate directly without caddy load balancer, the replicate succeed.
My alias:

alias1
  URL       : https://minio-proxy-1.mydomain
  AccessKey : <redacted>
  SecretKey : <redacted>
  API       : s3v4
  Path      : auto
  Src       : /root/.mc/config.json

alias2
  URL       : https://minio-proxy-2.mydomain
  AccessKey : <redacted>
  SecretKey : <redacted>
  API       : s3v4
  Path      : auto
  Src       : /root/.mc/config.json

2. Error messages and/or full log output:

mc: <DEBUG> PUT /minio/admin/v3/site-replication/add?api-version=1&force=false&replicateILMExpiry=false HTTP/1.1
Host: minio-proxy-1.mydomain
User-Agent: MinIO (linux; amd64) madmin-go/3.0.70 mc/RELEASE.2025-08-13T08-35-41Z
Content-Length: 318
Accept-Encoding: zstd,gzip
Authorization: AWS4-HMAC-SHA256 Credential=<redacted>/20251107//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: 82f560ca80760f159a5b55b6c887be9e3d3c171ddc832a3b7884f83776f0ea53
X-Amz-Date: 20251107T042150Z

mc: <DEBUG> HTTP/1.1 400 Bad Request
Content-Length: 449
Accept-Ranges: bytes
Alt-Svc: h3=":443"; ma=2592000
Content-Type: application/json
Date: Fri, 07 Nov 2025 04:21:50 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
Via: 1.1 Caddy
X-Amz-Id-2: 9963fc07681a583d5f89b796abb104743e0f55e237bb057ddd907dffcab8645a
X-Amz-Request-Id: 18759EB0A4ED5680
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

{"Code":"XMinioSiteReplicationPeerResp","Message":"Error received when contacting a peer site (unable to create admin client for <alias1>: Remote service endpoint minio-proxy-1.mydomain not available\n\tremote target is offline for endpoint minio-proxy-1.mydomain)","Resource":"/minio/admin/v3/site-replication/add","RequestId":"18759EB0A4ED5680","HostId":"9963fc07681a583d5f89b796abb104743e0f55e237bb057ddd907dffcab8645a"}
mc: <DEBUG> TLS Certificate found:
mc: <DEBUG>  >> Expires: 2025-11-07 09:15:18 +0000 UTC
mc: <DEBUG> TLS Certificate found:
mc: <DEBUG>  >> Expires: 2025-11-13 04:55:18 +0000 UTC
mc: <DEBUG> Response Time:  49.227743ms

mc: <ERROR> Unable to add sites for replication. Error received when contacting a peer site (unable to create admin client for <alias1>: Remote service endpoint minio-proxy-1.mydomain not available
        remote target is offline for endpoint https://minio-proxy-1.mydomain)
 (1) admin-replicate-add.go:122 cmd.mainAdminReplicateAdd(..) Tags: [alias1, alias2]
 (0) admin-replicate-add.go:122 cmd.mainAdminReplicateAdd(..)
 Release-Tag:RELEASE.2025-08-13T08-35-41Z | Commit:7394ce0dd2a8 | Host:<hostname> | OS:linux | Arch:amd64 | Lang:go1.24.6 | Mem:67 MiB/77 MiB | Heap:67 MiB/72 MiB.

no log in caddy’s journalctl

3. Caddy version:

v2.10.2 h1:g/gTYjGMD0dec+UgMw8SnfmJ3I9+M2TdvoRL/Ovu6U8=

4. How I installed and ran Caddy:

a. System environment:

using systemd

b. Command:

From systemd

/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

c. Service/unit/compose file:


[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

proxy 1

minio-proxy-1.mydomain{
  tls internal
  reverse_proxy http://minio-1.mydomain:9000 http://minio-2.mydomain:9000 {
     health_uri /minio/health/live
     lb_try_duration 5s
     #transport http {
     # tls_trust_pool file /opt/minio/certs/public.crt
    #}
  }
}

proxy 2

minio-proxy-2.mydomain{
  tls internal
  reverse_proxy http://minio-3.mydomain:9000 http://minio-4.mydomain:9000 {
     health_uri /minio/health/live
     lb_try_duration 5s
     #transport http {
     # tls_trust_pool file /opt/minio/certs/public.crt
    #}
  }
}

5. Links to relevant resources:

EDIT: Decided not to set replication through LB as setting replication directly from minio-1 to minio-3 also replicates the other node