Unable to reverse proxy for Minio?

1. The problem I’m having:

I’m running Minio as a service on my TrueNAS SCALE system, which serves the service itself over port 9090 and its console over port 9091 via HTTPS with a valid Let’s Encrypt cert. I want to make it available from outside my LAN using Caddy. But requests to that URL fail.

2. Error messages and/or full log output:

dan@sip:~$ curl -vL https://ipbx27.s3.familybrown.org
*   Trying 96.68.219.29:443...
* Connected to ipbx27.s3.familybrown.org (96.68.219.29) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, internal error (592):
* error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
* Closing connection 0
curl: (35) error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error

3. Caddy version:

2.6.4

4. How I installed and ran Caddy:

a. System environment:

OPNsense 23.1.1_2

b. Command:

Caddy runs as a service under OPNsense, installed via a third-party package (OPNsense Repo – Routerperformance). The service file doesn’t seem to be in the expected place in the filesystem (/usr/local/etc/rc.d).

c. Service/unit/compose file:

n/a

d. My complete Caddy config:

In relevant part:

ipbx20.s3.familybrown.org,
ipbx27.s3.familybrown.org,
s3.familybrown.org {
    log {
        output file /var/log/caddy/s3_access.log
    }
    reverse_proxy https://minio.familybrown.org:9090
}

5. Links to relevant resources:

According to this page, I should be able to set up the reverse proxy just as I have.

6. Additional Information:

When I try to access the Minio server from my OPNsense box (which is where I’m running Caddy), it’s successful:

[opn_admin@opnsense /var/log/caddy]$ curl -vL https://minio.familybrown.org:9090
*   Trying 192.168.1.10:9090...
* Connected to minio.familybrown.org (192.168.1.10) port 9090 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /usr/local/etc/ssl/cert.pem
*  CApath: none
* [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS handshake, Client hello (1):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Server hello (2):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Certificate (11):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, CERT verify (15):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Finished (20):
* [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: CN=familybrown.org
*  start date: Mar  2 18:41:54 2023 GMT
*  expire date: May 31 18:41:53 2023 GMT
*  subjectAltName: host "minio.familybrown.org" matched cert's "*.familybrown.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: minio.familybrown.org:9090
> User-Agent: curl/7.87.0
> Accept: */*
>
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< Accept-Ranges: bytes
< Content-Length: 252
< Content-Security-Policy: block-all-mixed-content
< Content-Type: application/xml
< Server: MinIO
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Vary: Origin
< Vary: Accept-Encoding
< X-Amz-Bucket-Region: us-east-2
< X-Amz-Request-Id: 1750F3975602CAF2
< X-Content-Type-Options: nosniff
< X-Xss-Protection: 1; mode=block
< Date: Wed, 29 Mar 2023 17:21:32 GMT
<
<?xml version="1.0" encoding="UTF-8"?>
* Connection #0 to host minio.familybrown.org left intact
<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/</Resource><Region>us-east-2</Region><RequestId>1750F3975602CAF2</RequestId><HostId>03edb5a2-fd6b-4717-a7ea-15909c190d15</HostId></Error>

What do you see in Caddy’s logs?

A whole lot of:

{"level":"warn","ts":1680120643.6689184,"logger":"http.acme_client","msg":"HTTP request failed; retrying","url":"https://acme-staging-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}

Apparently the OPNsense box had lost access to the outside, even though everything behind it was just fine–strange. A reboot of the router cleared the problem.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.