[Solved] What happens when the forward_auth service cannot be reached? All doors open?

1. The problem I’m having:

I am trying to set up a site served by caddy behind our organization’s (OIDC) Single Sign On. I want to use forward_auth to an oauth2-proxy running (as a container) on the same machine as caddy. At the moment I have not asked our IT dpt to register my service yet, so the oauth2-proxy fails to set up the OIDC and quits right away. But now the problem is that, even without signing in, I can access the site anyway. Caddy does not seem to even get to the error handling/login enforcing part inside the forward_auth directive, or it has no handling of 502-type failures rather than 401. Instead, with the forwarding server failing altogether, it just resumes processing of the Caddyfile after the forward_auth part and happily serves the webroot that should be accessible only for logged-in viewers. Am I understanding what happens correctly? What should I set up differently so as not to have an open site when, for whatever reason, the SSO mechanism itself fails?

2. Error messages and/or full log output:

curl output:

awagner@LEN124:~/vcs/mpilhlt/ansible$ curl -vL https://c107-104.cloud.gwdg.de/oauth2/login
* Host c107-104.cloud.gwdg.de:443 was resolved.
* IPv6: (none)
* IPv4: 141.5.107.104
*   Trying 141.5.107.104:443...
* Connected to c107-104.cloud.gwdg.de (141.5.107.104) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / X25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=c107-104.cloud.gwdg.de
*  start date: Jan 27 13:16:50 2025 GMT
*  expire date: Apr 27 13:16:49 2025 GMT
*  subjectAltName: host "c107-104.cloud.gwdg.de" matched cert's "c107-104.cloud.gwdg.de"
*  issuer: C=US; O=Let's Encrypt; CN=E6
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://c107-104.cloud.gwdg.de/oauth2/login
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: c107-104.cloud.gwdg.de]
* [HTTP/2] [1] [:path: /oauth2/login]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
> GET /oauth2/login HTTP/2
> Host: c107-104.cloud.gwdg.de
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/2 502
< access-control-allow-headers: *
< access-control-allow-methods: *
< alt-svc: h3=":443"; ma=2592000
< server: Caddy
< content-length: 0
< date: Mon, 27 Jan 2025 14:44:00 GMT
<
* Connection #0 to host c107-104.cloud.gwdg.de left intact

awagner@LEN124:~/vcs/mpilhlt/ansible$ curl -vL https://c107-104.cloud.gwdg.de/test.html
* Host c107-104.cloud.gwdg.de:443 was resolved.
* IPv6: (none)
* IPv4: 141.5.107.104
*   Trying 141.5.107.104:443...
* Connected to c107-104.cloud.gwdg.de (141.5.107.104) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / X25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=c107-104.cloud.gwdg.de
*  start date: Jan 27 13:16:50 2025 GMT
*  expire date: Apr 27 13:16:49 2025 GMT
*  subjectAltName: host "c107-104.cloud.gwdg.de" matched cert's "c107-104.cloud.gwdg.de"
*  issuer: C=US; O=Let's Encrypt; CN=E6
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://c107-104.cloud.gwdg.de/test.html
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: c107-104.cloud.gwdg.de]
* [HTTP/2] [1] [:path: /test.html]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
> GET /test.html HTTP/2
> Host: c107-104.cloud.gwdg.de
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/2 200
< access-control-allow-headers: *
< access-control-allow-methods: *
< alt-svc: h3=":443"; ma=2592000
< content-type: text/html; charset=utf-8
< server: Caddy
< vary: Accept-Encoding
< content-length: 59
< date: Mon, 27 Jan 2025 14:44:09 GMT
<
<!DOCTYPE html>
<html>
<body>
<h1>Test</h1>
</body>
* Connection #0 to host c107-104.cloud.gwdg.de left intact

3. Caddy version:

awagner@c107-104 ~ $ caddy version
v2.9.1 h1:OEYiZ7DbCzAWVb6TNEkjRcSCRGHVoZsJinoDR/n9oaY=

4. How I installed and ran Caddy:

a. System environment:

Ubuntu 22.04, systemd 249 (249.11-0ubuntu3.12)

b. Command:

sudo systemctl start caddy.service

c. Service/unit/compose file:

systemd service unit definition:

awagner@c107-104 ~ $ cat /etc/systemd/system/caddy.service
[Unit]
Description=Caddy HTTP/2 web server
Documentation=https://caddyserver.com/docs
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service

; Do not allow the process to be restarted in a tight loop. If the
; process fails to start, something critical needs to be fixed.
StartLimitIntervalSec=86400
StartLimitBurst=10


[Service]
Restart=on-failure
RestartSec=1080

; User and group the process will run as.
User=www-data
Group=podman

ExecStart=/usr/local/bin/caddy run --config=/etc/caddy/Caddyfile --watch
ExecReload=/usr/local/bin/caddy reload --config=/etc/caddy/Caddyfile
ExecStop=/usr/local/bin/caddy stop

; Limit the number of file descriptors; see `man systemd.exec` for more limit settings.
LimitNOFILE=1048576

; Use graceful shutdown with a reasonable timeout
KillMode=mixed
KillSignal=SIGQUIT
TimeoutStopSec=5s

; Use private /tmp and /var/tmp, which are discarded after caddy stops.
PrivateTmp=true
; Use a minimal /dev (May bring additional security if switched to 'true', but it may not work on Raspberry Pi's or other devices)
PrivateDevices=true
; Hide /home, /root, and /run/user. Nobody will steal your SSH-keys.
ProtectHome=false
; Make /usr, /boot, /etc and possibly some more folders read-only.
ProtectSystem=full

; The following additional security directives only work with systemd v229 or later.
; They further restrict privileges that can be gained by caddy.
; Note that you may have to add capabilities required by any plugins in use.
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

awagner@c107-104 ~ $ sudo caddy fmt --config /etc/caddy/Caddyfile
{
        order replace after encode
        log {
                output file /var/log/caddy/access.log
                format filter {
                        wrap console
                        fields {
                                request>remote_ip ip_mask {
                                        ipv4 16
                                        ipv6 32
                                }
                                request>remote_addr ip_mask {
                                        ipv4 16
                                        ipv6 32
                                }
                        }
                }
        }
        email wagner@lhlt.mpg.de
        #  acme_ca https://acme-staging-v02.api.letsencrypt.org/directory # Comment this out if everything works fine
        auto_https disable_redirects
}

c107-104.cloud.gwdg.de {
        # Requests to /oauth2/* are proxied to oauth2-proxy without authentication.
        # You can't use `reverse_proxy /oauth2/* oauth2-proxy.internal:4180` here because the reverse_proxy directive has lower precedence than the handle directive.
        handle /oauth2/* {
                reverse_proxy localhost:4180 {
                        # oauth2-proxy requires the X-Real-IP and X-Forwarded-{Proto,Host,Uri} headers.
                        # The reverse_proxy directive automatically sets X-Forwarded-{For,Proto,Host} headers.
                        header_up X-Real-IP {remote_host}
                        header_up X-Forwarded-Uri {uri}
                }
        }
        # Requests to other paths are first processed by oauth2-proxy for authentication.
        handle {
                forward_auth localhost:4180 {
                        uri /oauth2/auth
                        # uri /oauth2/ # if the above does not work, try this

                        # oauth2-proxy requires the X-Real-IP and X-Forwarded-{Proto,Host,Uri} headers.
                        # The forward_auth directive automatically sets the X-Forwarded-{For,Proto,Host,Method,Uri} headers.
                        header_up X-Real-IP {remote_host}

                        # If needed, you can copy headers from the oauth2-proxy response to the request sent to the upstream.
                        # Make sure to configure the --set-xauthrequest flag to enable this feature.
                        copy_headers X-Auth-Request-User X-Auth-Request-Email

                        # If oauth2-proxy returns a 401 status, redirect the client to the sign-in page.
                        @error status 401
                        handle_response @error {
                                redir * /oauth2/sign_in?rd={scheme}://{host}{uri}
                        }
                }
        }

        # Templates give static sites some dynamic features
        templates

        # Compress responses according to Accept-Encoding headers
        encode zstd gzip

        # Enable http2 pushing
        push

        # enable CORS headers
        header {
                # Access-Control-Allow-Origin *
                Access-Control-Allow-Methods *
                Access-Control-Allow-Headers *
        }

        handle_path /* {
                root * /var/data/caddy/site
                file_server
                # Make HTML file extension optional
                try_files {path}.html {path}.en.html {path}.de.html {path}.fr.html {path}.nl.html {path}.es.html {path}
        }
}

5. Links to relevant resources:

The Caddyfile above is a combination of what I have used successfully in a different context without SSO integration, and this document at the oauth2-proxy site: https://oauth2-proxy.github.io/oauth2-proxy/configuration/integration#configuring-for-use-with-the-caddy-v2-forward_auth-directive

It’s solved. It was a matter of nesting in my Caddyfile - the upstream services should be inside the global handler (that in turn contains the forward_auth block), not after it.

That was the only thing I had to change and now it’s working. If the forward_auth service canmot be reached, now I get an empty response instead of my upstream site. I guess I could add more status codes to the error handling, but that is an exercise for later…

2 Likes