Website login issues

[prod]

1. The problem I’m having:

I am having various issues when trying to log into to proxied sites. Two example:

BlueIris camera server. The site initially loads fine, but when trying to log in, the page displays the following error: Unable to contact Blue Iris Server

Bitwarden server. SSL is configured on the backend. The site initially loads fine, but when trying to log in, it hangs for about a minute (presumably timeout setting), then throws an error. If I hit login again, it will immediately login just fine. This cycle happens every time.

Not sure how to troubleshoot this.

Note: Sites are only accessible internally; no WAN access.

2. Error messages and/or full log output:

journalctl doesn’t seem to exist in the docker container. Here are access logs from enabling logging and trying to visit the Blue Iris server:

Logs - Pastebin.com

curl output won’t be an accurate representation because certificates are installed in browsers rather than OS trust store. Happy to work on this if curl output would be of use. If any other logs are needed, I can find them too.

3. Caddy version:

v2.8.4

4. How I installed and ran Caddy:

Caddy was built using xcaddy (2.8.4-builder) with Coraza plugin using a Dockerfile that is called in the compose file.

Network has an internal PKI. Certificates are managed on IDM server. A wildcard cert was generated for Caddy to use. Some servers use [require] SSL, and have SSL certificates from the PKI. The root cert was added for servers that have SSL on the backend.

a. System environment:

Ubuntu 24.04
Docker Compose

b. Command:

docker compose up -d

c. Service/unit/compose file:

services:
  caddy:
    build:
      dockerfile: ./Dockerfile
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - $PWD/Caddyfile:/etc/caddy/Caddyfile
      - $PWD/site:/srv
      - caddy_data:/data
      - caddy_config:/config
      - $PWD/ssl:/sslcerts

volumes:
  caddy_data:
  caddy_config:

d. My complete Caddy config:

{
    order coraza_waf first
}

*.contoso.com {
    coraza_waf {
      load_owasp_crs
      directives `
        Include @coraza.conf-recommended
        Include @crs-setup.conf.example
        Include @owasp_crs/*.conf
        SecRuleEngine On
        `
    }
    log {
      output file /var/log/access.log
    }
    tls /sslcerts/wildcard.pem /sslcerts/wildcard.key
    @wiki host wiki.contoso.com
    handle @wiki {
      reverse_proxy 192.168.0.10:3000
    }
    @cams host cams.contoso.com
    handle @cams {
      reverse_proxy 172.168.22.5
    }
    @bitwarden host bitwarden.contoso.com
    handle @bitwarden {
      reverse_proxy 192.168.0.9:443 {
        transport http {
          tls_trusted_ca_certs /sslcerts/root.pem
          tls_server_name bitwarden.contoso.com
         }
      }
    }
}

5. Links to relevant resources:

These are some native options that can be adjusted inside of Blue Iris

Error within Blue Iris (don’t get it when not using proxy)

[prod]

Bump.

Not sure how to troubleshoot this. Any direction or advice would be very appreciated!

[Whitestrake (Matthew Fay)]

Howdy @prod,

Firstly, I have to ask: are you running a homelab on Microsoft’s domain name?! Because that’d have to be some wild use of split horizon DNS.

As for troubleshooting, I don’t see an immediate answer for you, but I can offer some advice as to how I’d personally proceed.

It seems like this may be client/upstream related since I’m not seeing any Caddy errors in your logs and I’m not seeing any glaring logical mistakes, but it never hurts to eliminate as much extraneous configuration as possible in order to narrow down the space for complexity around the issue.

So, I’d start with one of the apps and just reverse proxy it. Minimum possible config for it to work. e.g.:

{
  debug
}

bitwarden.example.com {
  reverse_proxy 192.168.0.9:443 {
    transport http {
      tls_insecure_skip_verify
    }
  }
}

Then give it another shot and see what happens.

As for Blue Iris, do you get any web console errors? When I see errors like that on a page that loaded just fine I assume there’s some kind of AJAX / out-of-band issue (like trying to open a websocket to the wrong host or stuff like that).

[prod]

@Whitestrake

Firstly, I have to ask: are you running a homelab on Microsoft’s domain name?! Because that’d have to be some wild use of split horizon DNS.

Heh, nah. That would be some unique DNS work. Just a little :%s/domain.com/contoso.com/g work to maintain some privacy. Hard to rationalize going crazy with reverse proxy/WAF/etc while throwing any opsec out of the window

Looking at the browser console was certainly the right place to start. I found errors for both sites.

Blue Iris:
It seems there was an error with the login POST.

Bitwarden:
This one is interesting. I did try the basic Caddyfile, but got the same results. However, when looking at the browser console, if incorrect credentials are supplied, it immediately returns an unsuccessful error response despite appearing to continue hanging until timeout. If successful creds are supplied, no response is returned while it hangs until it times out, at which point a 504 is returned. Bitwarden is ran in a Docker container, and apparently behind Nginx (which returned the error).

After the timeout, if I hit login again, it instantly logs in. The console still shows a connection error (which includes an access_token that it seems to get somewhere), as well as three similar websocket errors including this one:

Unhandled Promise rejection: WebSocket failed to connect. The connection could not be found on the server, either the endpoint may not be a SignalR endpoint, the connection ID is not present on the server, or there is a proxy blocking WebSockets. If you have multiple servers check that sticky sessions are enabled. ; Zone: ; Task: null ; Value: Error: WebSocket failed to connect. The connection could not be found on the server, either the endpoint may not be a SignalR endpoint, the connection ID is not present on the server, or there is a proxy blocking WebSockets. If you have multiple servers check that sticky sessions are enabled.

[Whitestrake (Matthew Fay)]

@prod

Heh, nah. That would be some unique DNS work. Just a little :%s/domain.com/contoso.com/g work to maintain some privacy. Hard to rationalize going crazy with reverse proxy/WAF/etc while throwing any opsec out of the window

Ahh, right!

By the way, example.com is reserved by the IANA explicitly for this purpose.

@prod

Blue Iris:
It seems there was an error with the login POST.

Do you have any logs from the upstream for this request, any indication why it returned a 403?

@prod

Bitwarden:
This one is interesting. I did try the basic Caddyfile, but got the same results. However, when looking at the browser console, if incorrect credentials are supplied, it immediately returns an unsuccessful error response despite appearing to continue hanging until timeout. If successful creds are supplied, no response is returned while it hangs until it times out, at which point a 504 is returned. Bitwarden is ran in a Docker container, and apparently behind Nginx (which returned the error).

I don’t think you’ve specified if this is an official Bitwarden server or Vaultwarden, or how it’s configured, so we’re a little in the dark here. You mentioned a websocket failure, what host/port is this happening on? Is this connection going through Caddy or is it trying to go out-of-band? As for nginx - you’ll need to go through the logs from that server, and you may need to refer back to the installation/reverse-proxy documentation for whichever server you’re using to find out what exactly its requirements are.