App behind haproxy AND Caddy?

[chmedly]

1. The problem I’m having:

I’m trying to use paperless-ngx behind two layers of “proxy”. I’m finding that it works from a regular web browser but two different mobile apps are failing to connect. I’m looking for suggestions and insight about what might missing or incorrect with this somewhat odd setup

The architecture might be described like this
dns → VPS → haproxy sni filtering → rathole → localserver → caddy (for ssl certificates) → paperless-ngx

The purpose of this setup is to use a single VPS to route multiple domains to various servers. Instead of terminating the https for these sites in the VPS (running caddy on the VPS) I would rather that happen at my local servers leaving the VPS is a state that is not quite “zero trust” but much closer than having the encryption terminate there. Rathole is a method for tunneling from the VPS to my local servers that might be behind CGNat etc.

2. Error messages and/or full log output:

One app (paperless mobile) simply says "Could not establish a connection to the server."

The other app (paperless) shows "HandshakeException: Connection terminated" 

3. Caddy version:

2.7.4 

4. How I installed and ran Caddy:

Docker Compose
Linux ubuntu 22.04

  caddy:
    image: caddy:latest
    container_name: caddy_paperless
    restart: unless-stopped
    #ports:
      #- "80:80"
      #- "443:443"
      #- "443:443/udp"
    networks:
      - default
      - inter_services
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./site:/srv
      - caddy_data_paperless:/data
      - caddy_config_paperless:/config

  rathole:
    image: rapiz1/rathole
    command: --client /app/config.toml
    container_name: rathole
    restart: unless-stopped
    volumes:
      - ./rathole-config.toml:/app/config.toml:ro
    networks:
      - inter_services

d. My complete Caddy config:

paper.server.com:443 {
    reverse_proxy paperless_webserver:8000
}

haproxy config file

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
 daemon
 #user                haproxy
 #group               haproxy
 log                 /dev/log local6 debug
 maxconn             5000
 #chroot              /var/lib/haproxy
 #pidfile             /var/run/haproxy.pid

#---------------------------------------------------------------------
# common defaults 
#---------------------------------------------------------------------
defaults
 mode                 tcp
 log                  global
 option               dontlognull
 timeout connect      5s
 timeout client       10s
 timeout server       10s

#---------------------------------------------------------------------
# dedicated stats page
#---------------------------------------------------------------------
#listen stats
 #mode http
 #bind :22222
 #stats enable
 #stats uri            /haproxy?stats
 #stats realm          Haproxy\ Statistics
 #stats refresh        30s

listen stats
 bind :9000
 mode http
 stats enable
 stats hide-version
 stats realm Haproxy\ Statistics
 stats uri /haproxy_stats


#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------

frontend main_https_listen
 bind *:443
 mode tcp
 acl paper_filter req_ssl_sni -i paper.server.com
 use_backend paper_backend if paper_filter


backend paper_backend
 server paper rathole:443

[francislavoie]

Hard to say without seeing logs.

[chmedly]

So, can I conclude that you don’t see anything wrong in what I’ve presented so far?
I removed haproxy from the equation and the mobile apps are working. I’ll reconfigure with haproxy in place and try to get some logs.

[francislavoie]

Without seeing Caddy logs, I can’t make an assessment on what’s not working, because I have no relevant information to look at to tell me what the problem might be.

[chmedly]

Ok, how about a different line of question?
How would you do what I’m trying to do? Would you have a different recommendation?

Thanks,

[francislavoie]

I don’t have that need, so it’s not something I’ve considered, and therefore not something I’m comfortable trying to answer.

Please post your Caddy logs. That’s the information we need to be able to help you. We need evidence of the behaviour you’re describing. Without evidence, all we can do is make guesses, and that’s not a good use of my time or yours.

[chmedly]

I think I found the issue. I needed these lines in the haproxy config.

# Wait for a client hello for at most 5 seconds
tcp-request inspect-delay 5s
tcp-request content accept if { req.ssl_hello_type 1 }

Thank you for your help. Sorry for requesting it without a log.