GCP Multiple regions TCP Load Balancer Setup With Caddy

[bcode (Besart K)]

1. The problem I’m having:

I setup the caddy as ssl-manager on MIG on GCP, and it sends traffic to another mig with reverse_proxy and internal load balancer, and I am having troubles and inconsistent results when I setup GCP multiple regions Load Balancer.
It is working perfectly with Reginal GCP load balancer.

I have these configurations on Load Balancer

I have this configurations on MIG

With the current working regional Load Balancing
I dont need to setup any port

2. Error messages and/or full log output:

When I run this multiple times it shows different results, some errors some not.

curl http://one-of-the-domains.com
Client sent an HTTP request to an HTTPS server.

Other errors show something with tls version error
Also what I found is that tcp show remote client port to something random

"RemoteAddr":{"IP":"...","Port":38950,"Zone":""},"LocalAddr":{"IP":"....","Port":443,"Zone":""}}}}

3. Caddy version:

caddy:2.7.6-builder-alpine

4. How I installed and ran Caddy:

a. System environment:

Docker

b. Command:

caddy run -c Caddyfile

c. Service/unit/compose file:

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

d. My complete Caddy config:

{
	{$CADDY_DEBUG} 
	{$CADDY_LOCAL_CERTS}
	on_demand_tls {
		ask {$CADDY_SSL_ASK_ENDPOINT}
		burst 100
	}
	email {$CADDY_SSL_EMAIL}
	storage gcs {
		bucket-name {$CADDY_GCS_BUCKET_NAME}
	}
	auto_https disable_redirects
}

http://*.{$CADDY_SERVE_STAGING_DOMAIN}:80 {
	log "staging_domain_handler_http"
	reverse_proxy {$CADDY_REVERSE_PROXY_ENDPOINT} {
	}
}

:80 {
	log "http_handler"
	handle /health-check {
		respond "OK" 200
	}
	@ip_address {
		header_regexp Host ^(?:[0-9]{1,3}\.){3}[0-9]{1,3}
	}
	handle @ip_address {
		respond "" 400
	}
	handle {
		redir https://{host}{uri}
	}
}

:443 {
	log "main_domain_handler"
	handle /health-check {
		respond "OK" 200
	}
	reverse_proxy {$CADDY_REVERSE_PROXY_ENDPOINT} {
	}
	tls {
		on_demand
	}
}

5. Links to relevant resources:

[francislavoie]

@bcode

Also what I found is that tcp show remote client port to something random

That’s normal, that’s how TCP networking works. The remote client is assigned a random port that the server can use to write the response.

@bcode

When I run this multiple times it shows different results, some errors some not.

curl http://one-of-the-domains.com
Client sent an HTTP request to an HTTPS server.

This seems like you have an incorrect port mapping somewhere. Port 80 should go to port 80, and 443 to 443. Don’t cross the wires.

[bcode (Besart K)]

@francislavoie

This seems like you have an incorrect port mapping somewhere. Port 80 should go to port 80, and 443 to 443. Don’t cross the wires.

That’s what I thought, I think that the load balancer is just sending some requests with port 80 to 443 or vice versa. But I don’t see a way to map exactly 80-80, 443-443

[francislavoie]

This is more a question for GCP than Caddy.

[bcode (Besart K)]

I thought that maybe someone here had a similar problem, or someone with a more knowledge on GCP(than me) would help me with any idea on how I can solve this.

I hope that it is not a problem I posted here.
Thanks