Problems with dynamic upstreams using DNS A records

1. The problem I’m having:

I am trying to use Caddy as a reverse proxy with dynamic upstreams. The upstreams are supposed to be resolved via DNS A records.
The upstream servers and caddy are in a docker network.

This seems to “kinda” work. I can reach the backend from the frontend however the health_check does not seem to work. No matter what I make the heath check return the upstreams are still reachable. There are also no health check related messages in the log.
If I stop all upstream servers I get the following message:

{"level":"error","ts":1679922847.3636703,"logger":"http.handlers.reverse_proxy","msg":"failed getting dynamic upstreams; falling back to static upstreams","error":"lookup react-gateway on 127.0.0.11:53: server misbehaving"}

Which makes sense.

Also querying the API at reverse_proxy/upstreams returns an empty list. (see next point)

2. Error messages and/or full log output:

curl "http://localhost:2019/reverse_proxy/upstreams"
[]

3. Caddy version:

v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=

4. How I installed and ran Caddy:

Official Caddy docker image caddy:2.6

a. System environment:

b. Command:

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

c. Service/unit/compose file:

networks:
  default:
    driver: bridge
  frontend:
  backend:

  react-gateway:
    image: registry.example.com/react-gateway
    hostname: react-gateway
    restart: unless-stopped
    networks:
      frontend:
      backend:
    deploy:
      replicas: 4

  react-client:
    image: registry.example.com/react-client:caddy
    restart: unless-stopped
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
    networks:
      frontend:
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./caddy_data:/data

d. My complete Caddy config:

example.com {
	push

	encode zstd gzip

	header {
		# Hide "Server: Caddy"
		-Server

		Content-Security-Policy default-src 'self'
		X-XSS-Protection 1; mode=block
		Strict-Transport-Security max-age=31536000; includeSubDomains; preload
		X-Frame-Options DENY
		X-Frame-Options: SAMEORIGIN
		X-Content-Type-Options nosniff
		Referrer-Policy no-referrer-when-downgrade
		Content-Security-Policy upgrade-insecure-requests
		Feature-Policy "accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'self'; camera 'none'; encrypted-media 'none'; fullscreen 'self'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; midi 'none'; payment 'none'; picture-in-picture *; speaker 'none'; sync-xhr 'none'; usb 'none'; vr 'none'"
	}

	log {
		output stdout
		format json
	}

	handle /rest/* {
		reverse_proxy * {
			dynamic a {
				name react-gateway
				port 3001
				refresh 5s
				resolvers 127.0.0.11:53
			}
			header_up X-Real-IP {remote}

			health_uri /rest/infrastructure/isHealthy # Backend health check path
			# health_port     80 # Default same as backend port
			health_interval 20s
			health_timeout 10s
			#health_status 200
			health_body healthy
		}
	}
	handle {
		root * /var/www/html
		try_files {path} /index.html
		file_server
	}

}

5. Links to relevant resources:

The problem with active health checks + dynamic upstreams is that fetching the list of dynamic upstreams requires a request, since the dynamic upstreams interface takes the incoming request as input for maximum flexibility. Active health checks happen in the background, and aren’t triggered by a request, so technically there’s no known upstreams for active health checks to try.

This is mentioned in the docs, at the top of the dynamic upstreams section:

Same deal.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.