Retries for POST requests with body

[violuke (Luke Cousins)]

1. The problem I’m having:

Automatically retry POST requests (and all other HTTP verbs). In my testing GET requests are being correctly retried with no failures, but POST requests are not. POST requests are being properly retried if they don’t have a POST body, but with a body they are not. I believe the problem is related to buffering the post request bodies, but I’m not sure how to make that happen. Any help would be greatly appreciated. Thank you!

2. Error messages and/or full log output:

When the backend is down:

{"level":"error","ts":1713028574.890797,"logger":"http.log.error","msg":"readfrom tcp 172.18.0.13:48864->172.18.0.12:8000: http: invalid Read on closed Body","request":{"remote_ip":"10.0.10.1","remote_port":"51773","client_ip":"10.0.10.1","proto":"HTTP/1.1","method":"POST","host":"localhost","uri":"/dev/readiness-probe/","headers":{"User-Agent":["my-k6-user-agent"],"Content-Length":["31"]}},"duration":8.032010087,"status":502,"err_id":"m4he864dc","err_trace":"reverseproxy.statusError (reverseproxy.go:1267)"}

3. Caddy version:

2.7.6-alpine (docker)

4. How I installed and ran Caddy:

Docker compose

a. System environment:

Docker compose (Docker for Mac)

b. Command:

Docker compose

c. Service/unit/compose file:

  web:
    image: caddy:2.7.6-alpine
    container_name: breww-web
    volumes:
      - app:/app
      - ./_build/dev/caddy/Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data
      - caddy_config:/config
    ports:
      - "80:80"
    depends_on:
      - app
    networks:
      - app-tier

d. My complete Caddy config:

{
	auto_https off
}

:80 {
	root * /app
	encode gzip
	header -Server

	reverse_proxy {
		to app:8000

		lb_retries 8
		lb_policy least_conn
		lb_try_interval 1s  # Enough for KeyDB to have replicated a POST requests response cache
		lb_try_duration 8s  # Keep less than the cache time in IdempotencyMiddleware
		lb_retry_match {
			method "GET" "HEAD" "OPTIONS" "TRACE" "POST" "PUT" "DELETE"
		}

		header_up Host {http.request.host}
		header_up X-Real-IP {http.request.remote.host}
		header_up X-Forwarded-For {http.request.header.x-forwarded-for}
		header_up X-Caddy-Request-ID {http.request.uuid}  # Used by IdempotencyMiddleware to return the original response again without processing it multiple times

		# I've tried adding buffer_requests here, which doesn't break config, but doesn't solve the problem
		# buffer_requests
	}
}

5. Links to relevant resources:

Proxy load balancing retry issues

Adding buffer_requests to Caddyfile for reverse proxy

Thank you very much for any help you can offer! Here is the information below: 1. Caddy version (caddy version): v2.1.1 2. How I run Caddy: Caddy automatically starts with system, since I installed via the official caddy repository through the Debian package manager. a. System environment: OS: Debian Buster 10 b. Command: caddy reload c. Service/unit/compose file: N/A d. My complete Caddyfile or JSON config: # Add gzip compression to requests (webconf) { encode gzip } # Site imports…

---

Thank you for an amazing package and for your help!

[francislavoie]

That’s not supported currently, because the entire request body would need to be buffered (buffering with the current implementation is chunked) and that is not done by default. The retry logic would need to be able to have access to the buffer and be able to rewind it before a retry.

Typically, POST retries are unsafe because the point at which the upstream app may have triggered an error may be after having already performed some write operation.

The better thing to do would be to implement retries at the client-side, where fancier logic could be implemented based on the error response.

@violuke

		header_up Host {http.request.host}
		header_up X-Real-IP {http.request.remote.host}
		header_up X-Forwarded-For {http.request.header.x-forwarded-for}

Remove this stuff, this clobbers Caddy’s default header handling. Let Caddy do the right thing. reverse_proxy (Caddyfile directive) — Caddy Documentation

[violuke (Luke Cousins)]

Ah ok, thanks.

I appreciate that retrying POSTs is usually unsafe, but we’ve (in theory) implemented protection for this by leveraging header_up X-Caddy-Request-ID {http.request.uuid} to ensure that a cached response will be returned when a duplicate X-Caddy-Request-ID turns up within a short time of the original.

@francislavoie

Remove this stuff, this clobbers Caddy’s default header handling.

Will do! Not quite sure where they came from, to be honest, probably a bad example file I found online when we first started using Caddy.

[francislavoie]

It’s probably not impossible to implement, but it’s not been a commonly requested feature so it doesn’t exist yet. If you’re willing to dig into the code, you could take a shot at it.

[matt (Matt Holt)]

After investigating to see what it would take to implement this, I found out that the proxy is already capable of this, but I had to patch a bug by changing half a line of code first. Should work now.

lb_retries apparently not working

Let me know if it’s still not working!