Reverse_proxy - http2: request body larger than specified content length

1. The problem I’m having:

With curl/8.1.2, posting an azure alert json payload to caddy local host having a route /transform/test* with two handlers;
handler 1 -
my custom module, transforming json payload from azure alert to slack notification, increasing the payload length (from 767 to 1315)

handler 2 - reverse_proxy to slack web hook, reporting the error

I have tested an adjusted azure alert with length 1315 (adding dummy data), then everything is working with /transform/test*.

Is this use case, reverse_proxy of a transformed payload, kind of misuse?
If not, any advice?

Everything is in my repo

2. Error messages and/or full log output:

alert-t
2023/09/20 18:11:19.236	DEBUG	http.handlers.az_alert_slack_notification	before transformation	{"request": {"remote_ip": "127.0.0.1", "remote_port": "62723", "client_ip": "127.0.0.1", "proto": "HTTP/1.1", "method": "POST", "host": "localhost", "uri": "/transform/test", "headers": {"User-Agent": ["curl/8.1.2"], "Accept": ["*/*"], "Content-Type": ["application/json"], "Content-Length": ["767"]}}}
2023/09/20 18:11:19.236	DEBUG	http.handlers.az_alert_slack_notification	before body	{"body": "{\"schemaId\":\"azureMonitorCommonAlertSchema\",\"data\":{\"essentials\":{\"alertId\":\"/subscriptions/9876/providers/Microsoft.AlertsManagement/alerts/b9569717-bc32-442f-add5-83a997729330\",\"alertRule\":\"Test-Rule-1\",\"severity\":\"Sev4\",\"signalType\":\"Metric\",\"monitorCondition\":\"Fired\",\"monitoringService\":\"Platform\",\"alertTargetIDs\":[\"/subscriptions/1234/resourcegroups/pipelinealertrg/providers/microsoft.compute/virtualmachines/wcus-r2-gen2\"],\"configurationItems\":[\"wcus-r2-gen2\"],\"originAlertId\":\"3f2d4487-b0fc-4125-8bd5-7ad17384221e_PipeLineAlertRG_microsoft.insights_metricAlerts_WCUS-R2-Gen2_-117781227\",\"firedDateTime\":\"2019-03-22T13:58:24.3713213Z\",\"resolvedDateTime\":\"2019-03-22T14:03:16.2246313Z\",\"description\":\"\",\"essentialsVersion\":\"1.0\",\"alertContextVersion\":\"1.0\"}}}"}
2023/09/20 18:11:19.236	DEBUG	http.handlers.az_alert_slack_notification	transformed body	{"body": "{\"blocks\":[{\"type\":\"divider\"},{\"type\":\"header\",\"text\":{\"type\":\"plain_text\",\"text\":\"New monitor alert!\",\"emoji\":true}},{\"type\":\"divider\"},{\"type\":\"actions\",\"elements\":[{\"type\":\"button\",\"text\":{\"type\":\"plain_text\",\"text\":\"View alert in Monitor\",\"emoji\":true},\"url\":\"https://portal.azure.com/#blade/Microsoft_Azure_Monitoring/AlertDetailsTemplateBlade/alertId/%2Fsubscriptions%2F9876%2Fproviders%2FMicrosoft.AlertsManagement%2Falerts%2Fb9569717-bc32-442f-add5-83a997729330\",\"style\":\"primary\"}]},{\"type\":\"header\",\"text\":{\"type\":\"plain_text\",\"text\":\"Summary\",\"emoji\":true}},{\"type\":\"divider\"},{\"type\":\"section\",\"fields\":[{\"type\":\"mrkdwn\",\"text\":\"Fired (UTC)\"},{\"type\":\"plain_text\",\"text\":\"2019-03-22T13:58:24.3713213Z\",\"emoji\":true},{\"type\":\"mrkdwn\",\"text\":\"Alert name\"},{\"type\":\"plain_text\",\"text\":\"Test-Rule-1\",\"emoji\":true},{\"type\":\"mrkdwn\",\"text\":\"Severity\"},{\"type\":\"plain_text\",\"text\":\":speech_balloon:  Verbose(4)\",\"emoji\":true}]},{\"type\":\"header\",\"text\":{\"type\":\"plain_text\",\"text\":\"Resources\",\"emoji\":true}},{\"type\":\"divider\"},{\"type\":\"section\",\"fields\":[{\"type\":\"mrkdwn\",\"text\":\"\\u003chttps://portal.azure.com/#@nav.no/resource/subscriptions/1234/resourcegroups/pipelinealertrg/providers/microsoft.compute/virtualmachines/wcus-r2-gen2|wcus-r2-gen2\\u003e\"},{\"type\":\"plain_text\",\"text\":\":link:\",\"emoji\":true}]}]}"}
2023/09/20 18:11:19.237	DEBUG	http.handlers.az_alert_slack_notification	after transformation	{"request": {"remote_ip": "127.0.0.1", "remote_port": "62723", "client_ip": "127.0.0.1", "proto": "HTTP/1.1", "method": "POST", "host": "localhost", "uri": "/transform/test", "headers": {"User-Agent": ["curl/8.1.2"], "Accept": ["*/*"], "Content-Type": ["application/json"], "Content-Length": ["1315"]}}}
2023/09/20 18:11:19.237	DEBUG	http.handlers.reverse_proxy	selected upstream	{"dial": "hooks.slack.com:443", "total_upstreams": 1}
2023/09/20 18:11:19.382	DEBUG	http.handlers.reverse_proxy	upstream roundtrip	{"upstream": "hooks.slack.com:443", "duration": 0.145030334, "request": {"remote_ip": "127.0.0.1", "remote_port": "62723", "client_ip": "127.0.0.1", "proto": "HTTP/1.1", "method": "POST", "host": "hooks.slack.com:443", "uri": "/services/secret/secret", "headers": {"X-Forwarded-For": ["127.0.0.1"], "X-Forwarded-Proto": ["http"], "X-Forwarded-Host": ["localhost"], "Content-Length": ["1315"], "User-Agent": ["curl/8.1.2"], "Accept": ["*/*"], "Content-Type": ["application/json"]}}, "error": "http2: request body larger than specified content length"}
2023/09/20 18:11:19.382	ERROR	http.log.error	http2: request body larger than specified content length	{"request": {"remote_ip": "127.0.0.1", "remote_port": "62723", "client_ip": "127.0.0.1", "proto": "HTTP/1.1", "method": "POST", "host": "localhost", "uri": "/transform/test", "headers": {"Content-Length": ["767"], "User-Agent": ["curl/8.1.2"], "Accept": ["*/*"], "Content-Type": ["application/json"]}}, "duration": 0.145963458, "status": 502, "err_id": "zn45wg9fa", "err_trace": "reverseproxy.statusError (reverseproxy.go:1248)"}
HTTP/1.1 502 Bad Gateway
Server: Caddy
Date: Wed, 20 Sep 2023 18:11:19 GMT
Content-Length: 0

3. Caddy version:

v2.7.4 h1:J8nisjdOxnYHXlorUKXY75Gr6iBfudfoGhrJ8t7/flI=

4. How I installed and ran Caddy:

# in repo root

# new release of my module github.com/ttnesby/slack-block-builder/caddy-ext/azalertslacknotification
export def r-ca [] {
    let ver = $in
    gh release create ($ver) --notes "wip"
    b-ca $ver
}

# build caddy with custom module, using xcaddy
export def b-ca [ver: string] {
    let ext1 = $"github.com/ttnesby/slack-block-builder/caddy-ext/azalertslacknotification@($ver)" 
    ~/go/bin/xcaddy build --with ($ext1)
}

# start caddy with Caddyfile
export def u-ca [] {
    ./caddy start Caddyfile
}

# stop caddy
export def d-ca [] {
    ./caddy stop
}

a. System environment:

macOS Ventura 13.5.2 (22G91)
GOHOSTARCH=‘arm64’
GOHOSTOS=‘darwin’

b. Command:

./caddy start Caddyfile

c. Service/unit/compose file:

d. My complete Caddy config:

{
	log {
		format console
		level debug
	}
}

(to-slack) {
	reverse_proxy https://hooks.slack.com {
		header_up Host {upstream_hostport}
		rewrite {$SLACK_TEST_NOTIFICATION}
	}
}

http://localhost {
	#encode zstd gzip
	# rate_limit details
	# waf details

	route /transform/test* {
		az_alert_slack_notification
		#respond {http.request.body}
		import to-slack
	}

	route /direct/test* {
		import to-slack
	}

	handle /health* {
		respond "I'm fine" 200
	}
}

5. Links to relevant resources:

Well, it depends on what your handler is doing. I don’t have access to the code, so it’s hard to point anything out.

But searching for that error message, it comes from the Go stdlib. The way stdlib gets the content length is from req.ContentLength. Are you only updating the header and not req.ContentLength as well? Maybe that’ll resolve it.

1 Like

Spot on, thank you!

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.