1. The problem I’m having: log_appended fields not included in logs when connection is aborted
I use caddy to reverse proxy a few dozen domains. I call enable_logging, shown below, to setup a domain-specific log file which includes a few log_appended fields.
Today, I noticed cases where these log_appended fields were not present in the json log files. It correlates with cases where I see errors like “aborting with incomplete response” in the caddy application logs, which I understand to indicate that the connection was aborted before all data was returned to the caller.
Even in the event of an aborted response, I would still like the log_appended fields to be present in the loged output, but I’m unable to find a way to do so. I tried format append as well, but was ending up with null values for the appended fields.
2. Error messages and/or full log output:
JSON connection logs do not include log_appended fields.
App logs at the time have:
{
"level": "warn",
"ts": 1747593144.5305367,
"logger": "http.handlers.reverse_proxy",
"msg": "aborting with incomplete response",
...
"error": "reading: context canceled"
}
3. Caddy version:
2.10.0
4. How I installed and ran Caddy:
a. System environment:
Docker (host runs ubuntu server)
b. Command:
Run via docker compose
c. Service/unit/compose file:
Docker compose file is irrelevant for this issue.
d. My complete Caddy config:
Relevant snippet:
# Determine the caller IP. If the CF-Connecting-IP header is set, the call is from CloudFlare
# and this header will be used to as the client IP. Otherwise, fall back to IP used to connect to caddy
# (remote_host resolves to request.remote_ip in the logs)
# https://developers.cloudflare.com/fundamentals/get-started/reference/http-request-headers/#cf-connecting-ip
# This in particular was done to support sending the real client IP to vaultwarden.
(find_resolved_caller_ip) {
map {http.request.header.CF-Connecting-IP} {resolved_caller_ip} {call_originates_from} {
~.+ "${0}" "cloudflare"
default "{remote_host}" "local-network"
}
}
# Configures logging to a host-specific log file.
# Arg 0: Name of the log file (no extension)
(enable_logging) {
import find_resolved_caller_ip
log_append resolved_caller_ip {resolved_caller_ip}
log_append call_originates_from {call_originates_from}
# Generate and log the full URI, including rewrites
vars {
full_uri {http.request.scheme}://{http.request.host}{http.request.uri}
}
log_append full_uri {vars.full_uri}
log {
output file /var/log/caddyaccess/{args[0]}.jsonl {
mode 640
roll_size 10MiB
roll_keep 5
}
format json {
time_format iso8601
}
}
}
# All other reverse proxy includes should use this as a base to ensure consistent header
# handling.
# Arg 0: Path to proxy
# Arg 1: A block which will be injected into the reverse proxy. It minimally must provide
# the backend "to", which should have form: "to 127.0.0.1:port"
(base_create_reverse_proxy) {
import find_resolved_caller_ip
reverse_proxy {args[0]} {
{block}
header_up X-Real-IP {resolved_caller_ip}
header_up True-Client-IP {resolved_caller_ip}
# Set HSTS with a duration of 2 years
header_down Strict-Transport-Security "max-age=63072000; includeSubDomains"
}
encode zstd gzip
}
# Creates a reverse proxy with logging enabled
# Arg 0: Path to proxy (* for all)
# Arg 1: Internal port to proxy
# Arg 2: Log file name (no ext)
(create_reverse_proxy) {
import base_create_reverse_proxy {args[0]} {
to 127.0.0.1:{args[1]}
}
import enable_logging {args[2]}
}
sample_site.my_site.com {
import create_reverse_proxy * 8000 sample_site
}