Caddy reverse proxy HTTP 502 error reason logging for < 1% failing requests

1. Caddy version:

v2.4.3 h1:Y1FaV2N4WO3rBqxSYA8UZsZTQdN+PwcoOcAiZTM8C0I=

2. How I installed, and run Caddy:


a. System environment:

Ubuntu 20.04
Docker version 20.10.18, build b40c2f6

b. Command:


c. Service/unit/compose file:

version: '3'
  # Running Caddy as command
    image: caddy:v2.4.5
    container_name: caddy
      context: .
      dockerfile: Dockerfile
    restart: unless-stopped
    command: caddy run -config /etc/caddy/Caddyfile
      ACME_AGREE: "true"
    # We directly bind to the host ports and there is no Docker network here
    network_mode: host
      - $PWD/Caddyfile:/etc/caddy/Caddyfile
      - $PWD/data:/data
      - $PWD/data/config:/config
      - $PWD/index.html:/usr/share/caddy/index.html
      - $PWD/logs:/var/log/caddy # shared volume with caddy
      - /home/docs/html:/docs  # For docs static hosting
  # Logstash logging for caddy from Json logs
    image: logstash:8.2.0
    container_name: caddy-logstash
    network_mode: host
      - ~/oracle.env
      # elastisearch login vars
      - "5044:5044"
      - "5000:5000/tcp"
      - "5000:5000/udp"
      - "9600:9600"
      - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro,Z
      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro,Z
      - $PWD/logs:/data/logs #shared volume
      - caddy

d. My complete Caddy config:

# Caddy configuration for Trading Strategy website/backend APIs.
# Contains both production and staging environments.
    # Disable the Caddy admin API
    # This is personal preference, you can remove this if desired
    admin off

    log {
        output file /var/log/caddy/access.log
        format json

# Production frontend and backend and docs
# PYramid backend API server is localhost:3456
# SvelteKit Node.js frontend SSR server is localhost:3000
# {

    # Backend API request
    handle /api* {
        # This is the upstream Waitress server
        reverse_proxy {
            # Backend API must respond to an individual API call under 20s seconds
            transport http {
                response_header_timeout 20s

    # Special URL endpoint for Google and other bots
    # that are robots.txt blocked from API
    handle_path /sitemap/* {
        # Same as /api, just a path rewrite using Caddy handle_path
        rewrite * /api{path}

    # Serve docs as files rsync'ed to the server
    # See trading-strategy/.github/workflows/rsync-docs.yml for more details
    redir /docs /docs/
    handle_path /docs* {
        # See docker-compose
        # mapped to /home/docs/html
        root * /docs

    # SvelteKit production server from frontend repository
    handle {
        reverse_proxy {
        header_up X-Forwarded-Host {host}
            # Frontend must render the page under 20 seconds
            transport http {
                response_header_timeout 20s

    # Set the default 404 page
    handle_errors {
        respond "{http.error.status_code} {http.error.status_text}"

    # Create a log file.
    # This is mapped to local FS thru docker-compose, so you can read it like
    # tail -f logs/access.log  | jq
    # but it is also consumed by logstash log forwarder
    log {
        format json
        output file /var/log/caddy/access.log
        #output stdout
        # Caddy cannot have two log outputs at the same time


3. The problem I’m having:

Caddy connection reverse proxy fails for < 1% requests randomly. I would like to pin point the underlying cause (fault in Caddy, fault in operating system, fault in the upstream web server). This is on a Docker setup using localhost connections, so network issues can be ruled out.

I traced down the request to the upstream server using CloudFlare ray headers. It seems that the upstream web server never sees the request, which is natural because the error is 502 Bad Gateway.

I could not find out how to extract the underlying operating system error reason behind HTTP 502 responses Caddy is serving. Does Caddy have any sort of error logs where underlying error reasons would be logged and how would this be turned on? As the error reason could be related to timeout, sockets, etc.

I do not see anything relevant in the access log JSON or stdout output.

4. Error messages and/or full log output:

The problem is that there is no error messages in Caddy output or logs.

6. Links to relevant resources:

That’s quite an old version. Please upgrade to v2.6.2.

Remove these. That’s not needed. And it’s using a single dash for -config which will break when you upgrade to the latest version.

Remove this as well. There’s no reason to override that file.

You won’t need this either.

Impossible to say without logs. You need to show evidence of what happens with those requests.

You can turn on the debug global option, which will show more details in the logs.

But any 502 should be in the logs as an error.

Keep in mind, this only configures access logs. You need to look at Caddy’s process logs, which are written to stdout by default (so, look at your container logs).

1 Like

Thank you Francis. The problem is that the HTTP 500 codes get logged in the access log, but it does not tell why this error happened (cannot connect socket, some other reason).

I will try with debug option to see if it will give me a clue.

Responses with the 500 status code are probably coming from your upstream app itself, not from Caddy. In that case, it means your app had an error.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.