Caddy reverse proxy HTTP 502 error reason logging for < 1% failing requests

Help
1

1. Caddy version:

v2.4.3 h1:Y1FaV2N4WO3rBqxSYA8UZsZTQdN+PwcoOcAiZTM8C0I=

2. How I installed, and run Caddy:

Docker`

a. System environment:

Ubuntu 20.04
Docker version 20.10.18, build b40c2f6

b. Command:

caddy

c. Service/unit/compose file:

version: '3'
services:
  # https://hub.docker.com/_/caddy
  # https://caddyserver.com/docs/quick-starts/reverse-proxy
  # Running Caddy as command
  caddy:
    image: caddy:v2.4.5
    container_name: caddy
    build:
      context: .
      dockerfile: Dockerfile
    restart: unless-stopped
    command: caddy run -config /etc/caddy/Caddyfile
    environment:
      ACME_AGREE: "true"
    # We directly bind to the host ports and there is no Docker network here
    network_mode: host
    volumes:
      - $PWD/Caddyfile:/etc/caddy/Caddyfile
      - $PWD/data:/data
      - $PWD/data/config:/config
      - $PWD/index.html:/usr/share/caddy/index.html
      - $PWD/logs:/var/log/caddy # shared volume with caddy
      - /home/docs/html:/docs  # For docs static hosting
  # Logstash logging for caddy from Json logs
  caddy-logstash:
    image: logstash:8.2.0
    container_name: caddy-logstash
    network_mode: host
    env_file:
      - ~/oracle.env
    environment:
      # elastisearch login vars
      ECS_SERVER: ${ECS_SERVER}
      ECS_USER: ${ECS_USER}
      ECS_PASSWORD: ${ECS_PASSWORD}
    ports:
      - "5044:5044"
      - "5000:5000/tcp"
      - "5000:5000/udp"
      - "9600:9600"
    volumes:
      - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro,Z
      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro,Z
      - $PWD/logs:/data/logs #shared volume
    depends_on:
      - caddy

d. My complete Caddy config:

#
# Caddy configuration for Trading Strategy website/backend APIs.
#
# Contains both production and staging environments.
#
#
{
    # Disable the Caddy admin API
    # This is personal preference, you can remove this if desired
    admin off
    email no-reply@tradingstrategy.ai

    log {
        output file /var/log/caddy/access.log
        format json
    }
}

#
# Production frontend and backend and docs
#
# PYramid backend API server is localhost:3456
# SvelteKit Node.js frontend SSR server is localhost:3000
#
http://example.com {

    # Backend API request
    handle /api* {
        # This is the upstream Waitress server
        reverse_proxy 127.0.0.1:3456 {
            # Backend API must respond to an individual API call under 20s seconds
            transport http {
                response_header_timeout 20s
            }
        }
    }

    # Special URL endpoint for Google and other bots
    # that are robots.txt blocked from API
    handle_path /sitemap/* {
        # Same as /api, just a path rewrite using Caddy handle_path
        # https://webinstall.dev/caddy/
        rewrite * /api{path}
        reverse_proxy 127.0.0.1:3456
    }

    # Serve docs as files rsync'ed to the server
    # See trading-strategy/.github/workflows/rsync-docs.yml for more details
    # https://caddy.community/t/caddy-file-server-browse-not-working-when-root-a-specific-directory/15342/3
    redir /docs /docs/
    handle_path /docs* {
        # See docker-compose
        # mapped to /home/docs/html
        root * /docs
        file_server
    }


    # SvelteKit production server from frontend repository
    handle {
        reverse_proxy 127.0.0.1:3000 {
        header_up X-Forwarded-Host {host}
            # Frontend must render the page under 20 seconds
            transport http {
                response_header_timeout 20s
            }
        }
    }

    # Set the default 404 page
    # https://caddyserver.com/docs/caddyfile/directives/handle_errors
    handle_errors {
        respond "{http.error.status_code} {http.error.status_text}"
    }

    # Create a log file.
    # This is mapped to local FS thru docker-compose, so you can read it like
    # tail -f logs/access.log  | jq
    # but it is also consumed by logstash log forwarder
    log {
        format json
        output file /var/log/caddy/access.log
        #output stdout
        # Caddy cannot have two log outputs at the same time
    }

}

3. The problem I’m having:

Caddy connection reverse proxy fails for < 1% requests randomly. I would like to pin point the underlying cause (fault in Caddy, fault in operating system, fault in the upstream web server). This is on a Docker setup using localhost connections, so network issues can be ruled out.

I traced down the request to the upstream server using CloudFlare ray headers. It seems that the upstream web server never sees the request, which is natural because the error is 502 Bad Gateway.

I could not find out how to extract the underlying operating system error reason behind HTTP 502 responses Caddy is serving. Does Caddy have any sort of error logs where underlying error reasons would be logged and how would this be turned on? As the error reason could be related to timeout, sockets, etc.

I do not see anything relevant in the access log JSON or stdout output.

4. Error messages and/or full log output:

The problem is that there is no error messages in Caddy output or logs.

6. Links to relevant resources:

2

That’s quite an old version. Please upgrade to v2.6.2.

Remove these. That’s not needed. And it’s using a single dash for -config which will break when you upgrade to the latest version.

Remove this as well. There’s no reason to override that file.

You won’t need this either.

Impossible to say without logs. You need to show evidence of what happens with those requests.

You can turn on the debug global option, which will show more details in the logs.

But any 502 should be in the logs as an error.

Keep in mind, this only configures access logs. You need to look at Caddy’s process logs, which are written to stdout by default (so, look at your container logs).

1 Like
3

Thank you Francis. The problem is that the HTTP 500 codes get logged in the access log, but it does not tell why this error happened (cannot connect socket, some other reason).

I will try with debug option to see if it will give me a clue.

4

Responses with the 500 status code are probably coming from your upstream app itself, not from Caddy. In that case, it means your app had an error.

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.