Replace Response with invalid server returns 200 when it should return 502

1. The problem I’m having:

Using replace stream with reverse_proxy with an invalid/down server results in Caddy returning 200, when it should return 502.

CURL Request:

curl -vL https://error.getlynx.dev/                                                                                                                                                                                                                   ─╯
*   Trying 172.67.130.7...
* TCP_NODELAY set
* Connected to error.getlynx.dev (172.67.130.7) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=getlynx.dev
*  start date: May 10 09:24:27 2023 GMT
*  expire date: Aug  8 09:24:26 2023 GMT
*  subjectAltName: host "error.getlynx.dev" matched cert's "*.getlynx.dev"
*  issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1P5
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fb10900d600)
> GET / HTTP/2
> Host: error.getlynx.dev
> User-Agent: curl/7.64.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200 
< date: Fri, 12 May 2023 11:48:10 GMT
< content-type: text/plain; charset=utf-8
< content-length: 15
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
< source-server: dipper-v5 | jackbailey.dev
< cf-cache-status: DYNAMIC
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=WwoEyK4l6Eb3QrHnUhMa4glP6Z%2BQ4lrYTghEgUp3clIE%2FnYzYuDGBmDrJQf7ouWwEdOk%2FlcHPc%2BhWrQ6thWmnTB%2BPRMBxCxRH6ks%2FC1icGyidGGD7uf%2F73s7S%2B8Ya7aq7PXZHw%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 7c626e5f0d31dd81-LHR
< 
* Connection #0 to host error.getlynx.dev left intact
502 Bad Gateway* Closing connection 0

502 Bad Gateway is returned by my handle_errors config, which is:

handle_errors {
    respond "{err.status_code} {err.status_text}" {err.status_code}
}

This problem persists without handle_errors set.

2. Error messages and/or full log output:

Caddy Log:

{"level":"debug","ts":1683892090.8402948,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"lynx-demo:3001","duration":0.000989494,"request":{"remote_ip":"REDACTED","remote_port":"18136","proto":"HTTP/2.0","method":"GET","host":"error.getlynx.dev","uri":"/","headers":{"Accept-Encoding":["gzip"],"X-Forwarded-For":["REDACTED, REDACTED"],"Cf-Ray":["7c626e5f0d31dd81-LHR"],"X-Forwarded-Proto":["https"],"User-Agent":["curl/7.64.1"],"Accept":["*/*"],"Cdn-Loop":["cloudflare"],"Cf-Connecting-Ip":["REDACTED"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Ipcountry":["GB"],"X-Forwarded-Host":["error.getlynx.dev"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"error.getlynx.dev"}},"error":"dial tcp 172.18.0.19:3001: connect: connection refused"}
{"level":"debug","ts":1683892090.8406131,"logger":"http.log.error.log0","msg":"dial tcp 172.18.0.19:3001: connect: connection refused","request":{"remote_ip":"REDACTED","remote_port":"18136","proto":"HTTP/2.0","method":"GET","host":"error.getlynx.dev","uri":"/","headers":{"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"User-Agent":["curl/7.64.1"],"Accept":["*/*"],"Cf-Ipcountry":["GB"],"Cdn-Loop":["cloudflare"],"X-Forwarded-For":["REDACTED"],"Cf-Ray":["7c626e5f0d31dd81-LHR"],"Accept-Encoding":["gzip"],"Cf-Connecting-Ip":["REDACTED"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"error.getlynx.dev"}},"duration":0.00143785,"status":502,"err_id":"5zdk4qxhd","err_trace":"reverseproxy.statusError (reverseproxy.go:1299)"}

3. Caddy version:

v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=

4. How I installed and ran Caddy:

Docker via Dockerfile using xcaddy.

FROM caddy:2.6-builder AS builder

RUN xcaddy build \
  --with github.com/caddyserver/replace-response \
  --with github.com/caddy-dns/cloudflare \
  --with github.com/WeidiDeng/caddy-cloudflare-ip \
  --with github.com/git001/caddyv2-upload@64e3be6c858a53f5b9bd0fcd53941446a16cccf2

FROM caddy:2.6

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

a. System environment:

Debian 5.10, Docker.

b. Command:

n/a

c. Service/unit/compose file:

version: "3.7"
services:
  caddy:
    container_name: caddy
    build: .
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./Caddy/data:/data
      - ./Caddy/config:/config
      - ./sites:/sites
    networks:
      - caddy

d. My complete Caddy config:

{
        debug
        email EMAIL

        order replace after encode

        servers {
                trusted_proxies cloudflare {
                        interval 12h
                        timeout 15s
                }
        }
}

(default) {
        header Source-Server "dipper-v5 | jackbailey.dev"

        handle_errors {
                import handle_error
        }
}

(handle_error) {
        respond "{err.status_code} {err.status_text}" {err.status_code}
}

(analytics) {
        replace stream {
                "</head>" "<script async data-website-id=\"{args.0}\" src=\"https://analytics.jackbailey.dev/script.js\"></script></head>"
        }
}

(cf_cert) {
        tls {
                dns cloudflare {$CF_TOKEN}
        }
}

*.getlynx.dev,
getlynx.dev {
        import cf_cert
        import default

        @error host error.getlynx.dev
        handle @error {
                import analytics ffbd53e8-4f16-429a-a4c4-b5e9c37a75de
                reverse_proxy lynx-demo:3001
        }

        handle {
                respond "Invalid domain."
        }
}

5. Links to relevant resources:

replace-response

Okay I think this is a bug with the replace plugin, actually, specifically when stream mode is used. I was able to replicate the behaviour with a simplified config.

The bug is really subtle. I think I figured it out, but I’m not sure my fix is correct. I opened a PR:

1 Like

Okay, thank you for working out at least the start of a fix :slight_smile: .

Good to know I wasn’t going crazy

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.