Tracing module returning response status as 200 instead of 500

[Maaiz_Elahi]

1. The problem I’m having:

I am using tracing module with on demand tls and was trying how the trace will look for error scenario. To create error I am using upstream url in reverse proxy which doesn’t exist.
When hitting a domain, in caddy error logs I am seeing 502 which is expected, however in the opentelemetry trace generated http response status is 200 instead of 502.

2. Error messages and/or full log output:

{"level":"error","ts":1683552526.4414935,"logger":"http.log.error","msg":"dial hosted-service.postman-beta.tech:443: unknown network hosted-service.postman-beta.tech:443","request":{"remote_ip":"172.105.36.145","remote_port":"50625","proto":"HTTP/2.0","method":"GET","host":"v5.dvs.beta.postman.wtf","uri":"/","headers":{"Cache-Control":["max-age=0"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Ch-Ua":["\"Chromium\";v=\"112\", \"Google Chrome\";v=\"112\", \"Not:A-Brand\";v=\"99\""],"Sec-Ch-Ua-Platform":["\"macOS\""],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Dest":["document"],"Cookie":[]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"v5.dvs.beta.postman.wtf"}},"duration":0.000265643,"status":502,"err_id":"d5tp4qz73","err_trace":"reverseproxy.statusError (reverseproxy.go:1299)"}

3. Caddy version:

caddy-2.6.4

4. How I installed and ran Caddy:

a. System environment:

Caddy alpine docker image

b. Command:

command: ["caddy", "run", "--config", "/config/caddy/caddy.json"]

c. Service/unit/compose file:

Used kubernetes deployment

d. My complete Caddy config:

{
  "apps": {
    "http": {
      "http_port": 80,
      "servers": {
        "https": {
          "listen": [
            ":443"
          ],
          "listener_wrappers": [
            {
              "wrapper": "go_proxyproto"
            },
            {
              "wrapper": "tls"
            }
          ],
          "routes": [
            {              
              "handle": [
                {
                  "handler": "tracing",
                  "span": "{http.request.host}"
                },
                {
                  "handler": "reverse_proxy",
                  "transport": {
                    "protocol": "http",
                    "tls": {}
                  },
                  "upstreams": [
                    {
                      "dial": "localhost:3333"
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ],
          "tls_connection_policies": [
            {}
          ]
        },
        "health-check": {
          "listen": [
            ":9999"
          ],
          "routes": [
            {
              "handle": [
                {
                  "body": "OK",
                  "handler": "static_response"
                }
              ],
              "match": [
                {
                  "path": [
                    "/"
                  ]
                }
              ]
            }
          ]
        }
      }
    },
    "tls": {
      "automation": {
        "on_demand": {
          "ask": "http://localhost:9999"
        },
        "policies": [
          {
            "issuers": [
              {
                "module": "acme",
                "account_key": "<private_key_pem>"
              }
            ],
            "on_demand": true,
            "key_type": "rsa4096",
            "disable_ocsp_stapling": true
          }
        ]
      }
    }
  },
  "logging": {
    "logs": {
      "default": {
        "level": "DEBUG",
        "writer": {
          "output": "stdout"
        }
      }
    }
  },
  "storage": {
    "module": "mysql",
    "dsn": ""
  }
}

5. Links to relevant resources:

[francislavoie]

@Maaiz_Elahi

"msg":"dial hosted-service.postman-beta.tech:443: unknown network hosted-service.postman-beta.tech:443"

@Maaiz_Elahi

                  "upstreams": [
                    {
                      "dial": "localhost:3333"
                    }

I’m confused, the error message doesn’t match your config.

@Maaiz_Elahi

however in the opentelemetry trace generated http response status is 200 instead of 502.

I don’t use tracing myself, so I’m not sure how it’s meant to work. I don’t know why it writes 200.

But when an error happens in the middleware pipeline, it quits out of the regular route pipeline and instead runs the error middleware pipeline instead, so I think the tracing handler doesn’t get a chance to grab the real/final status code from the response writer.

Maybe if you add tracing to the error routes it might work?

[Maaiz_Elahi]

@francislavoie In the docs it is mentioned

Note that certain directives, for example reverse_proxy which may write a response with an HTTP status that is classified as an error, will not trigger the error routes.

[francislavoie]

That’s talking about an actual response coming from the upstream.

If the error itself came from the proxy handler, like failing to connect to the upstream, then an error is emitted and the error routes are used.