Add opentelemetry tracing support

Plugins
1

Hi, dear community,

I have seen few discussions about adding tracing support for Caddy Proxy on GitHub, but I haven’t seen an implementation so far.

I plan to make a PR that will add support for opentelemetry(tracing).

While I was working on the implementation, I’ve found this TODO caddy/routes.go at 2392478bd3cf3e2878b3fa0eac8859a6374e0fa8 · caddyserver/caddy · GitHub. This is a good idea because it will allow enabling tracing for every single request in one place. Nevertheless, in my opinion, adding it in the same way as metrics will create a dependency on opentelemetry(or another tracing implementation). In turn, make it “injectable” will require bigger changes to the core functionality.

My idea now is to create an implementation of caddyhttp.MiddlewareHandler. It will enable tracing on particular handlers and handle a basic tracing configuration, which can be overwritten by ENV variables (defined by opentelemetry specification). So it will be a standalone module that can be chained to the regular HTTP handlers.

Are there any objections/ideas regarding this? If not, I will be happy to create a PR in the next few days for further discussion.

Looking forward to hearing feedback.

Thank you so much for your attention and participation.

Best regards,
Andrii

1 Like
2

Yeah, I agree this should be modular/pluggable, rather than hard-coded, if practical.

These handlers would need to be injected in that place you linked to in the code, right? That seems to make sense to me; Caddy will just need to be upgraded to support that.

1 Like
3

Hi @andriikushch :wave:

OTel tracing support has been on my list for ages, ever since I started working on metrics in Caddy :sweat_smile:…

The main reason I’ve put it off is due to the Otel Go module having API-breaking changes fairly often. They’re at 1.0-RC3 now, so things should be settling down, though I’m noting from the changelog that there are still breaking API changes.

If you’d like to start work on a PR to add tracing, that’d be great, however I’d like to hold off merging it until the OTel module is at 1.0 (which shouldn’t be too long now…).

And please add me as a reviewer on the PR - I’ve got lots of past experience instrumenting services with tracing :wink:

2 Likes
4

Hi @hairyhenderson :wave:

Thank you for your feedback.

Absolutely agree with you that it makes sense to wait for the Otel release before the merge. :+1:

As soon as I will have a PR, I will add you as a reviewer.

Best regards,
Andrii

5

Hi @matt,

Thank you for a quick reply. :smiley:

These handlers would need to be injected in that place you linked to in the code, right? That seems to make sense to me; Caddy will just need to be upgraded to support that.

It is not exactly like that. I need to clarify my idea a little bit, I think there was an unclarity from my side. :slight_smile:

This was the initial idea: adding code to the lines mentioned in the previous message. I tried it, but I did not like the result. It does not allow enough flexibility for a configuration, in my opinion.

The current plan is to try another approach. Instead of adding it there, I want to create a standalone plugin/directive that can be applied to each handler. Following this https://caddyserver.com/docs/extending-caddy#complete-example. (Btw, thank you for having such a great example there, it really helps :+1: )

There are three main reasons why I decided to do it this way:

  1. I want to have the possibility to provide/overwrite an independent configuration for each handler (tracer name, resource name, span name, etc.).

  2. There will still be a way to provide a common configuration using environment variables from the open telemetry specification. The user doesn’t have to copy-paste the same configuration again and again.

  3. From a code perspective, I want to keep it decoupled as much as I can.

I want to implement a new module/directive and add it to the list in caddyconfig/httpcaddyfile/directives.go

So it will look like this:

var directiveOrder = []string{
...

// URI manipulation
"rewrite",
"uri",
"try_files",

// middleware handlers; some wrap responses
"opentelemetry",
...

An example of the configuration will be like this:

:2015

handle /orders {
        opentelemetry {
                tracer_name orders
                span_name proxy_order_request
                exporter_traces_protocol grpc
                service_name orders_service
                propagators tracecontext
                exporter_insecure true
                exporter_traces_endpoint myAPMServer
       }
       reverse_proxy 127.0.0.1:8081
}

handle /news {
        opentelemetry {
                tracer_name news
        }
        reverse_proxy 127.0.0.1:8082
}

handle /images {
        opentelemetry {
                tracer_name images
        }
        reverse_proxy 127.0.0.1:8083
}

What do you think about it?

Best regards,
Andrii

6

That’s fine I suppose, but I was initially thinking when I wrote that comment of being able to trace requests as they go through each and every middleware handler so you can see exactly the handler pipeline for each request, rather than manually having granularity at specified endpoints/URIs.

1 Like
7

Dear all,

I have created a PR to add support for OpenTelemetry tracing.

It uses a simplified configuration comparing to what I already mentioned in this thread.

And please add me as a reviewer on the PR - I’ve got lots of past experience instrumenting services with tracing :wink:

@hairyhenderson right now I do not have permission to assign a reviewer or assignees, please just follow the link to the PR.

For the next few days, I will be out of the network coverage and I will be glad to answer all the questions after the 9th of October.

If you will have any questions before this date, please write them in the PR and my colleague will be happy to answer them.

Looking forward to hearing your feedback. :slight_smile:

Thank you in advance!

Best regards,
Andrii

8

This topic was automatically closed after 30 days. New replies are no longer allowed.