Tracing configuration for insecure gRPC OpenTelemetry backend like Jaeger All-In-One

tobias0289 · August 4, 2022, 8:26am

1. Output of `caddy version`:

root@n311:~# docker compose exec caddy caddy version
v2.5.2 h1:eCJdLyEyAGzuQTa5Mh3gETnYWDClo1LjtQm2q9RNZrs=

2. How I run Caddy:

a. System environment:

Ubuntu 20.04.4 LTS
Docker 20.10.17
Docker compose v2.6.1

c. Docker compose file

version: '3.7'

services:
  caddy:
    image: caddy:latest
    ports:
     - "80:80"
    volumes:
       - ./Caddyfile:/etc/caddy/Caddyfile
    environment:
      - OTEL_SERVICE_NAME=caddy-app
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://<jaeger-ip>:4318
      - OTEL_EXPORTER_OTLP_INSECURE=true
      - OTEL_EXPORTER_OTLP_PROTOCOL=http/json

d. My complete Caddy config:

{
}

:80

# Compress responses according to Accept-Encoding headers
encode gzip zstd


tracing {
	span caddy_span
}

reverse_proxy {
	to 10.43.181.223:8888

	lb_policy least_conn
	header_down -x-powered-by
}

3. The problem I’m having:

No spans are visible in the Jaeger UI. Tested multiple configs but couldn’t find a working one. Jaeger is deployed on a VPS using the all-in-one docker image, using the example command:

docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.37

I tried the following configs:

      - OTEL_SERVICE_NAME=caddy-app
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://<jaeger-ip>:4318
      - OTEL_EXPORTER_OTLP_INSECURE=true
      - OTEL_EXPORTER_OTLP_PROTOCOL=http/json

      - OTEL_SERVICE_NAME=caddy-app
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://<jaeger-ip>:4318
      - OTEL_EXPORTER_OTLP_INSECURE=true
      - OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

      - OTEL_SERVICE_NAME=caddy-app
      - OTEL_EXPORTER_OTLP_ENDPOINT=https://<jaeger-ip>:4317
      - OTEL_EXPORTER_OTLP_INSECURE=true
      - OTEL_EXPORTER_OTLP_PROTOCOL=grpc

4. Error messages and/or full log output:

gRPC protocol config yields this error, hinting that the INSECURE parameter is ignored

root-caddy-1  | 2022/08/04 08:08:53 max retry time elapsed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"

HTTP protocol config (using both json and protobuf) yields this error:

root-caddy-1  | 2022/08/04 08:10:05 max retry time elapsed: rpc error: code = Unavailable desc = connection closed before server preface received

5. What I already tried:

Configs posted above.

6. Links to relevant resources:

tobias0289 · August 4, 2022, 3:09pm

caddy source shows that only the otlptracegrpc exporter is implemented, so expecting that only gRPC will work

prometheus had a similar bug that was fixed in this PR

tobias0289 · August 4, 2022, 3:16pm

maybe I can use another instance of Caddy as a gRPC reverse proxy in front of the Jaeger docker container and let Caddy issue an SSL certificate.

francislavoie · August 4, 2022, 6:34pm

I think you need to configure OTEL_EXPORTER_OTLP_TRACES_ENDPOINT as per the docs example:

francislavoie · August 4, 2022, 6:40pm

FYI @andriikushch if you could take a look at this, since you implemented this feature!

tobias0289 · August 5, 2022, 3:29am

tried this (along with the other INSECURE variables), same outcome. I also tried setting the endpoint to a non-existing domain and then I got a DNS resolver error message, so I know that the environmental variables set in docker-compose.yml indeed are visible by Caddy.

My plan is to add a reverse proxy in front of Jaeger for SSL termination and then it should all work. But if the insecure option can be enabled, that’d definitely be easier

andriikushch · August 5, 2022, 3:46pm

Hi @francislavoie , @tobias0289

I will be at my PC on Monday and can take a look.

Best regards,
Andrii

andriikushch · August 8, 2022, 10:56am

Hi @tobias0289,

I tried to run caddy/jaeger with the following config:

jaeger:
docker run -d --name jaeger -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 -e COLLECTOR_OTLP_ENABLED=true -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 -p 16686:16686 -p 4317:4317 -p 4318:4318 -p 14250:14250 -p 14268:14268 -p 14269:14269 -p 9411:9411 jaegertracing/all-in-one:1.37

Caddyfile:

{
}
:80
respond "Hello, world!"
encode gzip zstd
tracing {
	span caddy_span
}

docker-compose.yaml:

version: '3.7'

services:
  caddy:
    image: caddy:latest
    ports:
      - "80:80"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
    environment:
      - OTEL_SERVICE_NAME=caddy-app
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://<jaeger-ip>:4317
      - OTEL_EXPORTER_OTLP_INSECURE=true
      - OTEL_EXPORTER_OTLP_PROTOCOL=grpc
      - GRPC_GO_LOG_VERBOSITY_LEVEL=99
      - GRPC_GO_LOG_SEVERITY_LEVEL=info

After I executed:

 $ curl -D- localhost:80
HTTP/1.1 200 OK
Server: Caddy
Traceparent: 00-415300c2df22e0a9746fef4f5bcd33fb-835b921c7346826f-01
Date: Mon, 08 Aug 2022 10:28:00 GMT
Content-Length: 13

Hello, world!

I was able to see, trace information on UI (http://localhost:16686/search in my case).

I would suggest we try the following:

Could you please double-check the port in your configuration (4317)?
Try to verify connectivity between the caddy container and jaeger. For example execute from caddy container telnet <jaeger-ip> 4317.
Please add

- GRPC_GO_LOG_VERBOSITY_LEVEL=99
- GRPC_GO_LOG_SEVERITY_LEVEL=info

to the caddy’s docker-compose file and try to send a request.

After, please provide a full log from the jaeger and caddy containers.

I hope we will see a reason in the logs