Caddy load balancing does not work

kpinarci · January 17, 2023, 3:08pm

1. Output of `caddy version`: v2.6.1

2. How I run Caddy: docker container.

a. System environment: VM: Linux sles 15, Docker Server Version: 20.10.17-ce

c. Service/unit/compose file:

version: "3.7"
services:
  caddy:
    container_name: caddy
    image: caddy:2.6.1
    restart: always
    build:
     context: .
     dockerfile: Dockerfile
    volumes:
     - "/opt/docker/monitoring/caddy/Caddyfile:/etc/caddy/Caddyfile"
     - "/opt/docker/monitoring/caddy/data/caddy:/data"
     - "/opt/docker/monitoring/caddy/config:/config"
     - "/opt/docker/monitoring/caddy/log/:/var/log/"
    ports:
     - "80:80"
     - "443:443"
    labels:
      logging: "promtail"
      logging_jobname: "containerlogs"
    networks:
     - monitoring

networks:
 monitoring:
    external: true

d. My complete Caddy config:

{                                                                                                                                                                                                                               [43/1095]
  email     ci@example.com
  acme_ca   https://acme.example.com:9001/acme/acme/directory
#  ca_root   /usr/local/share/ca-certificates/ca_bundle.crt
   debug
}

#(logfile) {
#
#log {
#    level info
#    output file /var/log/mimir.log {
#        roll_size 10MiB
#        roll_keep 5
#        roll_keep_for 120h
#        }
#       format json {
#         time_format iso8601
#}
#}
#}

(logconsole) {

log {
        output stdout
        format json {
         time_format iso8601
}
}
}

grafana.example.com {
          reverse_proxy grafana:3000
          }
prometheus.example.com {
          reverse_proxy prometheus:9090
          basicauth /* {
            admin JDJhJDE0JEs3YWdTQm52cllTcnFPU2JVZÖLAJSDFÖASO"WNASByc3JpV0JVbHZmMnkuMkpQT1MzeXJSWnhT
          }
          }
promtail.example.com {
          reverse_proxy promtail:9080
          basicauth /* {
            admin JDJhJDE0JEs3YWdTQm52cllTcnFPU2JVZÖLAJSDFÖASO"WNASByc3JpV0JVbHZmMnkuMkpQT1MzeXJSWnhT
          }
          }
telegraf.example.com {
          reverse_proxy telegraf:9274
          basicauth /* {
            admin JDJhJDE0JEs3YWdTQm52cllTcnFPU2JVZÖLAJSDFÖASO"WNASByc3JpV0JVbHZmMnkuMkpQT1MzeXJSWnhT
          }
          }
loki.example.com {
          reverse_proxy loki:3100
          basicauth /* {
            admin JDJhJDE0JEs3YWdTQm52cllTcnFPU2JVZÖLAJSDFÖASO"WNASByc3JpV0JVbHZmMnkuMkpQT1MzeXJSWnhT
          }
          }
mimir.example.com {
            import logconsole
            reverse_proxy * {
            to mimir-1:8080
            to mimir-2:8080
            to mimir-3:8080

            lb_policy round_robin
            lb_try_duration 1s
            lb_try_interval 250s
            max_fails 1

          }
          }

3. The problem I’m having:

I have replicated Grafana Mimir, as load balancer I have used Caddy. When the Mimir containers are running everything works fine, as soon as one of the Mimir instances is killed, none of the containers are reachable.
I tested this with ngix to isolate the problem and to know if the problem is with the Mimir. With ngix, it works beautifully. I have tested all the “lb_policy”. Without success.

4. Error messages and/or full log output:

2023-01-17T11:15:20+01:00	{"level":"info","ts":"2023-01-17T10:15:20.038Z","logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"172.18.0.1","remote_port":"51046","proto":"HTTP/2.0","method":"POST","host":"mimir.example.com","uri":"/api/v1/push","headers":{"User-Agent":["Prometheus/2.37.4"],"X-Prometheus-Remote-Write-Version":["0.1.0"],"X-Scope-Orgid":["mimir"],"Content-Length":["58949"],"Content-Encoding":["snappy"],"Content-Type":["application/x-protobuf"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"mimir.example.com"}},"user_id":"","duration":0.008772353,"size":0,"status":200,"resp_headers":{"Date":["Tue, 17 Jan 2023 10:15:20 GMT"],"Content-Length":["0"],"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"]}}
2023-01-17T11:15:20+01:00	{"level":"debug","ts":1673950520.038858,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"mimir-1:8080","duration":0.008609509,"request":{"remote_ip":"172.18.0.1","remote_port":"51046","proto":"HTTP/2.0","method":"POST","host":"mimir.example.com","uri":"/api/v1/push","headers":{"Content-Encoding":["snappy"],"X-Prometheus-Remote-Write-Version":["0.1.0"],"X-Forwarded-For":["172.18.0.1"],"X-Forwarded-Proto":["https"],"X-Scope-Orgid":["mimir"],"Content-Type":["application/x-protobuf"],"User-Agent":["Prometheus/2.37.4"],"X-Forwarded-Host":["mimir.example.com"],"Content-Length":["58949"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"mimir.example.com"}},"headers":{"Content-Length":["0"],"Date":["Tue, 17 Jan 2023 10:15:20 GMT"]},"status":200}
2023-01-17T11:15:20+01:00	{"level":"debug","ts":1673950520.0301821,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"mimir-1:8080","total_upstreams":3}
2023-01-17T11:15:15+01:00	{"level":"debug","ts":1673950515.04695,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"mimir-2:8080","duration":0.030221084,"request":{"remote_ip":"172.18.0.1","remote_port":"51046","proto":"HTTP/2.0","method":"POST","host":"mimir.example.com","uri":"/api/v1/push","headers":{"User-Agent":["Prometheus/2.37.4"],"X-Prometheus-Remote-Write-Version":["0.1.0"],"X-Scope-Orgid":["mimir"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["mimir.example.com"],"Content-Type":["application/x-protobuf"],"Content-Encoding":["snappy"],"Content-Length":["19069"],"X-Forwarded-For":["172.18.0.1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"mimir.example.com"}},"error":"dial tcp: lookup mimir-2 on 127.0.0.11:53: server misbehaving"}
2023-01-17T11:15:15+01:00	{"level":"info","ts":"2023-01-17T10:15:15.025Z","logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"172.18.0.1","remote_port":"51046","proto":"HTTP/2.0","method":"POST","host":"mimir.example.com","uri":"/api/v1/push","headers":{"Content-Length":["20064"],"X-Scope-Orgid":["mimir"],"Content-Encoding":["snappy"],"Content-Type":["application/x-protobuf"],"User-Agent":["Prometheus/2.37.4"],"X-Prometheus-Remote-Write-Version":["0.1.0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"mimir.example.com"}},"user_id":"","duration":0.006742751,"size":0,"status":200,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Date":["Tue, 17 Jan 2023 10:15:15 GMT"],"Content-Length":["0"]}}
2023-01-17T11:15:15+01:00	{"level":"debug","ts":1673950515.0251415,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"mimir-3:8080","duration":0.006480216,"request":{"remote_ip":"172.18.0.1","remote_port":"51046","proto":"HTTP/2.0","method":"POST","host":"mimir.example.com","uri":"/api/v1/push","headers":{"X-Prometheus-Remote-Write-Version":["0.1.0"],"X-Scope-Orgid":["mimir"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["mimir.example.com"],"Content-Length":["20064"],"Content-Encoding":["snappy"],"Content-Type":["application/x-protobuf"],"User-Agent":["Prometheus/2.37.4"],"X-Forwarded-For":["172.18.0.1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"mimir.example.com"}},"headers":{"Date":["Tue, 17 Jan 2023 10:15:15 GMT"],"Content-Length":["0"]},"status":200}
2023-01-17T11:15:15+01:00	{"level":"debug","ts":1673950515.0185485,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"mimir-3:8080","total_upstreams":3}
2023-01-17T11:15:15+01:00	{"level":"debug","ts":1673950515.0166483,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"mimir-2:8080","total_upstreams":3}
2023-01-17T11:15:12+01:00	{"level":"info","ts":"2023-01-17T10:15:12.944Z","logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"172.18.0.1","remote_port":"51046","proto":"HTTP/2.0","method":"POST","host":"mimir.example.com","uri":"/api/v1/push","headers":{"Content-Encoding":["snappy"],"Content-Type":["application/x-protobuf"],"User-Agent":["Prometheus/2.37.4"],"X-Prometheus-Remote-Write-Version":["0.1.0"],"X-Scope-Orgid":["mimir"],"Content-Length":["119751"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"mimir.example.com"}},"user_id":"","duration":0.02324268,"size":0,"status":200,"resp_headers":{"Alt-Svc":["h3=\":443\"; ma=2592000"],"Date":["Tue, 17 Jan 2023 10:15:12 GMT"],"Content-Length":["0"],"Server":["Caddy"]}}
2023-01-17T11:15:12+01:00	{"level":"debug","ts":1673950512.944365,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"mimir-1:8080","duration":0.023062001,"request":{"remote_ip":"172.18.0.1","remote_port":"51046","proto":"HTTP/2.0","method":"POST","host":"mimir.example.com","uri":"/api/v1/push","headers":{"X-Forwarded-Proto":["https"],"Content-Type":["application/x-protobuf"],"X-Prometheus-Remote-Write-Version":["0.1.0"],"Content-Length":["119751"],"Content-Encoding":["snappy"],"X-Forwarded-For":["172.18.0.1"],"User-Agent":["Prometheus/2.37.4"],"X-Scope-Orgid":["mimir"],"X-Forwarded-Host":["mimir.example.com"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"mimir.example.com"}},"headers":{"Date":["Tue, 17 Jan 2023 10:15:12 GMT"],"Content-Length":["0"]},"status":200}
2023-01-17T11:15:12+01:00	{"level":"info","ts":"2023-01-17T10:15:12.940Z","logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"172.18.0.1","remote_port":"51046","proto":"HTTP/2.0","method":"POST","host":"mimir.example.com","uri":"/api/v1/push","headers":{"Content-Encoding":["snappy"],"Content-Type":["application/x-protobuf"],"User-Agent":["Prometheus/2.37.4"],"X-Prometheus-Remote-Write-Version":["0.1.0"],"X-Scope-Orgid":["mimir"],"Content-Length":["116491"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"mimir.example.com"}},"user_id":"","duration":0.02442146,"size":0,"status":200,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Date":["Tue, 17 Jan 2023 10:15:12 GMT"],"Content-Length":["0"]}}
2023-01-17T11:15:12+01:00	{"level":"debug","ts":1673950512.940705,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"mimir-3:8080","duration":0.024193038,"request":{"remote_ip":"172.18.0.1","remote_port":"51046","proto":"HTTP/2.0","method":"POST","host":"mimir.example.com","uri":"/api/v1/push","headers":{"Content-Encoding":["snappy"],"X-Scope-Orgid":["mimir"],"Content-Length":["116491"],"X-Forwarded-For":["172.18.0.1"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["mimir.example.com"],"Content-Type":["application/x-protobuf"],"User-Agent":["Prometheus/2.37.4"],"X-Prometheus-Remote-Write-Version":["0.1.0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"mimir.example.com"}},"headers":{"Date":["Tue, 17 Jan 2023 10:15:12 GMT"],"Content-Length":["0"]},"status":200}
2023-01-17T11:15:12+01:00	{"level":"debug","ts":1673950512.9229634,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"mimir-2:8080","duration":0.021805588,"request":{"remote_ip":"172.18.0.1","remote_port":"51046","proto":"HTTP/2.0","method":"POST","host":"mimir.example.com","uri":"/api/v1/push","headers":{"Content-Length":["29350"],"X-Forwarded-For":["172.18.0.1"],"X-Forwarded-Proto":["https"],"Content-Type":["application/x-protobuf"],"Content-Encoding":["snappy"],"X-Scope-Orgid":["mimir"],"X-Forwarded-Host":["mimir.example.com"],"User-Agent":["Prometheus/2.37.4"],"X-Prometheus-Remote-Write-Version":["0.1.0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"mimir.example.com"}},"error":"dial tcp: lookup mimir-2 on 127.0.0.11:53: server misbehaving"}
2023-01-17T11:15:12+01:00	{"level":"debug","ts":1673950512.9228976,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"mimir-2:8080","duration":0.024956551,"request":{"remote_ip":"172.18.0.1","remote_port":"51046","proto":"HTTP/2.0","method":"POST","host":"mimir.example.com","uri":"/api/v1/push","headers":{"Content-Length":["31867"],"Content-Type":["application/x-protobuf"],"X-Forwarded-Host":["mimir.example.com"],"X-Prometheus-Remote-Write-Version":["0.1.0"],"X-Scope-Orgid":["mimir"],"X-Forwarded-For":["172.18.0.1"],"X-Forwarded-Proto":["https"],"Content-Encoding":["snappy"],"User-Agent":["Prometheus/2.37.4"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"mimir.example.com"}},"error":"dial tcp: lookup mimir-2 on 127.0.0.11:53: server misbehaving"}
2023-01-17T11:15:12+01:00	{"level":"debug","ts":1673950512.9212196,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"mimir-1:8080","total_upstreams":3}
2023-01-17T11:15:12+01:00	{"level":"debug","ts":1673950512.9164207,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"mimir-3:8080","total_upstreams":3}
2023-01-17T11:15:12+01:00	{"level":"info","ts":"2023-01-17T10:15:12.912Z","logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"172.18.0.1","remote_port":"51046","proto":"HTTP/2.0","method":"POST","host":"mimir.example.com","uri":"/api/v1/push","headers":{"Content-Type":["application/x-protobuf"],"Content-Length":["37835"],"User-Agent":["Prometheus/2.37.4"],"X-Prometheus-Remote-Write-Version":["0.1.0"],"X-Scope-Orgid":["mimir"],"Content-Encoding":["snappy"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"mimir.example.com"}},"user_id":"","duration":0.011408476,"size":0,"status":200,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Content-Length":["0"],"Date":["Tue, 17 Jan 2023 10:15:12 GMT"]}}
2023-01-17T11:15:12+01:00	{"level":"debug","ts":1673950512.9124403,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"mimir-1:8080","duration":0.011056799,"request":{"remote_ip":"172.18.0.1","remote_port":"51046","proto":"HTTP/2.0","method":"POST","host":"mimir.example.com","uri":"/api/v1/push","headers":{"Content-Type":["application/x-protobuf"],"X-Forwarded-Proto":["https"],"User-Agent":["Prometheus/2.37.4"],"Content-Encoding":["snappy"],"Content-Length":["37835"],"X-Forwarded-For":["172.18.0.1"],"X-Forwarded-Host":["mimir.example.com"],"X-Prometheus-Remote-Write-Version":["0.1.0"],"X-Scope-Orgid":["mimir"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"mimir.example.com"}},"headers":{"Date":["Tue, 17 Jan 2023 10:15:12 GMT"],"Content-Length":["0"]},"status":200}
2023-01-17T11:15:12+01:00	{"level":"info","ts":"2023-01-17T10:15:12.912Z","logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"172.18.0.1","remote_port":"51046","proto":"HTTP/2.0","method":"POST","host":"mimir.example.com","uri":"/api/v1/push","headers":{"User-Agent":["Prometheus/2.37.4"],"X-Prometheus-Remote-Write-Version":["0.1.0"],"X-Scope-Orgid":["mimir"],"Content-Encoding":["snappy"],"Content-Type":["application/x-protobuf"],"Content-Length":["37897"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"mimir.example.com"}},"user_id":"","duration":0.012061571,"size":0,"status":200,"resp_headers":{"Alt-Svc":["h3=\":443\"; ma=2592000"],"Date":["Tue, 17 Jan 2023 10:15:12 GMT"],"Content-Length":["0"],"Server":["Caddy"]}}
2023-01-17T11:15:12+01:00	{"level":"debug","ts":1673950512.911698,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"mimir-3:8080","duration":0.01161726,"request":{"remote_ip":"172.18.0.1","remote_port":"51046","proto":"HTTP/2.0","method":"POST","host":"mimir.example.com","uri":"/api/v1/push","headers":{"X-Scope-Orgid":["mimir"],"Content-Encoding":["snappy"],"X-Forwarded-Host":["mimir.example.com"],"Content-Type":["application/x-protobuf"],"Content-Length":["37897"],"User-Agent":["Prometheus/2.37.4"],"X-Prometheus-Remote-Write-Version":["0.1.0"],"X-Forwarded-For":["172.18.0.1"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"mimir.example.com"}},"headers":{"Date":["Tue, 17 Jan 2023 10:15:12 GMT"],"Content-Length":["0"]},"status":200}
2023-01-17T11:15:12+01:00	{"level":"debug","ts":1673950512.9011023,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"mimir-1:8080","total_upstreams":3}
2023-01-17T11:15:12+01:00	{"level":"debug","ts":1673950512.9011018,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"mimir-2:8080","total_upstreams":3}
2023-01-17T11:15:12+01:00	{"level":"debug","ts":1673950512.8999825,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"mimir-3:8080","total_upstreams":3}
2023-01-17T11:15:12+01:00	{"level":"debug","ts":1673950512.8977985,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"mimir-2:8080","total_upstreams":3}

5. What I already tried:

Tested with nginx load balancer and it works, that’s exactly what I want to do with Caddy.

events {
    worker_connections 1024;
}

http {
    upstream backend {
        server mimir-1:8080 max_fails=1 fail_timeout=1s;
        server mimir-2:8080 max_fails=1 fail_timeout=1s;
        server mimir-3:8080 max_fails=1 fail_timeout=1s backup;
    }

    server {
        listen 9009;
        access_log /dev/null;
        location / {
            proxy_pass http://backend;
        }
    }
}

6. Links to relevant resources:

emilylange · January 17, 2023, 5:00pm

Remove the lb_try_interval 250s or set it to something closer to its default value (250ms)

You are waiting 250 seconds before trying the next upstream if one fails

kpinarci · January 18, 2023, 9:31am

Hi Emily,

thanks for the swift feedback, your hint, that helped me! I still have one problem, if I kill one of the 3 containers, then the Mimir is reachable, but if I kill two containers the GUI of Mimir is reachable, but no data arrives at Grafana Mimir. I get status code 502.

francislavoie · January 18, 2023, 9:35am

You’ll need to show your logs again. It’s impossible to know what’s going on without more evidence.

kpinarci · January 18, 2023, 10:12am

You are absolutely right, attached are the Caddy logs:


2023-01-18 11:07:39

{"level":"error","ts":"2023-01-18T10:07:39.841Z","logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"172.20.1.244","remote_port":"44710","proto":"HTTP/2.0","method":"POST","host":"mimir.example.com","uri":"/api/v1/push","headers":{"Content-Encoding":["snappy"],"Content-Type":["application/x-protobuf"],"User-Agent":["Prometheus/2.36.0"],"X-Prometheus-Remote-Write-Version":["0.1.0"],"X-Scope-Orgid":["mimir"],"Content-Length":["19604"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"mimir.example.com"}},"user_id":"","duration":0.271081433,"size":0,"status":502,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"]}}

2023-01-18 11:07:39

{"level":"error","ts":1674036459.840877,"logger":"http.log.error.log0","msg":"readfrom tcp 172.18.0.3:52374->172.18.0.9:8080: body closed by handler","request":{"remote_ip":"172.20.1.244","remote_port":"44710","proto":"HTTP/2.0","method":"POST","host":"mimir.example.com","uri":"/api/v1/push","headers":{"Content-Encoding":["snappy"],"Content-Type":["application/x-protobuf"],"User-Agent":["Prometheus/2.36.0"],"X-Prometheus-Remote-Write-Version":["0.1.0"],"X-Scope-Orgid":["mimir"],"Content-Length":["19604"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"mimir.example.com"}},"duration":0.271081433,"status":502,"err_id":"4nzufy4r5","err_trace":"reverseproxy.statusError (reverseproxy.go:1271)"}

2023-01-18 11:07:39

{"level":"debug","ts":1674036459.8404462,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"mimir-3:8080","duration":0.001081215,"request":{"remote_ip":"172.20.1.244","remote_port":"44710","proto":"HTTP/2.0","method":"POST","host":"mimir.example.com","uri":"/api/v1/push","headers":{"Content-Type":["application/x-protobuf"],"X-Prometheus-Remote-Write-Version":["0.1.0"],"X-Forwarded-For":["172.20.1.244"],"X-Scope-Orgid":["mimir"],"Content-Length":["19604"],"Content-Encoding":["snappy"],"User-Agent":["Prometheus/2.36.0"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["mimir.example.com"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"mimir.example.com"}},"error":"readfrom tcp 172.18.0.3:52374->172.18.0.9:8080: body closed by handler"}

2023-01-18 11:07:39

{"level":"debug","ts":1674036459.839271,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"mimir-3:8080","total_upstreams":3}

2023-01-18 11:07:39

{"level":"debug","ts":1674036459.5887415,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"mimir-2:8080","duration":0.018775056,"request":{"remote_ip":"172.20.1.244","remote_port":"44710","proto":"HTTP/2.0","method":"POST","host":"mimir.example.com","uri":"/api/v1/push","headers":{"X-Scope-Orgid":["mimir"],"Content-Length":["19604"],"Content-Encoding":["snappy"],"User-Agent":["Prometheus/2.36.0"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["mimir.example.com"],"Content-Type":["application/x-protobuf"],"X-Prometheus-Remote-Write-Version":["0.1.0"],"X-Forwarded-For":["172.20.1.244"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"mimir.example.com"}},"error":"dial tcp: lookup mimir-2 on 127.0.0.11:53: server misbehaving"}

2023-01-18 11:07:39

{"level":"debug","ts":1674036459.5698311,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"mimir-2:8080","total_upstreams":3}

2023-01-18 11:07:38

{"level":"error","ts":1674036458.420768,"logger":"http.log.error.log0","msg":"dial tcp: lookup mimir-1 on 127.0.0.11:53: server misbehaving","request":{"remote_ip":"172.20.1.244","remote_port":"61484","proto":"HTTP/2.0","method":"GET","host":"mimir.example.com","uri":"/static/mimir-icon.png","headers":{"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"],"Accept":["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-Mode":["no-cors"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-Dest":["image"],"Referer":["https://mimir.example.com/memberlist"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"],"Sec-Ch-Ua":["\"Not?A_Brand\";v=\"8\", \"Chromium\";v=\"108\", \"Google Chrome\";v=\"108\""]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"mimir.example.com"}},"duration":1.095027931,"status":502,"err_id":"htarfkeid","err_trace":"reverseproxy.statusError (reverseproxy.go:1271)"}

2023-01-18 11:07:38

{"level":"error","ts":"2023-01-18T10:07:38.420Z","logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"172.20.1.244","remote_port":"61484","proto":"HTTP/2.0","method":"GET","host":"mimir.example.com","uri":"/static/mimir-icon.png","headers":{"Sec-Ch-Ua-Mobile":["?0"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"],"Accept":["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-Mode":["no-cors"],"Sec-Ch-Ua":["\"Not?A_Brand\";v=\"8\", \"Chromium\";v=\"108\", \"Google Chrome\";v=\"108\""],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-Dest":["image"],"Referer":["https://mimir.example.com/memberlist"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"mimir.example.com"}},"user_id":"","duration":1.095027931,"size":0,"status":502,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"]}}

emilylange · January 18, 2023, 7:04pm

Can you share your Caddyfile and logs with debug logging (just like in your opening post) again?

What did you set lb_try_interval to now?
It describes the total amount spent load balancing a request, so it should be less than $lb_try_duration * $retries.

If you don’t feel like playing around with durations, just use lb_retries:

mimir.example.com {
	import logconsole
	reverse_proxy {
		to mimir-1:8080
		to mimir-2:8080
		to mimir-3:8080

		lb_policy round_robin
		lb_retries 2
	}
}

kpinarci · January 20, 2023, 1:39pm

Hello Emily,

with the configuration, i can reach the state of nginx. That helps me. Many thanks for your support.

caddy conf

mimir.example.com {
         import logconsole
         reverse_proxy * {
                 to mimir-1:8080
                 to mimir-2:8080
                 to mimir-3:8080

                 lb_policy round_robin
                 lb_try_duration 1s
                 max_fails 1
         }
}

Nginx conf

events {
    worker_connections 1024;
}

http {
    upstream backend {
        server mimir-1:8080 max_fails=1 fail_timeout=1s;
        server mimir-2:8080 max_fails=1 fail_timeout=1s;
        server mimir-3:8080 max_fails=1 fail_timeout=1s backup;
    }

    server {
        listen 9009;
        access_log /dev/null;
        location / {
            proxy_pass http://backend;
        }
    }
}

system · February 16, 2023, 3:08pm

This topic was automatically closed after 30 days. New replies are no longer allowed.