Can only connect to host if I put reverse_proxy

Help
1

1. The problem I’m having:

So, I want to use caddy-l4 plugin for all of my service. But strangely, I can connect one of my service (keycloak) but not the other (auth and grafana).

2. Error messages and/or full log output:

There’s no error message, only blank white page:

image
image746×612 7.37 KB

3. Caddy version:

v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy:

a. System environment:

Docker running on docker-desktop in windows 11

b. Command:

caddy docker-proxy

c. Service/unit/compose file:

x-env-files: &env_files
  - .env.dev
  - .env.shared
  - path: .env.local
    required: false

x-logging: &logging
  driver: json-file
  options:
    # Used by promtail to produce labels for Loki
    tag: "{{.ImageName}}|{{.Name}}|{{.ImageFullID}}|{{.FullID}}"

services:
  caddy:
    build:
      dockerfile: ./Dockerfile.caddy
      context: .
    container_name: r4gmi-caddy
    restart: unless-stopped
    logging: *logging
    command: ["caddy", "docker-proxy"]
    ports:
      - "80:80"
      - "443:443"
      - "5432:5432"
      - "6379:6379"
    volumes:
      # - ./caddy/dev.Caddyfile:/etc/caddy/Caddyfile
      - /var/run/docker.sock:/var/run/docker.sock
      - caddy-data:/data
      - caddy-config:/config
    env_file: *env_files
    networks:
      - r4gmi_net
    labels:
      caddy.1_debug:
      caddy.2_log.1_format: "json"
      caddy.2_log.2_level: "DEBUG"
      caddy.2_log.3_output: "stderr"
      caddy.2_order: "rate_limit before basicauth"
      caddy.2_email: "admin@r4gmi.localhost"

  auth:
    build:
      dockerfile: ./../../docker/Dockerfile.auth-dev
      context: ../backend/auth
    container_name: r4gmi-auth
    develop:
      watch:
        - action: sync
          path: ./../backend/auth/frontend
          target: /app/frontend
          ignore:
            - ./../backend/auth/frontend/node_modules/
        - action: sync
          path: ./../backend/auth/frontend/package.json
          target: /app/frontend
        - action: sync
          path: ./../backend/auth/backend
          target: /app/backend
        - action: sync
          path: ./../backend/auth/backend/Cargo.toml
          target: /app/backend
    restart: 'no'
    depends_on:
      - db
    env_file: *env_files
    networks:
      - r4gmi_net
    labels:
      caddy.layer4.:443.@auth: "tls sni auth.r4gmi.localhost"
      caddy.layer4.:443.route: "@auth"
      caddy.layer4.:443.route.1_tls:
      caddy.layer4.:443.route.2_proxy.upstream: "{{upstreams 8080}}"
      caddy_1: auth.r4gmi.localhost
      caddy_1.tls: internal
      caddy_1.reverse_proxy: "{{upstreams 8080}}"

  db:
    image: postgres:17.2-bullseye
    container_name: r4gmi-db
    restart: always
    volumes:
      - ./volumes/db/data:/var/lib/postgresql/data 700:Z
      - ./volumes/db/r4gmi/migrations:/docker-entrypoint-initdb.d/r4gmi:rw
    env_file: *env_files
    networks:
      - r4gmi_net
    labels:
      caddy.layer4.:443.@db: "tls sni db.r4gmi.localhost"
      caddy.layer4.:443.route: "@db"
      caddy.layer4.:443.route.1_tls:
      caddy.layer4.:443.route.2_proxy.upstream: "r4gmi-db:5432"
      caddy_2: db.r4gmi.localhost
      caddy_2.tls: internal

  redis:
    image: redis:7.4.1-bookworm
    container_name: r4gmi-redis
    restart: always
    env_file: *env_files
    networks:
      - r4gmi_net
    labels:
      caddy.layer4.:443.@redis: "tls sni redis.r4gmi.localhost"
      caddy.layer4.:443.route: "@redis"
      caddy.layer4.:443.route.1_tls:
      caddy.layer4.:443.route.2_proxy.upstream: "r4gmi-redis:6379"
      caddy_3: redis.r4gmi.localhost
      caddy_3.tls: internal

  prometheus:
    image: prom/prometheus:v2.55.1
    container_name: r4gmi-prometheus
    volumes:
      - ./prometheus/:/etc/prometheus/ 700:Z
      - prometheus-data:/prometheus
    env_file: *env_files
    networks:
      - r4gmi_net
    labels:
      caddy.layer4.:443.@prometheus: "tls sni prometheus.r4gmi.localhost"
      caddy.layer4.:443.route: "@prometheus"
      caddy.layer4.:443.route.1_tls:
      caddy.layer4.:443.route.2_proxy.upstream: "r4gmi-prometheus:9090"
      caddy_4: prometheus.r4gmi.localhost
      caddy_4.tls: internal


  node_exporter:
    image: prom/node-exporter:v1.8.2
    container_name: r4gmi-node-exporter
    command:
      - '--path.rootfs=/host --cap-add=SYS_TIME'
    network_mode: host
    pid: host
    restart: unless-stopped
    volumes:
      - './volumes/node_exporter:/host:ro,rslave'
    env_file: *env_files
    
  grafana:
    image: grafana/grafana:10.4.12
    container_name: r4gmi-grafana
    volumes:
      - grafana-data:/var/lib/grafana
      # - ./grafana/provisioning:/etc/grafana/provisioning
      # - ./grafana/dashboards:/etc/grafana/dashboards
    env_file: *env_files
    networks:
      - r4gmi_net
    labels:
      caddy.layer4.:443.@grafana: "tls sni grafana.r4gmi.localhost"
      caddy.layer4.:443.route: "@grafana"
      caddy.layer4.:443.route.1_tls:
      caddy.layer4.:443.route.2_proxy.upstream: "r4gmi-grafana:3000"
      caddy_5: grafana.r4gmi.localhost
      caddy_5.tls: internal

  keycloak:
    build:
      dockerfile: ./Dockerfile.keycloak
      context: .
    container_name: r4gmi-keycloak
    command: start-dev
    restart: unless-stopped
    env_file: *env_files
    networks:
      - r4gmi_net
    labels:
      caddy.layer4.:443.@keycloak: "tls sni keycloak.r4gmi.localhost"
      caddy.layer4.:443.route: "@keycloak"
      caddy.layer4.:443.route.1_tls:
      caddy.layer4.:443.route.2_proxy.upstream: "r4gmi-keycloak:8080"
      caddy_6: keycloak.r4gmi.localhost
      caddy_6.tls: internal

  questdb:
    image: questdb/questdb:6.6.1
    hostname: questdb
    container_name: r4gmi-questdb
    env_file: *env_files
    networks:
      - r4gmi_net
    labels:
      caddy.layer4.:443.@questdb: "tls sni questdb.r4gmi.localhost"
      caddy.layer4.:443.route: "@questdb"
      caddy.layer4.:443.route.1_tls:
      caddy.layer4.:443.route.2_proxy.upstream: "r4gmi-questdb:9000"
      caddy_7: questdb.r4gmi.localhost
      caddy_7.tls: internal

volumes:
  grafana-data:
  caddy-data:
  caddy-config:
  prometheus-data:
  
networks:
  r4gmi_net:
    name: r4gmi_net
    external: true

d. My complete Caddy config:

{
        debug
        email admin<at>r4gmi.localhost
        log {
                format json
                level DEBUG
                output stderr
        }
        order rate_limit before basicauth
        layer4 {
                :443 {
                        @auth tls sni auth.r4gmi.localhost
                        @db tls sni db.r4gmi.localhost
                        @grafana tls sni grafana.r4gmi.localhost
                        @keycloak tls sni keycloak.r4gmi.localhost
                        @prometheus tls sni prometheus.r4gmi.localhost
                        @questdb tls sni questdb.r4gmi.localhost
                        @redis tls sni redis.r4gmi.localhost
                        route @auth {
                                tls
                                proxy {
                                        upstream 172.18.0.9:8080
                                }
                        }
                        route @db {
                                tls
                                proxy {
                                        upstream r4gmi-db:5432
                                }
                        }
                        route @grafana {
                                tls
                                proxy {
                                        upstream r4gmi-grafana:3000
                                }
                        }
                        route @keycloak {
                                tls
                                proxy {
                                        upstream r4gmi-keycloak:8080
                                }
                        }
                        route @prometheus {
                                tls
                                proxy {
                                        upstream r4gmi-prometheus:9090
                                }
                        }
                        route @questdb {
                                tls
                                proxy {
                                        upstream r4gmi-questdb:9000
                                }
                        }
                        route @redis {
                                tls
                                proxy {
                                        upstream r4gmi-redis:6379
                                }
                        }
                }
        }
}
auth.r4gmi.localhost {
        reverse_proxy 172.18.0.9:8080
        tls internal
}
db.r4gmi.localhost {
        tls internal
}
grafana.r4gmi.localhost {
        tls internal
}
keycloak.r4gmi.localhost {
        tls internal
}
prometheus.r4gmi.localhost {
        tls internal
}
questdb.r4gmi.localhost {
        tls internal
}
redis.r4gmi.localhost {
        tls internal
}
2

You haven’t shared any logs, but I’m willing to bet your requests are not going to the layer4 config. You have both HTTP and layer4 app listening on 443. How should Caddy know which one handles the request? You’re seeing the http app of Caddy handling the requests instead of layer4, and that’s why you see it works when you add reverse_proxy and blank page without.

Just have the http sites listen on a different port than 443, so 443 remains free for the layer4. So, for example, change to this:

-grafana.r4gmi.localhost {
+grafana.r4gmi.localhost:9443 {
        tls internal
}
3

This is what I thought too, however keycloak works fine.

I use tls internal because it’s development env, so I want caddy to generate the root.crt (if i’m not mistaken, layer4 didn’t generate the cert).

I changed it to 9443 like your suggestion, but still white blank page and keycloak getting white blank page too. Once I remove the 9443, I can access keycloak normally.

here’s the log when caddy start:

{"level":"debug","ts":1735726722.1635122,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
{"level":"info","ts":1735726722.1635149,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1735726722.163516,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["grafana.r4gmi.localhost","questdb.r4gmi.localhost","redis.r4gmi.localhost","auth.r4gmi.localhost","db.r4gmi.localhost","prometheus.r4gmi.localhost","keycloak.r4gmi.localhost"]}
{"level":"warn","ts":1735726722.163684,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [grafana.r4gmi.localhost]: no OCSP server specified in certificate","identifiers":["grafana.r4gmi.localhost"]}
{"level":"debug","ts":1735726722.1636927,"logger":"tls.cache","msg":"added certificate to cache","subjects":["grafana.r4gmi.localhost"],"expiration":1735755004,"managed":true,"issuer_key":"local","hash":"768e0f62aae7d4d29c892bb36cec60000f1a939fb4795f33fba83bfac93f4ac6","cache_size":1,"cache_capacity":10000}
{"level":"debug","ts":1735726722.1637058,"logger":"events","msg":"event","name":"cached_managed_cert","id":"c268a7f4-e6c4-4f3d-a3f3-9df9349d593a","origin":"tls","data":{"sans":["grafana.r4gmi.localhost"]}}
{"level":"warn","ts":1735726722.1637785,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [questdb.r4gmi.localhost]: no OCSP server specified in certificate","identifiers":["questdb.r4gmi.localhost"]}
{"level":"debug","ts":1735726722.163781,"logger":"tls.cache","msg":"added certificate to cache","subjects":["questdb.r4gmi.localhost"],"expiration":1735755004,"managed":true,"issuer_key":"local","hash":"6602646bab457030384bdf40f8b1053c7bd1c66210a87c111d429e56817e36d6","cache_size":2,"cache_capacity":10000}
{"level":"debug","ts":1735726722.163785,"logger":"events","msg":"event","name":"cached_managed_cert","id":"502757cc-dab2-4157-a715-41bb70c5f964","origin":"tls","data":{"sans":["questdb.r4gmi.localhost"]}}
{"level":"warn","ts":1735726722.1638613,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [redis.r4gmi.localhost]: no OCSP server specified in certificate","identifiers":["redis.r4gmi.localhost"]}
{"level":"debug","ts":1735726722.1638644,"logger":"tls.cache","msg":"added certificate to cache","subjects":["redis.r4gmi.localhost"],"expiration":1735755004,"managed":true,"issuer_key":"local","hash":"e79cb9cb6b530ec356be70f963007c4b7d1e4e977b62a12c70cbf45ee0bb1279","cache_size":3,"cache_capacity":10000}
{"level":"debug","ts":1735726722.1638687,"logger":"events","msg":"event","name":"cached_managed_cert","id":"3cbf818f-22fb-4c72-a7ae-46839254e659","origin":"tls","data":{"sans":["redis.r4gmi.localhost"]}}
{"level":"warn","ts":1735726722.1639323,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [auth.r4gmi.localhost]: no OCSP server specified in certificate","identifiers":["auth.r4gmi.localhost"]}
{"level":"debug","ts":1735726722.1639345,"logger":"tls.cache","msg":"added certificate to cache","subjects":["auth.r4gmi.localhost"],"expiration":1735755004,"managed":true,"issuer_key":"local","hash":"af477918d4f8e93059a0e06f2bbbcc10784816ea410a4b25c8e8c7c797c8f126","cache_size":4,"cache_capacity":10000}
{"level":"debug","ts":1735726722.1639378,"logger":"events","msg":"event","name":"cached_managed_cert","id":"d86f30be-2b71-40c3-beaa-ee3fada8c2a7","origin":"tls","data":{"sans":["auth.r4gmi.localhost"]}}
{"level":"warn","ts":1735726722.163993,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [db.r4gmi.localhost]: no OCSP server specified in certificate","identifiers":["db.r4gmi.localhost"]}
{"level":"debug","ts":1735726722.1639943,"logger":"tls.cache","msg":"added certificate to cache","subjects":["db.r4gmi.localhost"],"expiration":1735755004,"managed":true,"issuer_key":"local","hash":"24fc00d91a8e268d81550e05ddd2656afeb55011868c740b0d2f33f3bec63d8b","cache_size":5,"cache_capacity":10000}
{"level":"debug","ts":1735726722.163999,"logger":"events","msg":"event","name":"cached_managed_cert","id":"43bccad1-86dd-4f65-849e-3edc1f601514","origin":"tls","data":{"sans":["db.r4gmi.localhost"]}}
{"level":"warn","ts":1735726722.1640604,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [prometheus.r4gmi.localhost]: no OCSP server specified in certificate","identifiers":["prometheus.r4gmi.localhost"]}
{"level":"debug","ts":1735726722.1640627,"logger":"tls.cache","msg":"added certificate to cache","subjects":["prometheus.r4gmi.localhost"],"expiration":1735755004,"managed":true,"issuer_key":"local","hash":"8d1173b55a29008075dd24e58f387c6b3d4df939011de0fefaed85c44168c299","cache_size":6,"cache_capacity":10000}
{"level":"debug","ts":1735726722.164067,"logger":"events","msg":"event","name":"cached_managed_cert","id":"54d7c6d2-95c0-4729-ad9e-9eb8c822dcc7","origin":"tls","data":{"sans":["prometheus.r4gmi.localhost"]}}
{"level":"warn","ts":1735726722.1641266,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [keycloak.r4gmi.localhost]: no OCSP server specified in certificate","identifiers":["keycloak.r4gmi.localhost"]}
{"level":"debug","ts":1735726722.1641285,"logger":"tls.cache","msg":"added certificate to cache","subjects":["keycloak.r4gmi.localhost"],"expiration":1735755004,"managed":true,"issuer_key":"local","hash":"d5b7dbe98e1ca9f2925614ed6d4474ba82f3cd46dc70c85dfe2d0d549910149e","cache_size":7,"cache_capacity":10000}
{"level":"debug","ts":1735726722.164132,"logger":"events","msg":"event","name":"cached_managed_cert","id":"baa454ef-e345-452e-9f2a-1d1515aa6602","origin":"tls","data":{"sans":["keycloak.r4gmi.localhost"]}}
{"level":"debug","ts":1735726722.1641526,"logger":"layer4","msg":"listening","address":"tcp/[::]:443"}
{"level":"info","ts":1735726722.164192,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1735726722.1641943,"logger":"admin.api","msg":"load complete"}
{"level":"info","ts":1735726722.1643808,"logger":"docker-proxy","msg":"Successfully configured","server":"localhost"}
{"level":"info","ts":1735726722.1687245,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"d9ecdc59-25d8-4969-aa2b-905db7dc3663","try_again":1735813122.1687229,"try_again_in":86399.999999761}
{"level":"info","ts":1735726722.1688988,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1735726722.1775548,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}

and here’s when I access grafana.r4gmi.localhost:

{"level":"debug","ts":1735726814.3259068,"logger":"layer4","msg":"prefetched","remote":"172.18.0.1:58538","bytes":1949}
{"level":"debug","ts":1735726814.3259304,"logger":"layer4","msg":"matching","remote":"172.18.0.1:58538","matcher":"layer4.matchers.tls","matched":false}
{"level":"debug","ts":1735726814.3259342,"logger":"layer4","msg":"matching","remote":"172.18.0.1:58538","matcher":"layer4.matchers.tls","matched":false}
{"level":"debug","ts":1735726814.3259385,"logger":"layer4.matchers.tls","msg":"matched","remote":"172.18.0.1:58538","server_name":"grafana.r4gmi.localhost"}
{"level":"debug","ts":1735726814.32594,"logger":"layer4","msg":"matching","remote":"172.18.0.1:58538","matcher":"layer4.matchers.tls","matched":true}
{"level":"debug","ts":1735726814.3260877,"logger":"events","msg":"event","name":"tls_get_certificate","id":"51469df0-0106-4618-8874-05f561792481","origin":"tls","data":{"client_hello":{"CipherSuites":[6682,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"grafana.r4gmi.localhost","SupportedCurves":[2570,4588,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[39578,772,771],"RemoteAddr":{"IP":"172.18.0.1","Port":58538,"Zone":""},"LocalAddr":{"IP":"172.18.0.5","Port":443,"Zone":""}}}}
{"level":"debug","ts":1735726814.3260975,"logger":"tls.handshake","msg":"choosing certificate","identifier":"grafana.r4gmi.localhost","num_choices":1}
{"level":"debug","ts":1735726814.3261063,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"grafana.r4gmi.localhost","subjects":["grafana.r4gmi.localhost"],"managed":true,"issuer_key":"local","hash":"768e0f62aae7d4d29c892bb36cec60000f1a939fb4795f33fba83bfac93f4ac6"}
{"level":"debug","ts":1735726814.326111,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"172.18.0.1","remote_port":"58538","subjects":["grafana.r4gmi.localhost"],"managed":true,"expiration":1735755004,"hash":"768e0f62aae7d4d29c892bb36cec60000f1a939fb4795f33fba83bfac93f4ac6"}
{"level":"debug","ts":1735726814.3268476,"logger":"layer4.handlers.tls","msg":"terminated TLS","remote":"172.18.0.1:58538","server_name":"grafana.r4gmi.localhost"}
{"level":"debug","ts":1735726814.3272178,"logger":"layer4.handlers.proxy","msg":"dial upstream","remote":"172.18.0.1:58538","upstream":"r4gmi-grafana:3000"}
{"level":"error","ts":1735726814.3275485,"logger":"layer4.handlers.proxy","msg":"upstream connection","local_address":"172.18.0.5:43762","remote_address":"172.18.0.3:3000","error":"writeto tcp 172.18.0.5:43762->172.18.0.3:3000: read tcp 172.18.0.5:43762->172.18.0.3:3000: read: connection reset by peer"}
{"level":"debug","ts":1735726814.328568,"logger":"layer4","msg":"connection stats","remote":"172.18.0.1:58538","read":2764,"written":1773,"duration":0.002824257}
{"level":"debug","ts":1735726819.373327,"logger":"layer4","msg":"matching","remote":"172.18.0.1:58540","error":"consumed all prefetched bytes","matcher":"layer4.matchers.tls","matched":false}
{"level":"debug","ts":1735726819.373868,"logger":"layer4","msg":"prefetched","remote":"172.18.0.1:58540","bytes":1885}
{"level":"debug","ts":1735726819.3738997,"logger":"layer4","msg":"matching","remote":"172.18.0.1:58540","matcher":"layer4.matchers.tls","matched":false}
{"level":"debug","ts":1735726819.3739069,"logger":"layer4","msg":"matching","remote":"172.18.0.1:58540","matcher":"layer4.matchers.tls","matched":false}
{"level":"debug","ts":1735726819.37391,"logger":"layer4.matchers.tls","msg":"matched","remote":"172.18.0.1:58540","server_name":"grafana.r4gmi.localhost"}
{"level":"debug","ts":1735726819.3739114,"logger":"layer4","msg":"matching","remote":"172.18.0.1:58540","matcher":"layer4.matchers.tls","matched":true}
{"level":"debug","ts":1735726819.37405,"logger":"events","msg":"event","name":"tls_get_certificate","id":"ab944b5d-72af-47c1-92e8-551635083e6d","origin":"tls","data":{"client_hello":{"CipherSuites":[31354,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"grafana.r4gmi.localhost","SupportedCurves":[56026,4588,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[31354,772,771],"RemoteAddr":{"IP":"172.18.0.1","Port":58540,"Zone":""},"LocalAddr":{"IP":"172.18.0.5","Port":443,"Zone":""}}}}
{"level":"debug","ts":1735726819.3740647,"logger":"tls.handshake","msg":"choosing certificate","identifier":"grafana.r4gmi.localhost","num_choices":1}
{"level":"debug","ts":1735726819.3740733,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"grafana.r4gmi.localhost","subjects":["grafana.r4gmi.localhost"],"managed":true,"issuer_key":"local","hash":"768e0f62aae7d4d29c892bb36cec60000f1a939fb4795f33fba83bfac93f4ac6"}
{"level":"debug","ts":1735726819.3740892,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"172.18.0.1","remote_port":"58540","subjects":["grafana.r4gmi.localhost"],"managed":true,"expiration":1735755004,"hash":"768e0f62aae7d4d29c892bb36cec60000f1a939fb4795f33fba83bfac93f4ac6"}
{"level":"debug","ts":1735726819.378721,"logger":"layer4.handlers.tls","msg":"terminated TLS","remote":"172.18.0.1:58540","server_name":"grafana.r4gmi.localhost"}
{"level":"debug","ts":1735726819.3790934,"logger":"layer4.handlers.proxy","msg":"dial upstream","remote":"172.18.0.1:58540","upstream":"r4gmi-grafana:3000"}
{"level":"debug","ts":1735726819.3805325,"logger":"layer4","msg":"connection stats","remote":"172.18.0.1:58540","read":2700,"written":1771,"duration":0.007213983}
{"level":"debug","ts":1735726829.3276258,"logger":"events","msg":"event","name":"tls_get_certificate","id":"02de6a50-1d57-400d-b2a4-e9aa7b3f10be","origin":"tls","data":{"client_hello":{"CipherSuites":[10794,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"grafana.r4gmi.localhost","SupportedCurves":[27242,4588,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[27242,772,771],"RemoteAddr":{"IP":"172.18.0.1","Port":64044,"Zone":""},"LocalAddr":{"IP":"172.18.0.5","Port":443,"Zone":""}}}}
{"level":"debug","ts":1735726829.327654,"logger":"tls.handshake","msg":"choosing certificate","identifier":"grafana.r4gmi.localhost","num_choices":1}
{"level":"debug","ts":1735726829.327664,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"grafana.r4gmi.localhost","subjects":["grafana.r4gmi.localhost"],"managed":true,"issuer_key":"local","hash":"768e0f62aae7d4d29c892bb36cec60000f1a939fb4795f33fba83bfac93f4ac6"}
{"level":"debug","ts":1735726829.3276668,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"172.18.0.1","remote_port":"64044","subjects":["grafana.r4gmi.localhost"],"managed":true,"expiration":1735755004,"hash":"768e0f62aae7d4d29c892bb36cec60000f1a939fb4795f33fba83bfac93f4ac6"}
4

Are you use you’ve changed it? The log says otherwise

5

I think I paste the wrong log, here’s the correct one:

{"level":"info","ts":1735736948.2761736,"logger":"docker-proxy","msg":"Sending configuration to","server":"localhost"}
{"level":"info","ts":1735736948.2765903,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"43410","headers":{"Accept-Encoding":["gzip"],"Content-Length":["2197"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
{"level":"info","ts":1735736948.2769854,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//[::1]:2019","//127.0.0.1:2019","//localhost:2019"]}
{"level":"info","ts":1735736948.2773588,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1735736948.2774365,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1735736948.2774696,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv1"}
{"level":"debug","ts":1735736948.277521,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{"subjects":["prometheus.r4gmi.localhost","keycloak.r4gmi.localhost","questdb.r4gmi.localhost","redis.r4gmi.localhost","auth.r4gmi.localhost","db.r4gmi.localhost","grafana.r4gmi.localhost"]},{}]}},"http":{"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{},{}]},"srv0":{"listen":[":443"],"routes":[{"terminal":true},{"terminal":true},{"terminal":true},{"terminal":true},{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"172.18.0.9:8080"}]}]}]}],"terminal":true},{"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}},"srv1":{"listen":[":9443"],"routes":[{"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}}
{"level":"warn","ts":1735736948.2777567,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":1735736948.278177,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000a0b180"}
{"level":"info","ts":1735736948.2823908,"msg":"warning: \"certutil\" is not available, install \"certutil\" with \"apt install libnss3-tools\" or \"yum install nss-tools\" and try again"}
{"level":"info","ts":1735736948.2824197,"msg":"define JAVA_HOME environment variable to use the Java trust"}
{"level":"info","ts":1735736948.3423808,"msg":"certificate installed properly in linux trusts"}
{"level":"info","ts":1735736948.3425405,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1735736948.3425877,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
{"level":"debug","ts":1735736948.3426902,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
{"level":"info","ts":1735736948.3427126,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1735736948.3427315,"logger":"http","msg":"enabling HTTP/3 listener","addr":":9443"}
{"level":"debug","ts":1735736948.3427632,"logger":"http","msg":"starting server loop","address":"[::]:9443","tls":true,"http3":true}
{"level":"info","ts":1735736948.3427663,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
{"level":"debug","ts":1735736948.3427737,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
{"level":"info","ts":1735736948.342775,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1735736948.3427763,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["questdb.r4gmi.localhost","redis.r4gmi.localhost","auth.r4gmi.localhost","db.r4gmi.localhost","grafana.r4gmi.localhost","prometheus.r4gmi.localhost","keycloak.r4gmi.localhost"]}
{"level":"warn","ts":1735736948.34293,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [questdb.r4gmi.localhost]: no OCSP server specified in certificate","identifiers":["questdb.r4gmi.localhost"]}
{"level":"debug","ts":1735736948.3429492,"logger":"tls.cache","msg":"added certificate to cache","subjects":["questdb.r4gmi.localhost"],"expiration":1735755004,"managed":true,"issuer_key":"local","hash":"6602646bab457030384bdf40f8b1053c7bd1c66210a87c111d429e56817e36d6","cache_size":1,"cache_capacity":10000}
{"level":"debug","ts":1735736948.3429656,"logger":"events","msg":"event","name":"cached_managed_cert","id":"7e3e2581-2fec-4a2d-81c1-f633562b6c02","origin":"tls","data":{"sans":["questdb.r4gmi.localhost"]}}
{"level":"warn","ts":1735736948.343041,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [redis.r4gmi.localhost]: no OCSP server specified in certificate","identifiers":["redis.r4gmi.localhost"]}
{"level":"debug","ts":1735736948.3430524,"logger":"tls.cache","msg":"added certificate to cache","subjects":["redis.r4gmi.localhost"],"expiration":1735755004,"managed":true,"issuer_key":"local","hash":"e79cb9cb6b530ec356be70f963007c4b7d1e4e977b62a12c70cbf45ee0bb1279","cache_size":2,"cache_capacity":10000}
{"level":"debug","ts":1735736948.3430583,"logger":"events","msg":"event","name":"cached_managed_cert","id":"2c02a0a9-f16a-40f2-822f-09d6740b2151","origin":"tls","data":{"sans":["redis.r4gmi.localhost"]}}
{"level":"warn","ts":1735736948.3433316,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [auth.r4gmi.localhost]: no OCSP server specified in certificate","identifiers":["auth.r4gmi.localhost"]}
{"level":"debug","ts":1735736948.3433735,"logger":"tls.cache","msg":"added certificate to cache","subjects":["auth.r4gmi.localhost"],"expiration":1735755004,"managed":true,"issuer_key":"local","hash":"af477918d4f8e93059a0e06f2bbbcc10784816ea410a4b25c8e8c7c797c8f126","cache_size":3,"cache_capacity":10000}
{"level":"debug","ts":1735736948.3434105,"logger":"events","msg":"event","name":"cached_managed_cert","id":"7c6dcf04-2678-4d22-8dde-9370a042c359","origin":"tls","data":{"sans":["auth.r4gmi.localhost"]}}
{"level":"warn","ts":1735736948.343508,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [db.r4gmi.localhost]: no OCSP server specified in certificate","identifiers":["db.r4gmi.localhost"]}
{"level":"debug","ts":1735736948.3435116,"logger":"tls.cache","msg":"added certificate to cache","subjects":["db.r4gmi.localhost"],"expiration":1735755004,"managed":true,"issuer_key":"local","hash":"24fc00d91a8e268d81550e05ddd2656afeb55011868c740b0d2f33f3bec63d8b","cache_size":4,"cache_capacity":10000}
{"level":"debug","ts":1735736948.3435175,"logger":"events","msg":"event","name":"cached_managed_cert","id":"b13c30ff-366e-4370-93dc-8d78bc981756","origin":"tls","data":{"sans":["db.r4gmi.localhost"]}}
{"level":"warn","ts":1735736948.3436453,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [grafana.r4gmi.localhost]: no OCSP server specified in certificate","identifiers":["grafana.r4gmi.localhost"]}
{"level":"debug","ts":1735736948.3436682,"logger":"tls.cache","msg":"added certificate to cache","subjects":["grafana.r4gmi.localhost"],"expiration":1735755004,"managed":true,"issuer_key":"local","hash":"768e0f62aae7d4d29c892bb36cec60000f1a939fb4795f33fba83bfac93f4ac6","cache_size":5,"cache_capacity":10000}
{"level":"debug","ts":1735736948.3436775,"logger":"events","msg":"event","name":"cached_managed_cert","id":"9c80c772-609e-4414-881f-63c2202c2fcf","origin":"tls","data":{"sans":["grafana.r4gmi.localhost"]}}
{"level":"warn","ts":1735736948.3437452,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [prometheus.r4gmi.localhost]: no OCSP server specified in certificate","identifiers":["prometheus.r4gmi.localhost"]}
{"level":"debug","ts":1735736948.343757,"logger":"tls.cache","msg":"added certificate to cache","subjects":["prometheus.r4gmi.localhost"],"expiration":1735755004,"managed":true,"issuer_key":"local","hash":"8d1173b55a29008075dd24e58f387c6b3d4df939011de0fefaed85c44168c299","cache_size":6,"cache_capacity":10000}
{"level":"debug","ts":1735736948.3437622,"logger":"events","msg":"event","name":"cached_managed_cert","id":"22ba2c62-ce0f-40df-955b-19ed38f82959","origin":"tls","data":{"sans":["prometheus.r4gmi.localhost"]}}
{"level":"warn","ts":1735736948.3438377,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [keycloak.r4gmi.localhost]: no OCSP server specified in certificate","identifiers":["keycloak.r4gmi.localhost"]}
{"level":"debug","ts":1735736948.343849,"logger":"tls.cache","msg":"added certificate to cache","subjects":["keycloak.r4gmi.localhost"],"expiration":1735755004,"managed":true,"issuer_key":"local","hash":"d5b7dbe98e1ca9f2925614ed6d4474ba82f3cd46dc70c85dfe2d0d549910149e","cache_size":7,"cache_capacity":10000}
{"level":"debug","ts":1735736948.3438537,"logger":"events","msg":"event","name":"cached_managed_cert","id":"26d1e827-8724-4e1e-8d45-ad1353667f4c","origin":"tls","data":{"sans":["keycloak.r4gmi.localhost"]}}
{"level":"debug","ts":1735736948.3438742,"logger":"layer4","msg":"listening","address":"tcp/[::]:443"}
{"level":"info","ts":1735736948.3439102,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1735736948.3439214,"logger":"admin.api","msg":"load complete"}
{"level":"info","ts":1735736948.3442402,"logger":"docker-proxy","msg":"Successfully configured","server":"localhost"}
{"level":"info","ts":1735736948.3498926,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"d9ecdc59-25d8-4969-aa2b-905db7dc3663","try_again":1735823348.3498912,"try_again_in":86399.999999732}
{"level":"info","ts":1735736948.3500547,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1735736948.350173,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
{"level":"debug","ts":1735736978.20679,"logger":"docker-proxy","msg":"Skipping default Caddyfile because no path is set"}
{"level":"debug","ts":1735736978.2068348,"logger":"docker-proxy","msg":"Skipping swarm config caddyfiles because swarm is not available"}
{"level":"debug","ts":1735736978.245238,"logger":"docker-proxy","msg":"Skipping swarm services because swarm is not available"}
{"level":"debug","ts":1735737008.2222617,"logger":"docker-proxy","msg":"Skipping default Caddyfile because no path is set"}
{"level":"debug","ts":1735737008.2222822,"logger":"docker-proxy","msg":"Skipping swarm config caddyfiles because swarm is not available"}
{"level":"debug","ts":1735737008.2593832,"logger":"docker-proxy","msg":"Skipping swarm services because swarm is not available"}

and here’s after I open grafana:

{"level":"debug","ts":1735737049.5205104,"logger":"tls.handshake","msg":"choosing certificate","identifier":"grafana.r4gmi.localhost","num_choices":1}
{"level":"debug","ts":1735737049.5205202,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"grafana.r4gmi.localhost","subjects":["grafana.r4gmi.localhost"],"managed":true,"issuer_key":"local","hash":"768e0f62aae7d4d29c892bb36cec60000f1a939fb4795f33fba83bfac93f4ac6"}
{"level":"debug","ts":1735737049.5205355,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"172.18.0.1","remote_port":"58680","subjects":["grafana.r4gmi.localhost"],"managed":true,"expiration":1735755004,"hash":"768e0f62aae7d4d29c892bb36cec60000f1a939fb4795f33fba83bfac93f4ac6"}
{"level":"debug","ts":1735737049.5372202,"logger":"layer4.handlers.tls","msg":"terminated TLS","remote":"172.18.0.1:58680","server_name":"grafana.r4gmi.localhost"}
{"level":"debug","ts":1735737049.5377524,"logger":"layer4.handlers.proxy","msg":"dial upstream","remote":"172.18.0.1:58680","upstream":"r4gmi-grafana:3000"}
{"level":"error","ts":1735737049.5381243,"logger":"layer4.handlers.proxy","msg":"upstream connection","local_address":"172.18.0.5:48152","remote_address":"172.18.0.3:3000","error":"writeto tcp 172.18.0.5:48152->172.18.0.3:3000: read tcp 172.18.0.5:48152->172.18.0.3:3000: read: connection reset by peer"}
{"level":"debug","ts":1735737049.539126,"logger":"layer4","msg":"connection stats","remote":"172.18.0.1:58680","read":2691,"written":1772,"duration":0.019134322}
{"level":"debug","ts":1735737050.6451972,"logger":"layer4","msg":"matching","remote":"172.18.0.1:58688","error":"consumed all prefetched bytes","matcher":"layer4.matchers.tls","matched":false}
{"level":"debug","ts":1735737050.645397,"logger":"layer4","msg":"prefetched","remote":"172.18.0.1:58688","bytes":1981}
{"level":"debug","ts":1735737050.6454144,"logger":"layer4","msg":"matching","remote":"172.18.0.1:58688","matcher":"layer4.matchers.tls","matched":false}
{"level":"debug","ts":1735737050.6454182,"logger":"layer4","msg":"matching","remote":"172.18.0.1:58688","matcher":"layer4.matchers.tls","matched":false}
{"level":"debug","ts":1735737050.645424,"logger":"layer4.matchers.tls","msg":"matched","remote":"172.18.0.1:58688","server_name":"grafana.r4gmi.localhost"}
{"level":"debug","ts":1735737050.6454268,"logger":"layer4","msg":"matching","remote":"172.18.0.1:58688","matcher":"layer4.matchers.tls","matched":true}
{"level":"debug","ts":1735737050.645572,"logger":"events","msg":"event","name":"tls_get_certificate","id":"c0d37a32-2dfd-49ab-9aa2-ae27475d359b","origin":"tls","data":{"client_hello":{"CipherSuites":[14906,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"grafana.r4gmi.localhost","SupportedCurves":[2570,4588,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[39578,772,771],"RemoteAddr":{"IP":"172.18.0.1","Port":58688,"Zone":""},"LocalAddr":{"IP":"172.18.0.5","Port":443,"Zone":""}}}}
{"level":"debug","ts":1735737050.64558,"logger":"tls.handshake","msg":"choosing certificate","identifier":"grafana.r4gmi.localhost","num_choices":1}
{"level":"debug","ts":1735737050.6455894,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"grafana.r4gmi.localhost","subjects":["grafana.r4gmi.localhost"],"managed":true,"issuer_key":"local","hash":"768e0f62aae7d4d29c892bb36cec60000f1a939fb4795f33fba83bfac93f4ac6"}
{"level":"debug","ts":1735737050.6455927,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"172.18.0.1","remote_port":"58688","subjects":["grafana.r4gmi.localhost"],"managed":true,"expiration":1735755004,"hash":"768e0f62aae7d4d29c892bb36cec60000f1a939fb4795f33fba83bfac93f4ac6"}
{"level":"debug","ts":1735737050.6463163,"logger":"layer4.handlers.tls","msg":"terminated TLS","remote":"172.18.0.1:58688","server_name":"grafana.r4gmi.localhost"}
{"level":"debug","ts":1735737050.6466436,"logger":"layer4.handlers.proxy","msg":"dial upstream","remote":"172.18.0.1:58688","upstream":"r4gmi-grafana:3000"}
{"level":"debug","ts":1735737050.6475494,"logger":"layer4","msg":"connection stats","remote":"172.18.0.1:58688","read":2796,"written":1773,"duration":0.002368618}
{"level":"debug","ts":1735737068.2341223,"logger":"docker-proxy","msg":"Skipping default Caddyfile because no path is set"}
{"level":"debug","ts":1735737068.234144,"logger":"docker-proxy","msg":"Skipping swarm config caddyfiles because swarm is not available"}
{"level":"debug","ts":1735737068.2743852,"logger":"docker-proxy","msg":"Skipping swarm services because swarm is not available"}
{"level":"debug","ts":1735737098.2380545,"logger":"docker-proxy","msg":"Skipping default Caddyfile because no path is set"}
{"level":"debug","ts":1735737098.2380965,"logger":"docker-proxy","msg":"Skipping swarm config caddyfiles because swarm is not available"}
{"level":"debug","ts":1735737098.2793589,"logger":"docker-proxy","msg":"Skipping swarm services because swarm is not available"}
{"level":"debug","ts":1735737124.137367,"logger":"layer4","msg":"matching","remote":"172.18.0.1:55710","error":"consumed all prefetched bytes","matcher":"layer4.matchers.tls","matched":false}
{"level":"debug","ts":1735737124.1375642,"logger":"layer4","msg":"prefetched","remote":"172.18.0.1:55710","bytes":1949}
{"level":"debug","ts":1735737124.1375945,"logger":"layer4","msg":"matching","remote":"172.18.0.1:55710","matcher":"layer4.matchers.tls","matched":false}
{"level":"debug","ts":1735737124.137598,"logger":"layer4","msg":"matching","remote":"172.18.0.1:55710","matcher":"layer4.matchers.tls","matched":false}
{"level":"debug","ts":1735737124.1376026,"logger":"layer4.matchers.tls","msg":"matched","remote":"172.18.0.1:55710","server_name":"grafana.r4gmi.localhost"}
{"level":"debug","ts":1735737124.1376035,"logger":"layer4","msg":"matching","remote":"172.18.0.1:55710","matcher":"layer4.matchers.tls","matched":true}
{"level":"debug","ts":1735737124.1377625,"logger":"events","msg":"event","name":"tls_get_certificate","id":"a26866ed-456c-4ae8-847f-c686919632c8","origin":"tls","data":{"client_hello":{"CipherSuites":[51914,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"grafana.r4gmi.localhost","SupportedCurves":[43690,4588,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[14906,772,771],"RemoteAddr":{"IP":"172.18.0.1","Port":55710,"Zone":""},"LocalAddr":{"IP":"172.18.0.5","Port":443,"Zone":""}}}}
{"level":"debug","ts":1735737124.137771,"logger":"tls.handshake","msg":"choosing certificate","identifier":"grafana.r4gmi.localhost","num_choices":1}
{"level":"debug","ts":1735737124.1377838,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"grafana.r4gmi.localhost","subjects":["grafana.r4gmi.localhost"],"managed":true,"issuer_key":"local","hash":"768e0f62aae7d4d29c892bb36cec60000f1a939fb4795f33fba83bfac93f4ac6"}
{"level":"debug","ts":1735737124.1377878,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"172.18.0.1","remote_port":"55710","subjects":["grafana.r4gmi.localhost"],"managed":true,"expiration":1735755004,"hash":"768e0f62aae7d4d29c892bb36cec60000f1a939fb4795f33fba83bfac93f4ac6"}
{"level":"debug","ts":1735737124.1385756,"logger":"layer4.handlers.tls","msg":"terminated TLS","remote":"172.18.0.1:55710","server_name":"grafana.r4gmi.localhost"}
{"level":"debug","ts":1735737124.1390133,"logger":"layer4.handlers.proxy","msg":"dial upstream","remote":"172.18.0.1:55710","upstream":"r4gmi-grafana:3000"}
{"level":"error","ts":1735737124.1393495,"logger":"layer4.handlers.proxy","msg":"upstream connection","local_address":"172.18.0.5:40438","remote_address":"172.18.0.3:3000","error":"writeto tcp 172.18.0.5:40438->172.18.0.3:3000: read tcp 172.18.0.5:40438->172.18.0.3:3000: read: connection reset by peer"}
{"level":"debug","ts":1735737124.1408799,"logger":"layer4","msg":"connection stats","remote":"172.18.0.1:55710","read":2764,"written":1773,"duration":0.003530419}
{"level":"debug","ts":1735737126.4435835,"logger":"layer4","msg":"matching","remote":"172.18.0.1:61850","error":"consumed all prefetched bytes","matcher":"layer4.matchers.tls","matched":false}
{"level":"debug","ts":1735737126.4437046,"logger":"layer4","msg":"prefetched","remote":"172.18.0.1:61850","bytes":1981}
{"level":"debug","ts":1735737126.4437304,"logger":"layer4","msg":"matching","remote":"172.18.0.1:61850","matcher":"layer4.matchers.tls","matched":false}
{"level":"debug","ts":1735737126.4437351,"logger":"layer4","msg":"matching","remote":"172.18.0.1:61850","matcher":"layer4.matchers.tls","matched":false}
{"level":"debug","ts":1735737126.443739,"logger":"layer4.matchers.tls","msg":"matched","remote":"172.18.0.1:61850","server_name":"grafana.r4gmi.localhost"}
{"level":"debug","ts":1735737126.4437401,"logger":"layer4","msg":"matching","remote":"172.18.0.1:61850","matcher":"layer4.matchers.tls","matched":true}
{"level":"debug","ts":1735737126.4438996,"logger":"events","msg":"event","name":"tls_get_certificate","id":"d634dfda-983a-4eda-bfbc-93cb3991873f","origin":"tls","data":{"client_hello":{"CipherSuites":[39578,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"grafana.r4gmi.localhost","SupportedCurves":[27242,4588,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[2570,772,771],"RemoteAddr":{"IP":"172.18.0.1","Port":61850,"Zone":""},"LocalAddr":{"IP":"172.18.0.5","Port":443,"Zone":""}}}}
{"level":"debug","ts":1735737126.4439197,"logger":"tls.handshake","msg":"choosing certificate","identifier":"grafana.r4gmi.localhost","num_choices":1}
{"level":"debug","ts":1735737126.443928,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"grafana.r4gmi.localhost","subjects":["grafana.r4gmi.localhost"],"managed":true,"issuer_key":"local","hash":"768e0f62aae7d4d29c892bb36cec60000f1a939fb4795f33fba83bfac93f4ac6"}
{"level":"debug","ts":1735737126.4439306,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"172.18.0.1","remote_port":"61850","subjects":["grafana.r4gmi.localhost"],"managed":true,"expiration":1735755004,"hash":"768e0f62aae7d4d29c892bb36cec60000f1a939fb4795f33fba83bfac93f4ac6"}
{"level":"debug","ts":1735737126.445756,"logger":"layer4.handlers.tls","msg":"terminated TLS","remote":"172.18.0.1:61850","server_name":"grafana.r4gmi.localhost"}
{"level":"debug","ts":1735737126.4460773,"logger":"layer4.handlers.proxy","msg":"dial upstream","remote":"172.18.0.1:61850","upstream":"r4gmi-grafana:3000"}
{"level":"debug","ts":1735737126.4472823,"logger":"layer4","msg":"connection stats","remote":"172.18.0.1:61850","read":2796,"written":1773,"duration":0.003713864}
{"level":"debug","ts":1735737127.552624,"logger":"layer4","msg":"matching","remote":"172.18.0.1:61862","error":"consumed all prefetched bytes","matcher":"layer4.matchers.tls","matched":false}
{"level":"debug","ts":1735737127.5528169,"logger":"layer4","msg":"prefetched","remote":"172.18.0.1:61862","bytes":1981}
{"level":"debug","ts":1735737127.55284,"logger":"layer4","msg":"matching","remote":"172.18.0.1:61862","matcher":"layer4.matchers.tls","matched":false}
{"level":"debug","ts":1735737127.5528443,"logger":"layer4","msg":"matching","remote":"172.18.0.1:61862","matcher":"layer4.matchers.tls","matched":false}
{"level":"debug","ts":1735737127.552848,"logger":"layer4.matchers.tls","msg":"matched","remote":"172.18.0.1:61862","server_name":"grafana.r4gmi.localhost"}
{"level":"debug","ts":1735737127.5528493,"logger":"layer4","msg":"matching","remote":"172.18.0.1:61862","matcher":"layer4.matchers.tls","matched":true}
{"level":"debug","ts":1735737127.5530124,"logger":"events","msg":"event","name":"tls_get_certificate","id":"6c829531-6863-4172-bf9f-ccbe13f59f82","origin":"tls","data":{"client_hello":{"CipherSuites":[47802,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"grafana.r4gmi.localhost","SupportedCurves":[51914,4588,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[35466,772,771],"RemoteAddr":{"IP":"172.18.0.1","Port":61862,"Zone":""},"LocalAddr":{"IP":"172.18.0.5","Port":443,"Zone":""}}}}
{"level":"debug","ts":1735737127.55302,"logger":"tls.handshake","msg":"choosing certificate","identifier":"grafana.r4gmi.localhost","num_choices":1}
{"level":"debug","ts":1735737127.553028,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"grafana.r4gmi.localhost","subjects":["grafana.r4gmi.localhost"],"managed":true,"issuer_key":"local","hash":"768e0f62aae7d4d29c892bb36cec60000f1a939fb4795f33fba83bfac93f4ac6"}
{"level":"debug","ts":1735737127.5530307,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"172.18.0.1","remote_port":"61862","subjects":["grafana.r4gmi.localhost"],"managed":true,"expiration":1735755004,"hash":"768e0f62aae7d4d29c892bb36cec60000f1a939fb4795f33fba83bfac93f4ac6"}
{"level":"debug","ts":1735737127.5542574,"logger":"layer4.handlers.tls","msg":"terminated TLS","remote":"172.18.0.1:61862","server_name":"grafana.r4gmi.localhost"}
{"level":"debug","ts":1735737127.5563133,"logger":"layer4.handlers.proxy","msg":"dial upstream","remote":"172.18.0.1:61862","upstream":"r4gmi-grafana:3000"}
{"level":"debug","ts":1735737127.5572364,"logger":"layer4","msg":"connection stats","remote":"172.18.0.1:61862","read":2796,"written":1772,"duration":0.004624066}
{"level":"debug","ts":1735737127.7602599,"logger":"events","msg":"event","name":"tls_get_certificate","id":"1d32289e-f56f-4471-bc0c-d31d29f5feea","origin":"tls","data":{"client_hello":{"CipherSuites":[47802,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"grafana.r4gmi.localhost","SupportedCurves":[39578,4588,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[43690,772,771],"RemoteAddr":{"IP":"172.18.0.1","Port":61866,"Zone":""},"LocalAddr":{"IP":"172.18.0.5","Port":443,"Zone":""}}}}
{"level":"debug","ts":1735737127.760285,"logger":"tls.handshake","msg":"choosing certificate","identifier":"grafana.r4gmi.localhost","num_choices":1}
{"level":"debug","ts":1735737127.7602944,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"grafana.r4gmi.localhost","subjects":["grafana.r4gmi.localhost"],"managed":true,"issuer_key":"local","hash":"768e0f62aae7d4d29c892bb36cec60000f1a939fb4795f33fba83bfac93f4ac6"}
{"level":"debug","ts":1735737127.7602973,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"172.18.0.1","remote_port":"61866","subjects":["grafana.r4gmi.localhost"],"managed":true,"expiration":1735755004,"hash":"768e0f62aae7d4d29c892bb36cec60000f1a939fb4795f33fba83bfac93f4ac6"}
{"level":"debug","ts":1735737128.2473958,"logger":"docker-proxy","msg":"Skipping default Caddyfile because no path is set"}
{"level":"debug","ts":1735737128.2474167,"logger":"docker-proxy","msg":"Skipping swarm config caddyfiles because swarm is not available"}
{"level":"debug","ts":1735737128.2969084,"logger":"docker-proxy","msg":"Skipping swarm services because swarm is not available"}
6

Interesting. So I see 2 things:

{"level":"error","ts":1735737124.1393495,"logger":"layer4.handlers.proxy","msg":"upstream connection","local_address":"172.18.0.5:40438","remote_address":"172.18.0.3:3000","error":"writeto tcp 172.18.0.5:40438->172.18.0.3:3000: read tcp 172.18.0.5:40438->172.18.0.3:3000: read: connection reset by peer"}
...
{"level":"debug","ts":1735737126.4435835,"logger":"layer4","msg":"matching","remote":"172.18.0.1:61850","error":"consumed all prefetched bytes","matcher":"layer4.matchers.tls","matched":false}

Per the first line, it seems that Grafana closes the connection. Can you investigate that?

For the second line, I’ll check with the dev more familiar with the layer4 app and the prefetching implementation in particular.

7

Thank you sir. I’ll try to investigate on the grafana side.

8

I can’t find anything beside grafana’s log. Here’s the log in the grafana:

logger=context userId=0 orgId=0 uname= t=2025-01-01T13:10:49.537950263Z level=info msg="Request Completed" method=PRI path=* status=302 remote_addr=172.18.0.5 time_ms=0 duration=60.795µs size=0 referer= handler=notfound status_source=server
logger=context userId=0 orgId=0 uname= t=2025-01-01T13:10:50.646897318Z level=info msg="Request Completed" method=PRI path=* status=302 remote_addr=172.18.0.5 time_ms=0 duration=135.384µs size=0 referer= handler=notfound status_source=server
logger=context userId=0 orgId=0 uname= t=2025-01-01T13:12:04.139221802Z level=info msg="Request Completed" method=PRI path=* status=302 remote_addr=172.18.0.5 time_ms=0 duration=106.406µs size=0 referer= handler=notfound status_source=server
logger=context userId=0 orgId=0 uname= t=2025-01-01T13:12:06.446290259Z level=info msg="Request Completed" method=PRI path=* status=302 remote_addr=172.18.0.5 time_ms=0 duration=43.521µs size=0 referer= handler=notfound status_source=server
logger=context userId=0 orgId=0 uname= t=2025-01-01T13:12:07.556531578Z level=info msg="Request Completed" method=PRI path=* status=302 remote_addr=172.18.0.5 time_ms=0 duration=48.402µs size=0 referer= handler=notfound status_source=server

:frowning:

9

I found this similar issue: Need help configuring caddy & l4 for git ssh access on domain - Help - Caddy Community

But it’s not exactly the same. It also returns blank page. Even on my auth, which is a simple rust server, serving frontend content:

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    let dist_path = if cfg!(feature = "dev") {
        "../frontend/dist"
    } else {
        "/usr/local/share/r4gmi-auth/dist"
    };

    HttpServer::new(move || {
        App::new()
            .service(Files::new("/", dist_path).index_file("index.html").prefer_utf8(true))
            .service(Files::new("/public", format!("{}/public", dist_path)).prefer_utf8(true))
            .default_service(web::get().to(greet))
    })
    .bind("0.0.0.0:8080")?
    .run()
    .await
}

It’s still blank page. Is it because I serve frontend? Why it’s not the case for keycloak?

10

I figured it out :slightly_smiling_face: Here’s what’s happening…

image
image736×980 85.5 KB

The proxy handler of layer-4 proxies the bytes as-is. The tls directive only removes the tls envelope, leaving the underneath bytes the same. HTTPS requests are sent as HTTP/2, so once you remove the TLS envelope, you’re left with h2c, which is the plaintext form of HTTP/2.

The reverse_proxy handler shipped in Caddy HTTP app translates HTTP/2 requests to whatever the upstream understands, which is often HTTP/1.1 if it’s plain HTTP. The proxy handler of layer-4 doesn’t do that because it’s not its job to translate semantics of the bytes. Its job is to only shuttle the bytes back-and-forth without messing with them. The upstream apps don’t understand this because they don’t speak HTTP/2 (h2c). The proxy fails, and Caddy delivers 0 bytes to the client which is why you’re seeing the blank page.

I caught this by setting up this config:

{
	debug
	layer4 {
		:443 {
			@file tls sni file.localhost
			route @file {
				tls
				proxy {
					upstream localhost:8000
				}
			}
		}
	}
}
file.localhost:9443 {
	log
	tls internal
}

The upstream is just Python’s simple HTTP server, ran with python -m http.server. Firefox requests failed with the message “Network Protocol Error” displayed. That was curious. Python logged these messages:

::1 - - [04/Jan/2025 19:42:41] code 505, message Invalid HTTP version (2.0)
::1 - - [04/Jan/2025 19:42:41] "PRI * HTTP/2.0" 505 -

See the PRI HTTP/2? I shutdown Python and ran another Caddy instance serving files hoping for more logs. Caddy printed these:

2025/01/04 16:45:37.230	DEBUG	http.handlers.file_server	sanitized path join	{"site_root": ".", "fs": "", "request_path": "*", "result": "*"}
2025/01/04 16:45:37.231	DEBUG	http.log.error	{id=qard88r3c} fileserver.(*FileServer).notFound (staticfiles.go:705): HTTP 404	{"request": {"remote_ip": "127.0.0.1", "remote_port": "59581", "client_ip": "127.0.0.1", "proto": "HTTP/2.0", "method": "PRI", "host": "", "uri": "*", "headers": {}}, "duration": 0.000189074, "status": 404, "err_id": "qard88r3c", "err_trace": "fileserver.(*FileServer).notFound (staticfiles.go:705)"}

Although Caddy returned 404, yet the browser error was the same. Also, this part of the log is the most important "proto": "HTTP/2.0", "method": "PRI". I could set up an h2c server with Caddy to further validate the theory, but I don’t think I need more validation than this.

Long story short, you need reverse_proxy because it knows how to translate HTTP/2 to HTTP/1.1 for upstream to handle, unless you can somehow get upstream to handle h2c, which most won’t bother to.

2 Likes
11

Very nice write up. TIL about this. So, do I still need layer4? or I just simple using reverse_proxy? I think I do for service like postgresql, or using ssh over tls ( I read some issue in docker-proxy re this). Thanks again for helping me…

12

As you said, for HTTP services, you won’t need layer4, but it’s needed for SSH-over-TLS and postgres. If you want to terminate TLS in front of PostgreSQL, then that’s a different beast because PostgreSQL TLS communication is their own thing (except for recent release, i.e. PostgreSQL 17). See my comment here and the conversation that follows.

1 Like