10th subdomain hits http2: panic serving - runtime error: invalid memory address or nil pointer dereference

Help
1

1. Output of caddy version:

2.4.5

2. How I run Caddy:

a. System environment:

Ubuntu 20.04.4 LTS
Docker version 20.10.16

b. Command:

 CMD ["caddy" "run" "--config" "/etc/caddy/Caddyfile" "--adapter" "caddyfile"]

c. Service/unit/compose file:

version: '2'
services:
  caddy:
    image: totemic_cattail_0h/docker-caddy-2:latest
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    environment:
      - CLOUDFLARE_EMAIL=my@thdomain.tld
      - CLOUDFLARE_API_TOKEN=[redacted]
    volumes:
      - "/home/ubuntu/caddy-data/Caddyfile:/etc/caddy/Caddyfile"
      - "/home/ubuntu/caddy-data/data:/data"
      - "/home/ubuntu/caddy-data/config:/config"

d. My complete Caddy config:

{
        email totemic_cattail_0h@gmail.com
        # debug
        # acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
        order authorize before reverse_proxy
        order authp before authorize
}

(secure) {
            authorize {
                allow roles primary
                validate bearer header
                set auth url https://auth.thdomain.tld
                inject headers with claims
            }
}

(eppsecure) {
            authorize {
                allow roles epprole
                validate bearer header
                set auth url https://auth.thdomain.tld
                inject headers with claims
            }
}

auth.thdomain.tld {
        route {
            authp {
                crypto default token lifetime 86400
                backend local /data/users.json local
                cookie domain thdomain.tld
                cookie lifetime 86400
                ui {
                    links {
                        "Plex" https://plex.thdomain.tld
                        "HBR" https://hbr.thdomain.tld
                        "Por" https://por.thdomain.tld
                        "Snr" https://snr.thdomain.tld
                        "Rdr" https://rdr.thdomain.tld
                        "Nzbh" https://nzbh.thdomain.tld
                        "Sab" https://sab.thdomain.tld
                        "Z2M" https://z2m.thdomain.tld
                    }
                }
                transform user {
                        match origin local
                        action add role authp/user
                }

            }


        }
}

plex.thdomain.tld {
        route {
                authorize {
                        primary yes
                        allow roles authp/user
                        validate bearer header
                        set auth url https://auth.thdomain.tld
                        inject headers with claims
                }
                reverse_proxy 192.168.50.91:32400
        }
        
}

fnf.thdomain.tld {
        route {
                import eppsecure
                respond "Hello world!"
        }
}

nzbh.thdomain.tld {
        route {
                import secure
                reverse_proxy 192.168.50.91:5076
        }
        
}

snr.thdomain.tld {
        route {
                import secure
                reverse_proxy 192.168.50.91:8989
        }
}

rdr.thdomain.tld {
        route {
                import secure
                reverse_proxy 192.168.50.91:7878
        }
}

por.thdomain.tld {
        route {
                import secure
                reverse_proxy 192.168.50.91:9000
        }
}

hbr.thdomain.tld {
        route { 
                import secure
                reverse_proxy 192.168.50.91:8581
        }
}

sab.thdomain.tld {
        route {
                import secure
                reverse_proxy 192.168.50.91:8080
        }
}

z2m.thdomain.tld {
                import secure
                reverse_proxy 192.168.50.91:9099
        }
}


tls {
        dns cloudflare {env.CLOUDFLARE_API_TOKEN}
        alpn disable_tlsalpn_challenge
}

3. The problem I’m having:

I have 9 subdomains (including auth) running fine on caddy for 1 year (only 1 user).

I’m now trying to add a new subdomain that can be accessed by another user in another role.

I added fnf.thdomain.tld (10th subdomain) yesterday; ran into the following:

  1. 10th subdomain constantly hits 520 error if I have 9 other subdomains also authorising with import secure or import eppsecure. Screenshot below.
  2. 520 error only occurs for the subdomain at the 10th position
  3. 520 does not occur if I have only 9 subdomains with import secure or import eppsecure.
  4. The exact subdomain is irrelevant, it’s always the 10th one hitting the error.
    1. In the logs below, it is z2m.thdomain.tld hits the error, however, when I first put fnf.thdomain.tld as the 10th subdomain in the Caddyfile, fnf.thdomain.tld was hitting the error.

image
image1854×1214 54.7 KB

4. Error messages and/or full log output:

{"level":"debug","ts":1665407485.1255064,"logger":"tls.handshake","msg":"choosing certificate","identifier":"hbr.thdomain.tld","num_choices":1}
{"level":"debug","ts":1665407485.1255949,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"hbr.thdomain.tld","subjects":["hbr.thdomain.tld"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"0e8e041d6fa8fe002d726187ca3af9e35804254815c508a474386a9c9823423a"}
{"level":"debug","ts":1665407485.125633,"logger":"tls.handshake","msg":"matched certificate in cache","subjects":["hbr.thdomain.tld"],"managed":true,"expiration":1668583049,"hash":"0e8e041d6fa8fe002d726187ca3af9e35804254815c508a474386a9c9823423a"}
{"level":"debug","ts":1665407485.1325824,"logger":"http.authentication.providers.authorize","msg":"token validation error","session_id":"","error":"no token found"}
{"level":"debug","ts":1665407485.132658,"logger":"http.authentication.providers.authorize","msg":"redirecting unauthorized user"}
{"level":"error","ts":1665407485.1327634,"logger":"http.handlers.authentication","msg":"auth provider returned error","provider":"authorize","error":"no token found"}
{"level":"debug","ts":1665407485.146789,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.50.91:8581","request":{"remote_addr":"172.70.147.74:40894","proto":"HTTP/2.0","method":"GET","host":"hbr.thdomain.tld","uri":"/assets/snapshot.jpg","headers":{"X-Forwarded-Proto":["https"],"Sec-Ch-Ua":["\"Google Chrome\";v=\"105\", \"Not)A;Brand\";v=\"8\", \"Chromium\";v=\"105\""],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"],"Sec-Fetch-Site":["same-origin"],"Accept-Encoding":["gzip"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"macOS\""],"Sec-Fetch-Mode":["no-cors"],"Cookie":["AUTHP_SESSION_ID=T90JSMIN22RPX4nh0nxj91yvafX9muUCJfioQZ1; access_token=REDACTED],"X-Token-User-Email":["totemic_cattail_0h@gmail.com"],"X-Token-Subject":["totemic_cattail_0h"],"Cf-Ipcountry":["SG"],"X-Forwarded-For":["138.75.0.1, 172.70.147.74"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept":["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"],"Referer":["https://hbr.thdomain.tld/login"],"Cf-Connecting-Ip":["138.75.0.1"],"Cdn-Loop":["cloudflare"],"Cf-Ray":["757f9a0dded0919c-SIN"],"Sec-Fetch-Dest":["image"],"Accept-Language":["en-GB,en-US;q=0.9,en;q=0.8"],"X-Token-User-Roles":["authp/user primary"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"hbr.thdomain.tld"}},"headers":{"Vary":["Origin"],"Accept-Ranges":["bytes"],"Content-Length":["671727"],"Date":["Mon, 10 Oct 2022 13:11:25 GMT"],"Referrer-Policy":["no-referrer"],"Cache-Control":["public,max-age=31536000,immutable"],"Last-Modified":["Sun, 19 Dec 2021 10:07:54 GMT"],"Content-Type":["image/jpeg"],"Connection":["keep-alive"],"Keep-Alive":["timeout=5"],"X-Download-Options":["noopen"],"X-Xss-Protection":["0"],"Expect-Ct":["max-age=0"],"X-Dns-Prefetch-Control":["off"],"X-Content-Type-Options":["nosniff"],"X-Permitted-Cross-Domain-Policies":["none"],"Etag":["W/\"a3fef-17dd2297c90\""],"Content-Security-Policy":["default-src 'self';script-src 'self' 'unsafe-inline' 'unsafe-eval';style-src 'self' 'unsafe-inline';img-src 'self' data: https://raw.githubusercontent.com https://user-images.githubusercontent.com;connect-src 'self' https://openweathermap.org https://api.openweathermap.org wss://hbr.thdomain.tld ws://hbr.thdomain.tld"]},"status":200}
{"level":"debug","ts":1665407501.7252436,"logger":"tls.handshake","msg":"choosing certificate","identifier":"z2m.thdomain.tld","num_choices":1}
{"level":"debug","ts":1665407501.7254107,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"z2m.thdomain.tld","subjects":["z2m.thdomain.tld"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"0ddf6b903704f0e8ece90531fe0b47aa9c6a42a18ed6d24b0bb0a30226b93db7"}
{"level":"debug","ts":1665407501.7254646,"logger":"tls.handshake","msg":"matched certificate in cache","subjects":["z2m.thdomain.tld"],"managed":true,"expiration":1669368180,"hash":"0ddf6b903704f0e8ece90531fe0b47aa9c6a42a18ed6d24b0bb0a30226b93db7"}
{"level":"debug","ts":1665407501.772071,"logger":"http.stdlib","msg":"http2: panic serving 162.158.178.28:34322: runtime error: invalid memory address or nil pointer dereference\ngoroutine 440 [running]:\nnet/http.(*http2serverConn).runHandler.func1(0x30528d8, 0x396dfca, 0x3054540)\n\tnet/http/h2_bundle.go:5825 +0x160\npanic({0x113f3d8, 0x25ddaf0})\n\truntime/panic.go:1038 +0x23c\ngithub.com/greenpau/caddy-authorize/pkg/validator.(*TokenValidator).Authorize(0x0, {0x1ce78c8, 0x2c42020}, 0x33d5200)\n\tgithub.com/greenpau/caddy-authorize@v1.3.18/pkg/validator/sources.go:140 +0x14\ngithub.com/greenpau/caddy-authorize/pkg/authz.Authorizer.Authenticate({{0x33c0498, 0x12}, {0x33d80f8, 0x7}, 0x0, {0x33c0480, 0x16}, 0x0, 0x0, {0x0, ...}, ...}, ...)\n\tgithub.com/greenpau/caddy-authorize@v1.3.18/pkg/authz/authorizer.go:134 +0x10c\ngithub.com/greenpau/caddy-authorize.AuthMiddleware.Authenticate({0x3163500}, {0xe78be530, 0x37e8d08}, 0x33d5200)\n\tgithub.com/greenpau/caddy-authorize@v1.3.18/plugin.go:76 +0x10c\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp/caddyauth.Authentication.ServeHTTP({0x0, 0x33ca8c0, 0x33c88a0}, {0xe78be530, 0x37e8d08}, 0x33d5200, {0x1cd8e6c, 0x36e55c0})\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyauth/caddyauth.go:76 +0xb8\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.(*metricsInstrumentedHandler).ServeHTTP(0x33da2a0, {0xe78be530, 0x37e8cf0}, 0x33d5200, {0x1cd8e6c, 0x36e55c0})\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/metrics.go:133 +0x4a8\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1({0xe78be530, 0x37e8cf0}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:263 +0x44\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x36e55d0, {0xe78be530, 0x37e8cf0}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57 +0x34\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1({0xe78be530, 0x37e8cf0}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:235 +0x2e4\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x2cade00, {0xe78be530, 0x37e8cf0}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57 +0x34\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.(*Subroute).ServeHTTP(0x33b3f40, {0xe78be530, 0x37e8cf0}, 0x33d5200, {0x1cd8e6c, 0x1adcf0c})\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/subroute.go:74 +0x6c\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.(*metricsInstrumentedHandler).ServeHTTP(0x33da2d0, {0x1ce6fdc, 0x30528d8}, 0x33d5200, {0x1cd8e6c, 0x1adcf0c})\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/metrics.go:133 +0x4a8\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1({0x1ce6fdc, 0x30528d8}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:263 +0x44\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x36e55b0, {0x1ce6fdc, 0x30528d8}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57 +0x34\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1({0x1ce6fdc, 0x30528d8}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:235 +0x2e4\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x2cac300, {0x1ce6fdc, 0x30528d8}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57 +0x34\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1({0x1ce6fdc, 0x30528d8}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:203 +0x260\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x2cac3c0, {0x1ce6fdc, 0x30528d8}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57 +0x34\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1({0x1ce6fdc, 0x30528d8}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:203 +0x260\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x2cac480, {0x1ce6fdc, 0x30528d8}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57 +0x34\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1({0x1ce6fdc, 0x30528d8}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:203 +0x260\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x2cac540, {0x1ce6fdc, 0x30528d8}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57 +0x34\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1({0x1ce6fdc, 0x30528d8}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:203 +0x260\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x2cac600, {0x1ce6fdc, 0x30528d8}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57 +0x34\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1({0x1ce6fdc, 0x30528d8}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:203 +0x260\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x2cac6c0, {0x1ce6fdc, 0x30528d8}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57 +0x34\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1({0x1ce6fdc, 0x30528d8}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:203 +0x260\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x2cac840, {0x1ce6fdc, 0x30528d8}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57 +0x34\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1({0x1ce6fdc, 0x30528d8}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:203 +0x260\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x2cac8a0, {0x1ce6fdc, 0x30528d8}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57 +0x34\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1({0x1ce6fdc, 0x30528d8}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:203 +0x260\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x2cac900, {0x1ce6fdc, 0x30528d8}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57 +0x34\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1({0x1ce6fdc, 0x30528d8}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:203 +0x260\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x2cac960, {0x1ce6fdc, 0x30528d8}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57 +0x34\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).enforcementHandler(0x31795e0, {0x1ce6fdc, 0x30528d8}, 0x33d5200, {0x1cd8e6c, 0x2cac960})\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/server.go:298 +0x1c8\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).wrapPrimaryRoute.func1({0x1ce6fdc, 0x30528d8}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/server.go:274 +0x44\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x33da310, {0x1ce6fdc, 0x30528d8}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57 +0x34\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0x31795e0, {0x1ce6fdc, 0x30528d8}, 0x33d5200)\n\tgithub.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/server.go:210 +0x690\nnet/http.serverHandler.ServeHTTP({0x33dc480}, {0x1ce6fdc, 0x30528d8}, 0x3072880)\n\tnet/http/server.go:2878 +0x3f0\nnet/http.initALPNRequest.ServeHTTP({{0x1ce7908, 0x2e28960}, 0x38a8600, {0x33dc480}}, {0x1ce6fdc, 0x30528d8}, 0x3072880)\n\tnet/http/server.go:3479 +0x180\nnet/http.(*http2serverConn).runHandler(0x3054540, 0x30528d8, 0x3072880, 0x3042d20)\n\tnet/http/h2_bundle.go:5832 +0x74\ncreated by net/http.(*http2serverConn).processHeaders\n\tnet/http/h2_bundle.go:5562 +0x590"}
{"level":"debug","ts":1665407503.0814548,"logger":"tls.handshake","msg":"choosing certificate","identifier":"z2m.thdomain.tld","num_choices":1}
{"level":"debug","ts":1665407503.0816019,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"z2m.thdomain.tld","subjects":["z2m.thdomain.tld"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"0ddf6b903704f0e8ece90531fe0b47aa9c6a42a18ed6d24b0bb0a30226b93db7"}

5. What I already tried:

  1. Do not use authp with the new subdomain fnf.thdomain.tld. however, this is not ideal.

6. Links to relevant resources:

2

The stack trace shows caddy-authorize; /cc @greenpau

3

@matt , the error message means that token in a cookie was not found.
Is there a difference in cookie handling between HTTP/2 vs /3?

4

Shouldn’t be, although the HTTP/3 server is different code from the HTTP/2 server.

Stack trace shows net/http.(*http2serverConn).runHandler – where are you getting HTTP/3 from?

PS. @totemic_cattail_0h You should upgrade your Caddy version to 2.6.1.

5

@totemic_cattail_0h , you should not be using caddy-authorize plugin. It is depreciated. Please switch to caddy-security.

6

@matt , just curious. It is unrelated to this issue. It came up in some other context.

7

This topic was automatically closed after 30 days. New replies are no longer allowed.