Panic serving XXX runtime error: invalid memory address or nil pointer dereference

neuronflow · April 12, 2020, 6:41pm

1. My Caddy version (`caddy version`):

caddy/caddy:alpine

2. How I run Caddy:

docker-compose-up

a. System environment:

docker

b. Command:

paste command here

c. Service/unit/compose file:

version: "3"
services:
  ##################################################################################################
  percy:
    # docker run -it -p 80:80 -p 443:443 -p 2019:2019 --rm --name perception_caddy perception_caddy
    image: perception_caddy
    container_name: percy
    hostname: percy
    user: root
    # user: ${CURRENT_UID}
    ports:
      - 443:443
      - 80:80
    extra_hosts:
      - dockerhost:${DOCKERHOST}
    volumes:
      # Just a note - as of the latest caddy/caddy images, these locations are now /config/caddy and /data/caddy. See the (new!) docs for some details: https://github.com/caddyserver/caddy-docker#️-a-note-about-persisted-data 1
      - "./caddy_secrets/data_lets_encrypt_storage:/data"
      - "./caddy_secrets/config_storage:/config"
    # sysctls:
    #   - net.ipv4.ip_unprivileged_port_start=0
    # cap_add:
    #   - CAP_NET_BIND_SERVICE
    restart: unless-stopped```

#### d. My complete Caddyfile or JSON config:

{	
    email   yes@gmail.com
    # acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}

translatum.xyz {
    respond "let the neurons flow - translatum.xyz!"
}

# perception studies
perception.translatum.xyz {
    reverse_proxy {
        to perceptionist:8888
    }
}


glio.translatum.xyz {
    reverse_proxy {
        to bratsstarz:80
    }
}


pneumonia.translatum.xyz {
    reverse_proxy {
        to pneustarz:80
    }
}


# kraken
kraken.translatum.xyz {
    respond "high 8 - long live the kraken on translatum.xyz!"
}


corona.translatum.xyz {
    reverse_proxy {
        to coronakraken:80
    }
}



dev.kraken.translatum.xyz {
    reverse_proxy {
        to http://dockerhost:8080
    }
}


cov19.kraken.translatum.xyz {
    reverse_proxy {
        to http://dockerhost:8081
    }
}


python.kraken.translatum.xyz {
    @aiohttp_allowed_methods {
        method POST
    }
    reverse_proxy @aiohttp_allowed_methods {
        to http://dockerhost:8889
    }
}

upload.kraken.translatum.xyz {
    @tusd_allowed_methods {
	method OPTIONS HEAD PATCH POST
}

    header {
            # TODO fix header is not active
            origin            https://corona.translatum.xyz https://cov19.kraken.translatum.xyz https://dev.kraken.translatum.xyz # both allowed origins in one line?
            allow_credentials true # should I set this to TRUE or false for tusd upload servers?
            # header_up         X-Forwarded-Proto https
            # header_down       X-Forwarded-Proto https 
            # request_header    +X-Forwarded-Proto https
    } 
            
    # tusd
    reverse_proxy @tusd_allowed_methods {
        header_up         X-Forwarded-Proto https
        header_down       X-Forwarded-Proto https 
        to http://dockerhost:3020
    }

# TODO remove metrics from here
    @tusd_metrics {
	method OPTIONS HEAD PATCH POST GET
}
        reverse_proxy /metrics @tusd_metrics {
        header_up         X-Forwarded-Proto https
        header_down       X-Forwarded-Proto https 
        to http://dockerhost:3020
    }
}

3. The problem I’m having:

Suddenly my file uploads don’t work anymore e.g. on cov19.translatum.xyz
I chnaged nothing the caddy container was running for 22h . I restarted all servers and build the caddy image new.

4. Error messages and/or full log output:

percy            | 2020/04/12 18:36:04 http2: panic serving 62.216.209.86:58643: runtime error: invalid memory address or nil pointer dereference
percy            | goroutine 84 [running]:
percy            | net/http.(*http2serverConn).runHandler.func1(0xc00000e0f0, 0xc000921f8e, 0xc00023f080)
percy            | 	net/http/h2_bundle.go:5713 +0x16b
percy            | panic(0x144d380, 0x2470800)
percy            | 	runtime/panic.go:969 +0x166
percy            | github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc000730000, 0x192e1e0, 0xc00000e0f0, 0xc0001ae500)
percy            | 	github.com/caddyserver/caddy/v2@v2.0.0-rc.2/modules/caddyhttp/server.go:203 +0x932
percy            | net/http.serverHandler.ServeHTTP(0xc00072ae00, 0x192e1e0, 0xc00000e0f0, 0xc000662c00)
percy            | 	net/http/server.go:2807 +0xa3
percy            | net/http.initALPNRequest.ServeHTTP(0x19335a0, 0xc0005ec540, 0xc0004fa380, 0xc00072ae00, 0x192e1e0, 0xc00000e0f0, 0xc000662c00)
percy            | 	net/http/server.go:3381 +0x8d
percy            | net/http.(*http2serverConn).runHandler(0xc00023f080, 0xc00000e0f0, 0xc000662c00, 0xc0005a6b80)
percy            | 	net/http/h2_bundle.go:5720 +0x8b
percy            | created by net/http.(*http2serverConn).processHeaders
percy            | 	net/http/h2_bundle.go:5454 +0x4e1
percy            | 2020/04/12 18:36:04 http2: panic serving 62.216.209.86:58643: runtime error: invalid memory address or nil pointer dereference
percy            | goroutine 82 [running]:
percy            | net/http.(*http2serverConn).runHandler.func1(0xc00000e0e0, 0xc000877f8e, 0xc00023f080)
percy            | 	net/http/h2_bundle.go:5713 +0x16b
percy            | panic(0x144d380, 0x2470800)
percy            | 	runtime/panic.go:969 +0x166
percy            | github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc000730000, 0x192e1e0, 0xc00000e0e0, 0xc000663000)
percy            | 	github.com/caddyserver/caddy/v2@v2.0.0-rc.2/modules/caddyhttp/server.go:203 +0x932
percy            | net/http.serverHandler.ServeHTTP(0xc00072ae00, 0x192e1e0, 0xc00000e0e0, 0xc000662800)
percy            | 	net/http/server.go:2807 +0xa3
percy            | net/http.initALPNRequest.ServeHTTP(0x19335a0, 0xc0005ec540, 0xc0004fa380, 0xc00072ae00, 0x192e1e0, 0xc00000e0e0, 0xc000662800)
percy            | 	net/http/server.go:3381 +0x8d
percy            | net/http.(*http2serverConn).runHandler(0xc00023f080, 0xc00000e0e0, 0xc000662800, 0xc0005a6b20)
percy            | 	net/http/h2_bundle.go:5720 +0x8b
percy            | created by net/http.(*http2serverConn).processHeaders
percy            | 	net/http/h2_bundle.go:5454 +0x4e1
percy            | 2020/04/12 18:36:04 http2: panic serving 62.216.209.86:58643: runtime error: invalid memory address or nil pointer dereference
percy            | goroutine 85 [running]:
percy            | net/http.(*http2serverConn).runHandler.func1(0xc00000e108, 0xc000455f8e, 0xc00023f080)
percy            | 	net/http/h2_bundle.go:5713 +0x16b
percy            | panic(0x144d380, 0x2470800)
percy            | 	runtime/panic.go:969 +0x166
percy            | github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc000730000, 0x192e1e0, 0xc00000e108, 0xc0001ae300)
percy            | 	github.com/caddyserver/caddy/v2@v2.0.0-rc.2/modules/caddyhttp/server.go:203 +0x932
percy            | net/http.serverHandler.ServeHTTP(0xc00072ae00, 0x192e1e0, 0xc00000e108, 0xc000662e00)
percy            | 	net/http/server.go:2807 +0xa3
percy            | net/http.initALPNRequest.ServeHTTP(0x19335a0, 0xc0005ec540, 0xc0004fa380, 0xc00072ae00, 0x192e1e0, 0xc00000e108, 0xc000662e00)
percy            | 	net/http/server.go:3381 +0x8d
percy            | net/http.(*http2serverConn).runHandler(0xc00023f080, 0xc00000e108, 0xc000662e00, 0xc0005a6ba0)
percy            | 	net/http/h2_bundle.go:5720 +0x8b
percy            | created by net/http.(*http2serverConn).processHeaders
percy            | 	net/http/h2_bundle.go:5454 +0x4e1
percy            | 2020/04/12 18:36:04 http2: panic serving 62.216.209.86:58643: runtime error: invalid memory address or nil pointer dereference
percy            | goroutine 83 [running]:
percy            | net/http.(*http2serverConn).runHandler.func1(0xc00000e0e8, 0xc00025bf8e, 0xc00023f080)
percy            | 	net/http/h2_bundle.go:5713 +0x16b
percy            | panic(0x144d380, 0x2470800)
percy            | 	runtime/panic.go:969 +0x166
percy            | github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc000730000, 0x192e1e0, 0xc00000e0e8, 0xc000663200)
percy            | 	github.com/caddyserver/caddy/v2@v2.0.0-rc.2/modules/caddyhttp/server.go:203 +0x932
percy            | net/http.serverHandler.ServeHTTP(0xc00072ae00, 0x192e1e0, 0xc00000e0e8, 0xc000662a00)
percy            | 	net/http/server.go:2807 +0xa3
percy            | net/http.initALPNRequest.ServeHTTP(0x19335a0, 0xc0005ec540, 0xc0004fa380, 0xc00072ae00, 0x192e1e0, 0xc00000e0e8, 0xc000662a00)
percy            | 	net/http/server.go:3381 +0x8d
percy            | net/http.(*http2serverConn).runHandler(0xc00023f080, 0xc00000e0e8, 0xc000662a00, 0xc0005a6b60)
percy            | 	net/http/h2_bundle.go:5720 +0x8b
percy            | created by net/http.(*http2serverConn).processHeaders
percy            | 	net/http/h2_bundle.go:5454 +0x4e1
percy            | 2020/04/12 18:36:04 http2: panic serving 62.216.209.86:58643: runtime error: invalid memory address or nil pointer dereference
percy            | goroutine 116 [running]:
percy            | net/http.(*http2serverConn).runHandler.func1(0xc0000c6088, 0xc00025bf8e, 0xc00023f080)
percy            | 	net/http/h2_bundle.go:5713 +0x16b
percy            | panic(0x144d380, 0x2470800)
percy            | 	runtime/panic.go:969 +0x166
percy            | github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc000730000, 0x192e1e0, 0xc0000c6088, 0xc0001ae700)
percy            | 	github.com/caddyserver/caddy/v2@v2.0.0-rc.2/modules/caddyhttp/server.go:203 +0x932
percy            | net/http.serverHandler.ServeHTTP(0xc00072ae00, 0x192e1e0, 0xc0000c6088, 0xc00090a600)
percy            | 	net/http/server.go:2807 +0xa3
percy            | net/http.initALPNRequest.ServeHTTP(0x19335a0, 0xc0005ec540, 0xc0004fa380, 0xc00072ae00, 0x192e1e0, 0xc0000c6088, 0xc00090a600)
percy            | 	net/http/server.go:3381 +0x8d
percy            | net/http.(*http2serverConn).runHandler(0xc00023f080, 0xc0000c6088, 0xc00090a600, 0xc00054e600)
percy            | 	net/http/h2_bundle.go:5720 +0x8b
percy            | created by net/http.(*http2serverConn).processHeaders
percy            | 	net/http/h2_bundle.go:5454 +0x4e1
percy            | 2020/04/12 18:36:04 http2: panic serving 62.216.209.86:58643: runtime error: invalid memory address or nil pointer dereference
percy            | goroutine 132 [running]:
percy            | net/http.(*http2serverConn).runHandler.func1(0xc00020c060, 0xc000877f8e, 0xc00023f080)
percy            | 	net/http/h2_bundle.go:5713 +0x16b
percy            | panic(0x144d380, 0x2470800)
percy            | 	runtime/panic.go:969 +0x166
percy            | github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc000730000, 0x192e1e0, 0xc00020c060, 0xc000663600)
percy            | 	github.com/caddyserver/caddy/v2@v2.0.0-rc.2/modules/caddyhttp/server.go:203 +0x932
percy            | net/http.serverHandler.ServeHTTP(0xc00072ae00, 0x192e1e0, 0xc00020c060, 0xc0001af000)
percy            | 	net/http/server.go:2807 +0xa3
percy            | net/http.initALPNRequest.ServeHTTP(0x19335a0, 0xc0005ec540, 0xc0004fa380, 0xc00072ae00, 0x192e1e0, 0xc00020c060, 0xc0001af000)
percy            | 	net/http/server.go:3381 +0x8d
percy            | net/http.(*http2serverConn).runHandler(0xc00023f080, 0xc00020c060, 0xc0001af000, 0xc000716da0)
percy            | 	net/http/h2_bundle.go:5720 +0x8b
percy            | created by net/http.(*http2serverConn).processHeaders
percy            | 	net/http/h2_bundle.go:5454 +0x4e1
percy            | 2020/04/12 18:36:04 http2: panic serving 62.216.209.86:58643: runtime error: invalid memory address or nil pointer dereference
percy            | goroutine 131 [running]:
percy            | net/http.(*http2serverConn).runHandler.func1(0xc00020c058, 0xc000455f8e, 0xc00023f080)
percy            | 	net/http/h2_bundle.go:5713 +0x16b
percy            | panic(0x144d380, 0x2470800)
percy            | 	runtime/panic.go:969 +0x166
percy            | github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc000730000, 0x192e1e0, 0xc00020c058, 0xc0001af200)
percy            | 	github.com/caddyserver/caddy/v2@v2.0.0-rc.2/modules/caddyhttp/server.go:203 +0x932
percy            | net/http.serverHandler.ServeHTTP(0xc00072ae00, 0x192e1e0, 0xc00020c058, 0xc0001aee00)
percy            | 	net/http/server.go:2807 +0xa3
percy            | net/http.initALPNRequest.ServeHTTP(0x19335a0, 0xc0005ec540, 0xc0004fa380, 0xc00072ae00, 0x192e1e0, 0xc00020c058, 0xc0001aee00)
percy            | 	net/http/server.go:3381 +0x8d
percy            | net/http.(*http2serverConn).runHandler(0xc00023f080, 0xc00020c058, 0xc0001aee00, 0xc000716d80)
percy            | 	net/http/h2_bundle.go:5720 +0x8b
percy            | created by net/http.(*http2serverConn).processHeaders
percy            | 	net/http/h2_bundle.go:5454 +0x4e1
percy            | 2020/04/12 18:36:04 http2: panic serving 62.216.209.86:58643: runtime error: invalid memory address or nil pointer dereference
percy            | goroutine 130 [running]:
percy            | net/http.(*http2serverConn).runHandler.func1(0xc00020c048, 0xc000459f8e, 0xc00023f080)
percy            | 	net/http/h2_bundle.go:5713 +0x16b
percy            | panic(0x144d380, 0x2470800)
percy            | 	runtime/panic.go:969 +0x166
percy            | github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc000730000, 0x192e1e0, 0xc00020c048, 0xc000663400)
percy            | 	github.com/caddyserver/caddy/v2@v2.0.0-rc.2/modules/caddyhttp/server.go:203 +0x932
percy            | net/http.serverHandler.ServeHTTP(0xc00072ae00, 0x192e1e0, 0xc00020c048, 0xc0001ae900)
percy            | 	net/http/server.go:2807 +0xa3
percy            | net/http.initALPNRequest.ServeHTTP(0x19335a0, 0xc0005ec540, 0xc0004fa380, 0xc00072ae00, 0x192e1e0, 0xc00020c048, 0xc0001ae900)
percy            | 	net/http/server.go:3381 +0x8d
percy            | net/http.(*http2serverConn).runHandler(0xc00023f080, 0xc00020c048, 0xc0001ae900, 0xc000716d60)
percy            | 	net/http/h2_bundle.go:5720 +0x8b
percy            | created by net/http.(*http2serverConn).processHeaders
percy            | 	net/http/h2_bundle.go:5454 +0x4e1
percy            | 2020/04/12 18:36:05 http2: panic serving 62.216.209.86:58643: runtime error: invalid memory address or nil pointer dereference
percy            | goroutine 154 [running]:
percy            | net/http.(*http2serverConn).runHandler.func1(0xc00000e1a8, 0xc00025ff8e, 0xc00023f080)
percy            | 	net/http/h2_bundle.go:5713 +0x16b
percy            | panic(0x144d380, 0x2470800)
percy            | 	runtime/panic.go:969 +0x166
percy            | github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc000730000, 0x192e1e0, 0xc00000e1a8, 0xc000663a00)
percy            | 	github.com/caddyserver/caddy/v2@v2.0.0-rc.2/modules/caddyhttp/server.go:203 +0x932
percy            | net/http.serverHandler.ServeHTTP(0xc00072ae00, 0x192e1e0, 0xc00000e1a8, 0xc000663800)
percy            | 	net/http/server.go:2807 +0xa3
percy            | net/http.initALPNRequest.ServeHTTP(0x19335a0, 0xc0005ec540, 0xc0004fa380, 0xc00072ae00, 0x192e1e0, 0xc00000e1a8, 0xc000663800)
percy            | 	net/http/server.go:3381 +0x8d
percy            | net/http.(*http2serverConn).runHandler(0xc00023f080, 0xc00000e1a8, 0xc000663800, 0xc0005a7b00)
percy            | 	net/http/h2_bundle.go:5720 +0x8b
percy            | created by net/http.(*http2serverConn).processHeaders
percy            | 	net/http/h2_bundle.go:5454 +0x4e1
percy            | 2020/04/12 18:36:05 http2: panic serving 62.216.209.86:58643: runtime error: invalid memory address or nil pointer dereference
percy            | goroutine 160 [running]:
percy            | net/http.(*http2serverConn).runHandler.func1(0xc00000e1f0, 0xc000687f8e, 0xc00023f080)
percy            | 	net/http/h2_bundle.go:5713 +0x16b
percy            | panic(0x144d380, 0x2470800)
percy            | 	runtime/panic.go:969 +0x166
percy            | github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc000730000, 0x192e1e0, 0xc00000e1f0, 0xc0001af400)
percy            | 	github.com/caddyserver/caddy/v2@v2.0.0-rc.2/modules/caddyhttp/server.go:203 +0x932
percy            | net/http.serverHandler.ServeHTTP(0xc00072ae00, 0x192e1e0, 0xc00000e1f0, 0xc000663c00)
percy            | 	net/http/server.go:2807 +0xa3
percy            | net/http.initALPNRequest.ServeHTTP(0x19335a0, 0xc0005ec540, 0xc0004fa380, 0xc00072ae00, 0x192e1e0, 0xc00000e1f0, 0xc000663c00)
percy            | 	net/http/server.go:3381 +0x8d
percy            | net/http.(*http2serverConn).runHandler(0xc00023f080, 0xc00000e1f0, 0xc000663c00, 0xc0005a7e80)
percy            | 	net/http/h2_bundle.go:5720 +0x8b
percy            | created by net/http.(*http2serverConn).processHeaders
percy            | 	net/http/h2_bundle.go:5454 +0x4e1
percy            | 2020/04/12 18:36:05 http2: panic serving 62.216.209.86:58643: runtime error: invalid memory address or nil pointer dereference
percy            | goroutine 162 [running]:
percy            | net/http.(*http2serverConn).runHandler.func1(0xc00000e210, 0xc000455f8e, 0xc00023f080)
percy            | 	net/http/h2_bundle.go:5713 +0x16b
percy            | panic(0x144d380, 0x2470800)
percy            | 	runtime/panic.go:969 +0x166
percy            | github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc000730000, 0x192e1e0, 0xc00000e210, 0xc000490200)
percy            | 	github.com/caddyserver/caddy/v2@v2.0.0-rc.2/modules/caddyhttp/server.go:203 +0x932
percy            | net/http.serverHandler.ServeHTTP(0xc00072ae00, 0x192e1e0, 0xc00000e210, 0xc000490000)
percy            | 	net/http/server.go:2807 +0xa3
percy            | net/http.initALPNRequest.ServeHTTP(0x19335a0, 0xc0005ec540, 0xc0004fa380, 0xc00072ae00, 0x192e1e0, 0xc00000e210, 0xc000490000)
percy            | 	net/http/server.go:3381 +0x8d
percy            | net/http.(*http2serverConn).runHandler(0xc00023f080, 0xc00000e210, 0xc000490000, 0xc0005a7ee0)
percy            | 	net/http/h2_bundle.go:5720 +0x8b
percy            | created by net/http.(*http2serverConn).processHeaders
percy            | 	net/http/h2_bundle.go:5454 +0x4e1
percy            | 2020/04/12 18:36:05 http2: panic serving 62.216.209.86:58643: runtime error: invalid memory address or nil pointer dereference
percy            | goroutine 161 [running]:
percy            | net/http.(*http2serverConn).runHandler.func1(0xc00000e200, 0xc000459f8e, 0xc00023f080)
percy            | 	net/http/h2_bundle.go:5713 +0x16b
percy            | panic(0x144d380, 0x2470800)
percy            | 	runtime/panic.go:969 +0x166
percy            | github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc000730000, 0x192e1e0, 0xc00000e200, 0xc00090a800)
percy            | 	github.com/caddyserver/caddy/v2@v2.0.0-rc.2/modules/caddyhttp/server.go:203 +0x932
percy            | net/http.serverHandler.ServeHTTP(0xc00072ae00, 0x192e1e0, 0xc00000e200, 0xc000663e00)
percy            | 	net/http/server.go:2807 +0xa3
percy            | net/http.initALPNRequest.ServeHTTP(0x19335a0, 0xc0005ec540, 0xc0004fa380, 0xc00072ae00, 0x192e1e0, 0xc00000e200, 0xc000663e00)
percy            | 	net/http/server.go:3381 +0x8d
percy            | net/http.(*http2serverConn).runHandler(0xc00023f080, 0xc00000e200, 0xc000663e00, 0xc0005a7ec0)
percy            | 	net/http/h2_bundle.go:5720 +0x8b
percy            | created by net/http.(*http2serverConn).processHeaders
percy            | 	net/http/h2_bundle.go:5454 +0x4e1
percy            | 2020/04/12 18:36:08 http2: panic serving 62.216.209.86:58643: runtime error: invalid memory address or nil pointer dereference
percy            | goroutine 103 [running]:
percy            | net/http.(*http2serverConn).runHandler.func1(0xc00019a020, 0xc000687f8e, 0xc00023f080)
percy            | 	net/http/h2_bundle.go:5713 +0x16b
percy            | panic(0x144d380, 0x2470800)
percy            | 	runtime/panic.go:969 +0x166
percy            | github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc000730000, 0x192e1e0, 0xc00019a020, 0xc000604300)
percy            | 	github.com/caddyserver/caddy/v2@v2.0.0-rc.2/modules/caddyhttp/server.go:203 +0x932
percy            | net/http.serverHandler.ServeHTTP(0xc00072ae00, 0x192e1e0, 0xc00019a020, 0xc000604100)
percy            | 	net/http/server.go:2807 +0xa3
percy            | net/http.initALPNRequest.ServeHTTP(0x19335a0, 0xc0005ec540, 0xc0004fa380, 0xc00072ae00, 0x192e1e0, 0xc00019a020, 0xc000604100)
percy            | 	net/http/server.go:3381 +0x8d
percy            | net/http.(*http2serverConn).runHandler(0xc00023f080, 0xc00019a020, 0xc000604100, 0xc000538100)
percy            | 	net/http/h2_bundle.go:5720 +0x8b
percy            | created by net/http.(*http2serverConn).processHeaders
percy            | 	net/http/h2_bundle.go:5454 +0x4e1
percy            | 2020/04/12 18:36:08 http2: panic serving 62.216.209.86:58643: runtime error: invalid memory address or nil pointer dereference
percy            | goroutine 180 [running]:
percy            | net/http.(*http2serverConn).runHandler.func1(0xc0000c60b8, 0xc000455f8e, 0xc00023f080)
percy            | 	net/http/h2_bundle.go:5713 +0x16b
percy            | panic(0x144d380, 0x2470800)
percy            | 	runtime/panic.go:969 +0x166
percy            | github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc000730000, 0x192e1e0, 0xc0000c60b8, 0xc000490400)
percy            | 	github.com/caddyserver/caddy/v2@v2.0.0-rc.2/modules/caddyhttp/server.go:203 +0x932
percy            | net/http.serverHandler.ServeHTTP(0xc00072ae00, 0x192e1e0, 0xc0000c60b8, 0xc00090ac00)
percy            | 	net/http/server.go:2807 +0xa3
percy            | net/http.initALPNRequest.ServeHTTP(0x19335a0, 0xc0005ec540, 0xc0004fa380, 0xc00072ae00, 0x192e1e0, 0xc0000c60b8, 0xc00090ac00)
percy            | 	net/http/server.go:3381 +0x8d
percy            | net/http.(*http2serverConn).runHandler(0xc00023f080, 0xc0000c60b8, 0xc00090ac00, 0xc00054ec60)
percy            | 	net/http/h2_bundle.go:5720 +0x8b
percy            | created by net/http.(*http2serverConn).processHeaders
percy            | 	net/http/h2_bundle.go:5454 +0x4e1
percy            | 2020/04/12 18:36:08 http2: panic serving 62.216.209.86:58643: runtime error: invalid memory address or nil pointer dereference
percy            | goroutine 179 [running]:
percy            | net/http.(*http2serverConn).runHandler.func1(0xc0000c60b0, 0xc000877f8e, 0xc00023f080)
percy            | 	net/http/h2_bundle.go:5713 +0x16b
percy            | panic(0x144d380, 0x2470800)
percy            | 	runtime/panic.go:969 +0x166
percy            | github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc000730000, 0x192e1e0, 0xc0000c60b0, 0xc00090ae00)
percy            | 	github.com/caddyserver/caddy/v2@v2.0.0-rc.2/modules/caddyhttp/server.go:203 +0x932
percy            | net/http.serverHandler.ServeHTTP(0xc00072ae00, 0x192e1e0, 0xc0000c60b0, 0xc00090aa00)
percy            | 	net/http/server.go:2807 +0xa3
percy            | net/http.initALPNRequest.ServeHTTP(0x19335a0, 0xc0005ec540, 0xc0004fa380, 0xc00072ae00, 0x192e1e0, 0xc0000c60b0, 0xc00090aa00)
percy            | 	net/http/server.go:3381 +0x8d
percy            | net/http.(*http2serverConn).runHandler(0xc00023f080, 0xc0000c60b0, 0xc00090aa00, 0xc00054ec20)
percy            | 	net/http/h2_bundle.go:5720 +0x8b
percy            | created by net/http.(*http2serverConn).processHeaders
percy            | 	net/http/h2_bundle.go:5454 +0x4e1
percy            | 2020/04/12 18:36:08 http2: panic serving 62.216.209.86:58643: runtime error: invalid memory address or nil pointer dereference
percy            | goroutine 188 [running]:
percy            | net/http.(*http2serverConn).runHandler.func1(0xc0000c60e8, 0xc000873f8e, 0xc00023f080)
percy            | 	net/http/h2_bundle.go:5713 +0x16b
percy            | panic(0x144d380, 0x2470800)
percy            | 	runtime/panic.go:969 +0x166
percy            | github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc000730000, 0x192e1e0, 0xc0000c60e8, 0xc00090b200)
percy            | 	github.com/caddyserver/caddy/v2@v2.0.0-rc.2/modules/caddyhttp/server.go:203 +0x932
percy            | net/http.serverHandler.ServeHTTP(0xc00072ae00, 0x192e1e0, 0xc0000c60e8, 0xc00090b000)
percy            | 	net/http/server.go:2807 +0xa3
percy            | net/http.initALPNRequest.ServeHTTP(0x19335a0, 0xc0005ec540, 0xc0004fa380, 0xc00072ae00, 0x192e1e0, 0xc0000c60e8, 0xc00090b000)
percy            | 	net/http/server.go:3381 +0x8d
percy            | net/http.(*http2serverConn).runHandler(0xc00023f080, 0xc0000c60e8, 0xc00090b000, 0xc00054f000)
percy            | 	net/http/h2_bundle.go:5720 +0x8b
percy            | created by net/http.(*http2serverConn).processHeaders
percy            | 	net/http/h2_bundle.go:5454 +0x4e1
percy            | 2020/04/12 18:36:13 http2: panic serving 62.216.209.86:58643: runtime error: invalid memory address or nil pointer dereference
percy            | goroutine 112 [running]:
percy            | net/http.(*http2serverConn).runHandler.func1(0xc00019a048, 0xc00025ff8e, 0xc00023f080)
percy            | 	net/http/h2_bundle.go:5713 +0x16b
percy            | panic(0x144d380, 0x2470800)
percy            | 	runtime/panic.go:969 +0x166
percy            | github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc000730000, 0x192e1e0, 0xc00019a048, 0xc000490800)
percy            | 	github.com/caddyserver/caddy/v2@v2.0.0-rc.2/modules/caddyhttp/server.go:203 +0x932
percy            | net/http.serverHandler.ServeHTTP(0xc00072ae00, 0x192e1e0, 0xc00019a048, 0xc000604700)
percy            | 	net/http/server.go:2807 +0xa3
percy            | net/http.initALPNRequest.ServeHTTP(0x19335a0, 0xc0005ec540, 0xc0004fa380, 0xc00072ae00, 0x192e1e0, 0xc00019a048, 0xc000604700)
percy            | 	net/http/server.go:3381 +0x8d
percy            | net/http.(*http2serverConn).runHandler(0xc00023f080, 0xc00019a048, 0xc000604700, 0xc0005384c0)
percy            | 	net/http/h2_bundle.go:5720 +0x8b
percy            | created by net/http.(*http2serverConn).processHeaders
percy            | 	net/http/h2_bundle.go:5454 +0x4e1
percy            | 2020/04/12 18:36:13 http2: panic serving 62.216.209.86:58643: runtime error: invalid memory address or nil pointer dereference
percy            | goroutine 111 [running]:
percy            | net/http.(*http2serverConn).runHandler.func1(0xc00019a040, 0xc000459f8e, 0xc00023f080)
percy            | 	net/http/h2_bundle.go:5713 +0x16b
percy            | panic(0x144d380, 0x2470800)
percy            | 	runtime/panic.go:969 +0x166
percy            | github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc000730000, 0x192e1e0, 0xc00019a040, 0xc000604b00)
percy            | 	github.com/caddyserver/caddy/v2@v2.0.0-rc.2/modules/caddyhttp/server.go:203 +0x932
percy            | net/http.serverHandler.ServeHTTP(0xc00072ae00, 0x192e1e0, 0xc00019a040, 0xc000604500)
percy            | 	net/http/server.go:2807 +0xa3
percy            | net/http.initALPNRequest.ServeHTTP(0x19335a0, 0xc0005ec540, 0xc0004fa380, 0xc00072ae00, 0x192e1e0, 0xc00019a040, 0xc000604500)
percy            | 	net/http/server.go:3381 +0x8d
percy            | net/http.(*http2serverConn).runHandler(0xc00023f080, 0xc00019a040, 0xc000604500, 0xc0005384a0)
percy            | 	net/http/h2_bundle.go:5720 +0x8b
percy            | created by net/http.(*http2serverConn).processHeaders
percy            | 	net/http/h2_bundle.go:5454 +0x4e1
percy            | 2020/04/12 18:36:13 http2: panic serving 62.216.209.86:58643: runtime error: invalid memory address or nil pointer dereference
percy            | goroutine 225 [running]:
percy            | net/http.(*http2serverConn).runHandler.func1(0xc00019a050, 0xc000877f8e, 0xc00023f080)
percy            | 	net/http/h2_bundle.go:5713 +0x16b
percy            | panic(0x144d380, 0x2470800)
percy            | 	runtime/panic.go:969 +0x166
percy            | github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc000730000, 0x192e1e0, 0xc00019a050, 0xc000490600)
percy            | 	github.com/caddyserver/caddy/v2@v2.0.0-rc.2/modules/caddyhttp/server.go:203 +0x932
percy            | net/http.serverHandler.ServeHTTP(0xc00072ae00, 0x192e1e0, 0xc00019a050, 0xc000604900)
percy            | 	net/http/server.go:2807 +0xa3
percy            | net/http.initALPNRequest.ServeHTTP(0x19335a0, 0xc0005ec540, 0xc0004fa380, 0xc00072ae00, 0x192e1e0, 0xc00019a050, 0xc000604900)
percy            | 	net/http/server.go:3381 +0x8d
percy            | net/http.(*http2serverConn).runHandler(0xc00023f080, 0xc00019a050, 0xc000604900, 0xc0005384e0)
percy            | 	net/http/h2_bundle.go:5720 +0x8b
percy            | created by net/http.(*http2serverConn).processHeaders
percy            | 	net/http/h2_bundle.go:5454 +0x4e1
percy            | 2020/04/12 18:36:13 http2: panic serving 62.216.209.86:58643: runtime error: invalid memory address or nil pointer dereference
percy            | goroutine 230 [running]:
percy            | net/http.(*http2serverConn).runHandler.func1(0xc00019a068, 0xc000705f8e, 0xc00023f080)
percy            | 	net/http/h2_bundle.go:5713 +0x16b
percy            | panic(0x144d380, 0x2470800)
percy            | 	runtime/panic.go:969 +0x166
percy            | github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc000730000, 0x192e1e0, 0xc00019a068, 0xc000604f00)
percy            | 	github.com/caddyserver/caddy/v2@v2.0.0-rc.2/modules/caddyhttp/server.go:203 +0x932
percy            | net/http.serverHandler.ServeHTTP(0xc00072ae00, 0x192e1e0, 0xc00019a068, 0xc000604d00)
percy            | 	net/http/server.go:2807 +0xa3
percy            | net/http.initALPNRequest.ServeHTTP(0x19335a0, 0xc0005ec540, 0xc0004fa380, 0xc00072ae00, 0x192e1e0, 0xc00019a068, 0xc000604d00)
percy            | 	net/http/server.go:3381 +0x8d
percy            | net/http.(*http2serverConn).runHandler(0xc00023f080, 0xc00019a068, 0xc000604d00, 0xc000538840)
percy            | 	net/http/h2_bundle.go:5720 +0x8b
percy            | created by net/http.(*http2serverConn).processHeaders
percy            | 	net/http/h2_bundle.go:5454 +0x4e1

5. What I already tried:

Restarted all servers updated caddy docker image to newest base image…docker system prune.

6. Links to relevant resources:

francislavoie · April 12, 2020, 7:44pm

Unfortunately RC2 has a bug - already been fixed on the master branch.

You can either revert to RC1 for the time being until the next release, or replace the /usr/bin/caddy binary in the container with a volume using the latest CI artifact: Fix some godocs · caddyserver/caddy@68cebb2 · GitHub

neuronflow · April 12, 2020, 7:51pm

Thank you I reverted to RC1, no more panic, however I am in CORS trouble now even though the same config worked before without trouble.

Access to XMLHttpRequest at ‘https://upload.kraken.translatum.xyz/corona-upload/’ from origin ‘https://cov19.kraken.translatum.xyz’ has been blocked by CORS policy: Response to preflight request doesn’t pass access control check: No ‘Access-Control-Allow-Origin’ header is present on the requested resource.

francislavoie · April 12, 2020, 8:49pm

Sorry, I don’t know enough about cors to help. Cors stuff typically isn’t Caddy’s fault, so I’d be surprised if it was due to Caddy updating that it stopped working. You can set cors response headers in your config that might tell the browser to be more lenient, but you’ll need to look into that yourself (unless @Whitestrake can help here)

Whitestrake · April 12, 2020, 8:53pm

The requested resource must have a CORS response header.

I don’t see you setting one at all in your Caddyfile. Is the upstream server meant to be issuing one?

neuronflow · April 12, 2020, 9:09pm

Hi from my understanding I must set a field like:

Access-Control-Allow-Origin https://corona.translatum.xyz/ https://dev.kraken.translatum.xyz/ https://cov19.kraken.translatum.xyz/

How can I achieve that this is only active for the upload.translatum.xyz url?

How can i achieve that it is for all urls? It seems I somehow set the header the wrong way…an example in the documentation would be really helpful.

Whitestrake · April 12, 2020, 9:12pm

Use the header directive to set a response header. There are examples at the bottom of the documentation:

Simply add the header directive to this site in your Caddyfile.

Do you mean, how do you add this header to all of your sites? (Add one header directive to each site in your Caddyfile that should send this header.)

Or how do you set a CORS policy that allows all origins? (Use Access-Control-Allow-Origin *.)

neuronflow · April 12, 2020, 9:36pm

Thanks for the quick response, we struggle with the syntax. We tried like this but our header seemed to have no effect at all. Is there example code we can look at?

upload.kraken.translatum.xyz {
    @tusd_allowed_methods {
	method OPTIONS HEAD PATCH POST
}

    header {
            # TODO fix header is not active
            origin            https://corona.translatum.xyz https://cov19.kraken.translatum.xyz https://dev.kraken.translatum.xyz # both allowed origins in one line?
            allow_credentials true # should I set this to TRUE or false for tusd upload servers?
            # header_up         X-Forwarded-Proto https
            # header_down       X-Forwarded-Proto https 
            # request_header    +X-Forwarded-Proto https
    } 
            
    # tusd
    reverse_proxy @tusd_allowed_methods {
        header_up         X-Forwarded-Proto https
        header_down       X-Forwarded-Proto https 
        to http://dockerhost:3020
    }

# TODO remove metrics from here
    @tusd_metrics {
	method OPTIONS HEAD PATCH POST GET
}
        reverse_proxy /metrics @tusd_metrics {
        header_up         X-Forwarded-Proto https
        header_down       X-Forwarded-Proto https 
        to http://dockerhost:3020
    }
}

francislavoie · April 12, 2020, 9:50pm

You need to wrap header values with quotes if there are spaces in the value, otherwise the Caddyfile parser considers them separate arguments.

Whitestrake · April 13, 2020, 7:55am

Also, in addition to @francislavoie’s comment, I still don’t see you setting any Access-Control-Allow-Origin header anywhere. I can see an origin header being set with that content, but that’s not going to cover it.

system · May 13, 2020, 7:56am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.