Caddy crash after upgrade v2.7.2 from v2.6.4

W0n9 · August 3, 2023, 7:42am

1. The problem I’m having:

caddy crash after upgrade v2.7.2 from v2.6.4

2. Error messages and/or full log output:

8月 03 14:51:53 NewCaddy systemd[1]: Started Caddy.
8月 03 14:51:53 NewCaddy caddy[1591605]: caddy.HomeDir=/var/lib/caddy
8月 03 14:51:53 NewCaddy caddy[1591605]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
8月 03 14:51:53 NewCaddy caddy[1591605]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
8月 03 14:51:53 NewCaddy caddy[1591605]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
8月 03 14:51:53 NewCaddy caddy[1591605]: caddy.Version=v2.7.2 h1:QqThyoyUFAv1B7A2NMeaWlz7xmgKqU49PXBX08A+6xg=
8月 03 14:51:53 NewCaddy caddy[1591605]: runtime.GOOS=linux
8月 03 14:51:53 NewCaddy caddy[1591605]: runtime.GOARCH=amd64
8月 03 14:51:53 NewCaddy caddy[1591605]: runtime.Compiler=gc
8月 03 14:51:53 NewCaddy caddy[1591605]: runtime.NumCPU=16
8月 03 14:51:53 NewCaddy caddy[1591605]: runtime.GOMAXPROCS=16
8月 03 14:51:53 NewCaddy caddy[1591605]: runtime.Version=go1.20.7
8月 03 14:51:53 NewCaddy caddy[1591605]: os.Getwd=/
8月 03 14:51:53 NewCaddy caddy[1591605]: LANG=zh_CN.UTF-8
8月 03 14:51:53 NewCaddy caddy[1591605]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
8月 03 14:51:53 NewCaddy caddy[1591605]: HOME=/var/lib/caddy
8月 03 14:51:53 NewCaddy caddy[1591605]: LOGNAME=caddy
8月 03 14:51:53 NewCaddy caddy[1591605]: USER=caddy
8月 03 14:51:53 NewCaddy caddy[1591605]: INVOCATION_ID=a7249b07f5dc4641b77e45a70d90d891
8月 03 14:51:53 NewCaddy caddy[1591605]: JOURNAL_STREAM=8:962767775
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.1151452,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"warn","ts":1691045513.116646,"logger":"caddyfile","msg":"Placeholder {args.0} deprecated, use {args[0]} instead"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.1612985,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.1622434,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0004e6600"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.1623552,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"warn","ts":1691045513.1624315,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","h
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.1628265,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.166393,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.1718645,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.1746206,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.1750474,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.1778736,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.178231,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.1809976,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.1813412,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.1839685,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.184481,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.1871123,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.1875393,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.190509,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.191071,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.194098,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.1944644,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.196886,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.1973703,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2000952,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2004476,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2031069,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2046206,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2073767,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2077804,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.210709,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2111328,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2137358,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2143247,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2168136,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2171214,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2198417,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.220278,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2230465,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2234306,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2260604,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.22643,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.229136,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2295516,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.232313,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.232693,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.235696,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2360413,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2390244,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.239356,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2421484,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2425027,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2452998,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2456965,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2483072,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2487435,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2512512,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2516313,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2540069,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2543724,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2568827,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2573276,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2599058,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2603161,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2632115,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.2636094,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.266504,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.266888,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.3047216,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.3050756,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.3079932,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.3083735,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.3104901,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.3108678,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.3132403,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.3137724,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.316058,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.316444,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.3189216,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.3193362,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.322051,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.322513,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.3253868,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.3256934,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.328465,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.3300958,"logger":"http.handlers.waf_chaitin","msg":"Provisioning WAF plugin instance"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.333279,"logger":"http.handlers.waf_chaitin","msg":"waf plugin validate"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.3340855,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.3341367,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.3344657,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.3345726,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.334671,"logger":"http.log","msg":"server running","name":"srv2","protocols":["h1","h2","h3"]}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.334695,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["*.buct.edu.cn","experimental-auth-endpoint.buct.edu.cn"]}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.3355553,"logger":"tls","msg":"finished cleaning storage units"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.3369114,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
8月 03 14:51:53 NewCaddy caddy[1591605]: {"level":"info","ts":1691045513.3369353,"msg":"serving initial configuration"}
8月 03 14:51:57 NewCaddy caddy[1591605]: {"level":"info","ts":1691045517.6671886,"msg":"Unsolicited response received on idle HTTP channel starting with \"File Not Modified\\r\\n\"; err=<nil>"}
8月 03 14:51:57 NewCaddy caddy[1591605]: {"level":"info","ts":1691045517.6811957,"msg":"Unsolicited response received on idle HTTP channel starting with \"File Not Modified\\r\\n\"; err=<nil>"}
8月 03 14:51:57 NewCaddy caddy[1591605]: {"level":"info","ts":1691045517.6813276,"msg":"Unsolicited response received on idle HTTP channel starting with \"File Not Modified\\r\\n\"; err=<nil>"}
8月 03 14:51:57 NewCaddy caddy[1591605]: {"level":"info","ts":1691045517.6845403,"msg":"Unsolicited response received on idle HTTP channel starting with \"File Not Modified\\r\\n\"; err=<nil>"}
8月 03 14:51:59 NewCaddy caddy[1591605]: {"level":"info","ts":1691045519.2223284,"msg":"Unsolicited response received on idle HTTP channel starting with \"File Not Modified\\r\\n\"; err=<nil>"}
8月 03 14:52:28 NewCaddy caddy[1591605]: {"level":"error","ts":1691045548.326144,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"121.195.132.11:443","duration":0.025321442,"request":{"remoted92:532","remote_port":"51902","client_ip":"2001:da8:237:200:d9c7:c614:2d92:532","proto":"HTTP/3.0","method":"GET","host":"www.buct.edu.cn","uri":"/_upload/article/images/b4/ad/6654543e48c98d973d3cc2043a79/86a82685-39d0-4b83-8978-7509d-Mobile":["?0"],"Sec-Fetch-Mode":["no-cors"],"Sec-Fetch-Dest":["image"],"Sec-Ch-Ua-Platform":["\"Windows\""],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["www.buct.edu.cn"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) cko) Chrome/115.0.0.0 Safari/537.36 Edg/115.0.1901.188"],"Cookie":[],"Accept-Encoding":["gzip, deflate, br"],"Sec-Ch-Ua":["\"Not/A)Brand\";v=\"99\", \"Microsoft Edge\";v=\"115\", \"Chromium\";v=\"115\""],"Accept-Language":["zh-CN,zh;q=.6"],"X-Forwarded-For":["2001:da8:237:200:d9c7:c614:2d92:532"],"Accept":["image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"],"Sec-Fetch-Site":["same-origin"],"Referer":["https://www.buct.edu.cn/main.htm"]},"tls":{"resumed":true,"vroto":"h3","server_name":"www.buct.edu.cn"}},"error":"writing: stream 12 canceled by remote with error code 268"}
8月 03 14:52:28 NewCaddy caddy[1591605]: {"level":"error","ts":1691045548.3263478,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"121.195.132.11:443","duration":0.023191798,"request":{"remot2d92:532","remote_port":"51902","client_ip":"2001:da8:237:200:d9c7:c614:2d92:532","proto":"HTTP/3.0","method":"GET","host":"www.buct.edu.cn","uri":"/_upload/article/images/e8/1e/6bd2e1c7449eac6def3263cf9861/6b624334-edd5-4866-95b3-6aa6h-Site":["same-origin"],"Cookie":[],"X-Forwarded-For":["2001:da8:237:200:d9c7:c614:2d92:532"],"Sec-Fetch-Dest":["image"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-Mode":["no-cors"],"Sec-Ch-Ua-Mobile":["?0"],"User-Agent":["Mozilla AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36 Edg/115.0.1901.188"],"Accept":["image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"],"Referer":["https://www.buct.edu.cn/main.htm"],"X-Forwarded-Host":["www.buctN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6"],"Sec-Ch-Ua":["\"Not/A)Brand\";v=\"99\", \"Microsoft Edge\";v=\"115\", \"Chromium\";v=\"115\""],"Accept-Encoding":["gzip, deflate, br"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":true,"proto":"h3","server_name":"www.buct.edu.cn"}},"error":"writing: stream 0 canceled by remote with error code 268"}
8月 03 14:52:28 NewCaddy caddy[1591605]: {"level":"error","ts":1691045548.3287551,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"121.195.132.11:443","duration":0.020698119,"request":{"remot2d92:532","remote_port":"51902","client_ip":"2001:da8:237:200:d9c7:c614:2d92:532","proto":"HTTP/3.0","method":"GET","host":"www.buct.edu.cn","uri":"/_upload/article/images/ae/72/0feefb0848f395edb3301a9e6c46/71a13e4a-3217-4665-9185-a489nt":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36 Edg/115.0.1901.188"],"X-Forwarded-For":["2001:da8:237:200:d9c7:c614:2d92:532"],"X-Forwarded-Host":["www.buct.edu.cn"]t-Encoding":["gzip, deflate, br"],"Sec-Fetch-Mode":["no-cors"],"Referer":["https://www.buct.edu.cn/main.htm"],"X-Forwarded-Proto":["https"],"Accept-Language":["zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6"],"Sec-Ch-Ua-Mobile":["?0"]""],"Accept":["image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"],"Cookie":[],"Sec-Ch-Ua":["\"Not/A)Brand\";v=\"99\", \"Microsoft Edge\";v=\"115\", \"Chromium\";v=\"115\""],"Sec-Fetch-Site":["same-origin"]},"tls":{"resumed":true,"proto":"h3","server_name":"www.buct.edu.cn"}},"error":"writing: stream 4 canceled by remote with error code 268"}
8月 03 14:52:29 NewCaddy caddy[1591605]: panic: runtime error: invalid memory address or nil pointer dereference
8月 03 14:52:29 NewCaddy caddy[1591605]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0xa6506a]
8月 03 14:52:29 NewCaddy caddy[1591605]: goroutine 20030 [running]:
8月 03 14:52:29 NewCaddy caddy[1591605]: github.com/quic-go/quic-go/internal/ackhandler.(*sentPacketHandler).ReceivedAck(0xc0023270c0?, 0xc0023270c0?, 0xa5?, {0xc001d63802?, 0x10200002c?, 0x2b874e0?})
8月 03 14:52:29 NewCaddy caddy[1591605]:         github.com/quic-go/quic-go@v0.37.1/internal/ackhandler/sent_packet_handler.go:298 +0x6a
8月 03 14:52:29 NewCaddy caddy[1591605]: github.com/quic-go/quic-go.(*connection).handleAckFrame(0xc001d63800, 0xc002327100, 0xe0?)
8月 03 14:52:29 NewCaddy caddy[1591605]:         github.com/quic-go/quic-go@v0.37.1/connection.go:1484 +0x4e
8月 03 14:52:29 NewCaddy caddy[1591605]: github.com/quic-go/quic-go.(*connection).handleFrame(0xc001d63800, {0x1f116e0?, 0xc002327100?}, 0x40?, {{0x82, 0x92, 0xc8, 0x7c, 0x0, 0x0, ...}, ...})
8月 03 14:52:29 NewCaddy caddy[1591605]:         github.com/quic-go/quic-go@v0.37.1/connection.go:1280 +0xed
8月 03 14:52:29 NewCaddy caddy[1591605]: github.com/quic-go/quic-go.(*connection).handleFrames(0xc001d63800, {0xc0024f2c0e?, 0xc00216b860?, 0x40deca?}, {{0x82, 0x92, 0xc8, 0x7c, 0x0, 0x0, ...}, ...}, ...)
8月 03 14:52:29 NewCaddy caddy[1591605]:         github.com/quic-go/quic-go@v0.37.1/connection.go:1253 +0x365
8月 03 14:52:29 NewCaddy caddy[1591605]: github.com/quic-go/quic-go.(*connection).handleUnpackedLongHeaderPacket(0xc001d63800, 0xc001425b00, 0xa7?, {0x60?, 0x0?, 0x2b874e0?}, 0x4a)
8月 03 14:52:29 NewCaddy caddy[1591605]:         github.com/quic-go/quic-go@v0.37.1/connection.go:1194 +0x6aa
8月 03 14:52:29 NewCaddy caddy[1591605]: github.com/quic-go/quic-go.(*connection).handleLongHeaderPacket(0xc001d63800, {0xc001abee40, {0x1f12f68, 0xc001425a10}, {0xc12af18b6fc266cf, 0x88d317b02, 0x2b874e0}, {0xc0024f2c00, 0x4a, 0x5ac},
8月 03 14:52:29 NewCaddy caddy[1591605]:         github.com/quic-go/quic-go@v0.37.1/connection.go:967 +0x765
8月 03 14:52:29 NewCaddy caddy[1591605]: github.com/quic-go/quic-go.(*connection).handlePacketImpl(0xc001d63800, {0xc001abee40, {0x1f12f68, 0xc001425a10}, {0xc12af18b6fc266cf, 0x88d317b02, 0x2b874e0}, {0xc0024f2c00, 0x4a, 0x5ac}, ...})
8月 03 14:52:29 NewCaddy caddy[1591605]:         github.com/quic-go/quic-go@v0.37.1/connection.go:849 +0x1a8
8月 03 14:52:29 NewCaddy caddy[1591605]: github.com/quic-go/quic-go.(*connection).run(0xc001d63800)
8月 03 14:52:29 NewCaddy caddy[1591605]:         github.com/quic-go/quic-go@v0.37.1/connection.go:560 +0x525
8月 03 14:52:29 NewCaddy caddy[1591605]: created by github.com/quic-go/quic-go.(*baseServer).handleInitialImpl
8月 03 14:52:29 NewCaddy caddy[1591605]:         github.com/quic-go/quic-go@v0.37.1/server.go:672 +0x7cd
8月 03 14:52:29 NewCaddy systemd[1]: caddy.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
8月 03 14:52:29 NewCaddy systemd[1]: caddy.service: Failed with result 'exit-code'.
8月 03 14:52:35 NewCaddy systemd[1]: caddy.service: Scheduled restart job, restart counter is at 6.
8月 03 14:52:35 NewCaddy systemd[1]: Stopped Caddy.

3. Caddy version:

v2.7.2 h1:QqThyoyUFAv1B7A2NMeaWlz7xmgKqU49PXBX08A+6xg=

4. How I installed and ran Caddy:

systemd service

a. System environment:

Linux NewCaddy 5.15.0-58-generic #64~20.04.1-Ubuntu SMP Fri Jan 6 16:42:31 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

b. Command:

systemctl restart caddy

c. Service/unit/compose file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE
Restart=always
RestartSec=5
StartLimitIntervalSec=0

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

show below

5. Links to relevant resources:

plugins:

matt · August 3, 2023, 5:20pm

Thanks; we’re currently trying to figure this out over in

github.com/caddyserver/caddy

getCertDuringHandshake causes panic

opened 09:07AM - 03 Aug 23 UTC

otbutz

upstream

needs info

caddy v2.7.2 installed via the apt repository: ``` Aug 03 08:55:31 proxy cad…dy[1163492]: panic: runtime error: invalid memory address or nil pointer dereference Aug 03 08:55:31 proxy caddy[1163492]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x903750] Aug 03 08:55:31 proxy caddy[1163492]: goroutine 57 [running]: Aug 03 08:55:31 proxy caddy[1163492]: github.com/caddyserver/certmagic.(*Config).getCertDuringHandshake(0xc000543520, {0x1f09a88, 0xc000194008}, _, _) Aug 03 08:55:31 proxy caddy[1163492]: github.com/caddyserver/certmagic@v0.19.1/handshake.go:378 +0x1390 Aug 03 08:55:31 proxy caddy[1163492]: github.com/caddyserver/certmagic.(*Config).GetCertificateWithContext(0xc000543520, {0x1f09a88, 0xc000194008}, 0xc000543450) Aug 03 08:55:31 proxy caddy[1163492]: github.com/caddyserver/certmagic@v0.19.1/handshake.go:84 +0xbff Aug 03 08:55:31 proxy caddy[1163492]: github.com/caddyserver/certmagic.(*Config).GetCertificate(0xc000138ee0?, 0xc0001dc1b0?) Aug 03 08:55:31 proxy caddy[1163492]: github.com/caddyserver/certmagic@v0.19.1/handshake.go:50 +0x2a Aug 03 08:55:31 proxy caddy[1163492]: github.com/caddyserver/caddy/v2/modules/caddytls.(*ConnectionPolicy).buildStandardTLSConfig.func1(0xc000543450) Aug 03 08:55:31 proxy caddy[1163492]: github.com/caddyserver/caddy/v2@v2.7.2/modules/caddytls/connpolicy.go:232 +0x14f Aug 03 08:55:31 proxy caddy[1163492]: github.com/quic-go/qtls-go1-20.(*config).getCertificate(0xc0008d7800, 0xc000543450) Aug 03 08:55:31 proxy caddy[1163492]: github.com/quic-go/qtls-go1-20@v0.3.0/common.go:1086 +0x42 Aug 03 08:55:31 proxy caddy[1163492]: github.com/quic-go/qtls-go1-20.(*serverHandshakeStateTLS13).pickCertificate(0xc000631be8) Aug 03 08:55:31 proxy caddy[1163492]: github.com/quic-go/qtls-go1-20@v0.3.0/handshake_server_tls13.go:415 +0x66 Aug 03 08:55:31 proxy caddy[1163492]: github.com/quic-go/qtls-go1-20.(*serverHandshakeStateTLS13).handshake(0xc000631be8) Aug 03 08:55:31 proxy caddy[1163492]: github.com/quic-go/qtls-go1-20@v0.3.0/handshake_server_tls13.go:60 +0x53 Aug 03 08:55:31 proxy caddy[1163492]: github.com/quic-go/qtls-go1-20.(*Conn).serverHandshake(0xc00053d180, {0x1f09a50, 0xc000562fa0}) Aug 03 08:55:31 proxy caddy[1163492]: github.com/quic-go/qtls-go1-20@v0.3.0/handshake_server.go:53 +0x188 Aug 03 08:55:31 proxy caddy[1163492]: github.com/quic-go/qtls-go1-20.(*Conn).handshakeContext(0xc00053d180, {0x1f09af8, 0xc0001753e0}) Aug 03 08:55:31 proxy caddy[1163492]: github.com/quic-go/qtls-go1-20@v0.3.0/conn.go:1540 +0x3ce Aug 03 08:55:31 proxy caddy[1163492]: github.com/quic-go/qtls-go1-20.(*Conn).HandshakeContext(0xc00005dfd0?, {0x1f09af8?, 0xc0001753e0?}) Aug 03 08:55:31 proxy caddy[1163492]: github.com/quic-go/qtls-go1-20@v0.3.0/conn.go:1480 +0x25 Aug 03 08:55:31 proxy caddy[1163492]: created by github.com/quic-go/qtls-go1-20.(*QUICConn).Start Aug 03 08:55:31 proxy caddy[1163492]: github.com/quic-go/qtls-go1-20@v0.3.0/quic.go:179 +0xcf ``` Might be related to https://github.com/golang/go/issues/61639 ?

I should note that we’re using 2.7.2 for this site and our main Caddy website with no issues. We haven’t been able to reproduce the problem.

If you restart your server (maybe even power-cycle) does the error still happen?

W0n9 · August 4, 2023, 2:08am

I have tried restart caddy and even reboot my server, it still have this problem.
I have rolled back to v2.6.4 now

system · September 3, 2023, 2:09am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.