Caddy hangs trying to stop it

Help
1

1. The problem I’m having:

I’m sorry you closed the thread linked below, since I’m having the same issue.
caddy stop hangs and does not exit.

2. Error messages and/or full log output:

I’m getting the following immediate response when trying to stop:

2023/09/20 19:19:41.985	INFO	admin.api	received request	{"method": "POST", "host": "localhost:2019", "uri": "/stop", "remote_ip": "127.0.0.1", "remote_port": "56182", "headers": {"Accept-Encoding":["gzip"],"Content-Length":["0"],"Origin":["http://localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
2023/09/20 19:19:41.985	WARN	admin.api	exiting; byeee!! 👋

And then it hangs untill, some time later it throws:

2023/09/20 19:37:43.949	ERROR	http.handlers.reverse_proxy	aborting with incomplete response	  {"upstream": "localhost:11181", "duration": 0.186942461, "request": {"remote_ip": "127.0.0.1", "remote_port": "53698", "client_ip": "127.0.0.1", "proto": "HTTP/2.0", "method": "GET", "host": "localbluff.stage.whatever.biz", "uri": "/my-client/public/__webpack_hmr", "headers": {"X-Forwarded-For": ["127.0.0.1"], "Sec-Ch-Ua-Mobile": ["?0"], "Accept-Language": ["en-US,en;q=0.9"], "Sec-Fetch-Dest": ["empty"], "Accept": ["text/event-stream"], "Cache-Control": ["no-cache"], "Sec-Ch-Ua-Platform": ["\"macOS\""], "X-Forwarded-Host": ["localbluff.stage.whatever.biz"], "Referer": ["https:/localbluff.stage.whatever.biz/insurance/travel/match"], "Sec-Ch-Ua": ["\"Chromium\";v=\"116\", \"Not)A;Brand\";v=\"24\", \"Google Chrome\";v=\"116\""], "Sec-Fetch-Site": ["same-origin"], "X-Forwarded-Proto": ["https"], "Accept-Encoding": ["gzip, deflate, br"], "Cookie": [], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"], "Sec-Fetch-Mode": ["cors"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "localbluff.stage.whatever.biz"}}, "error": "reading: context canceled"}
2023/09/20 19:37:45.129	INFO	http: panic serving 127.0.0.1:56182: close of closed channel
goroutine 1812 [running]:
net/http.(*conn).serve.func1()
	net/http/server.go:1868 +0xb9
panic({0x235f8c0?, 0x2a43a90?})
	runtime/panic.go:920 +0x270
github.com/caddyserver/caddy/v2/modules/caddytls.(*TLS).Stop(0xc0002730a0)
	github.com/caddyserver/caddy/v2@v2.7.4/modules/caddytls/tls.go:318 +0x27
github.com/caddyserver/caddy/v2.unsyncedStop({{0x2a5b7c8, 0xc0003f6ff0}, 0xc000242a50, 0xc0003f6f50, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}})
	github.com/caddyserver/caddy/v2@v2.7.4/caddy.go:669 +0xb6
github.com/caddyserver/caddy/v2.Stop()
	github.com/caddyserver/caddy/v2@v2.7.4/caddy.go:638 +0xbc
github.com/caddyserver/caddy/v2.exitProcess({0x2a5b6e8, 0x37370e0}, 0xc0003b8a10)
	github.com/caddyserver/caddy/v2@v2.7.4/caddy.go:712 +0xf3
github.com/caddyserver/caddy/v2.handleStop({0x5?, 0x25b01a1?}, 0x5?)
	github.com/caddyserver/caddy/v2@v2.7.4/admin.go:1069 +0xae
github.com/caddyserver/caddy/v2.AdminHandlerFunc.ServeHTTP(0x1375320?, {0x2a56960?, 0xc0002a01c8?}, 0xc00063b5f8?)
	github.com/caddyserver/caddy/v2@v2.7.4/admin.go:1273 +0x29
github.com/caddyserver/caddy/v2.(*AdminConfig).newAdminHandler.func2.1({0x2a56960, 0xc0002a01c8}, 0xc0003ee000)
	github.com/caddyserver/caddy/v2@v2.7.4/admin.go:240 +0x7b
net/http.HandlerFunc.ServeHTTP(0x2a56810?, {0x2a56960?, 0xc0002a01c8?}, 0x240dce0?)
	net/http/server.go:2136 +0x29
github.com/caddyserver/caddy/v2.(*AdminConfig).newAdminHandler.func1.instrumentHandlerCounter.func1({0x2a56810?, 0xc000422000?}, 0xc0003ee000)
	github.com/caddyserver/caddy/v2@v2.7.4/metrics.go:47 +0x6f
net/http.HandlerFunc.ServeHTTP(0xc0001fdf98?, {0x2a56810?, 0xc000422000?}, 0x12fa3df?)
	net/http/server.go:2136 +0x29
net/http.(*ServeMux).ServeHTTP(0xc0002e0440?, {0x2a56810, 0xc000422000}, 0xc0003ee000)
	net/http/server.go:2514 +0x142
github.com/caddyserver/caddy/v2.adminHandler.serveHTTP({0xc0002e0440, 0x0, 0x1, {0xc0001fdf98, 0x3, 0x3}, 0x0}, {0x2a56810, 0xc000422000}, 0xc0003ee000)
	github.com/caddyserver/caddy/v2@v2.7.4/admin.go:837 +0x55f
github.com/caddyserver/caddy/v2.adminHandler.ServeHTTP({0xc0002e0440, 0x0, 0x1, {0xc0001fdf98, 0x3, 0x3}, 0x0}, {0x2a56810, 0xc000422000}, 0xc0003ee000)
	github.com/caddyserver/caddy/v2@v2.7.4/admin.go:789 +0x7de
net/http.serverHandler.ServeHTTP({0xc0007f31d0?}, {0x2a56810?, 0xc000422000?}, 0x6?)
	net/http/server.go:2938 +0x8e
net/http.(*conn).serve(0xc00040a000, {0x2a5b790, 0xc0003857a0})
	net/http/server.go:2009 +0x5f4
created by net/http.(*Server).Serve in goroutine 57
	net/http/server.go:3086 +0x5cb
2023/09/20 19:37:55.130	ERROR	admin.api	failed to stop local admin server gracefully	r": "shutting down admin server: context deadline exceeded"}
2023/09/20 19:37:55.130	ERROR	admin.api	unclean shutdown	{"exit_code": 3}
2023/09/20 19:37:55.134	WARN	failed using API to stop instance	{"error": "performing request: Post \"http://localhost:2019/stop\": EOF"}
Error: performing request: Post "http://localhost:2019/stop": EOF

3. Caddy version:

v2.7.4 h1:J8nisjdOxnYHXlorUKXY75Gr6iBfudfoGhrJ8t7/flI=

4. How I installed and ran Caddy:

Homebrew, and run it with caddy start

a. System environment:

MacOS Ventura 13.5.2

b. Command:

caddy start

c. Service/unit/compose file:

I have no idea what this is

d. My complete Caddy config:

Caddyfile:

localbluff.stage.whatever.biz {
	tls internal

	import flows-client/app.Caddyfile
	import insurance-client/app.Caddyfile

	handle {
		reverse_proxy https://stage.whatever.biz {
			header_up host stage.whatever.biz
		}
	}
}

flows-client/app.Caddyfile

@flows path_regexp \/(home-)?insurance(?:\/[^/]+)?\/match

handle @flows {
    reverse_proxy localhost:11181
}

handle /flows-client* {
    reverse_proxy localhost:11181
}

insurance-client/app.Caddyfile

handle /insurance/auto/view-your-car-insurance-matches {
   reverse_proxy localhost:13841
}

handle /insurance/life/life-insurance-quotes {
   reverse_proxy localhost:13841
}

@result_pages path_regexp ^\/insurance\/[^/]+\/match\/results$

handle @result_pages {
    reverse_proxy localhost:13841
}

handle /insurance/account/create-password {
   reverse_proxy localhost:13841
}

handle /insurance/public/build* {
   reverse_proxy localhost:13841
}

handle /insurance/public/__webpack_hmr {
   reverse_proxy localhost:13841
}

Even if you repeatedly mention that domains should be left intact, unfortunately I’m not allowed to share the company I work for, so the localbluff and the whatever bit in localbluff.stage.whatever.biz and https://stage.whatever.biz are redacted.
If you cannot bear with that, unfortunately I’ll have to get along without your help.

2

It looks like the second half of the help template is missing. Can you please fill that out so we can help you? Otherwise we’ll just be guessing.

3

Edited, I hope that’s enough.
It actually works great, it’s just when i try to stop it.

4

Ok, thanks – I’ll investigate. Looks like @Mohammed90 already took a look and pinged me with some clues… interesting!

5

Did you splice anything out of your logs, or is it literally just those 2 lines and then the error + panic?

Additionally, is anything else happening prior to this, like reloading config or similar? I’ll need the full logs and unredacted please.

unfortunately I’m not allowed to share the company I work for, so the localbluff and the whatever bit in localbluff.stage.whatever.biz and https://stage.whatever.biz are redacted.

You company can pay for private support and I’ll be happy to work with you without having to share things publicly! Here’s our sponsorship plans, I’d recommend a Business or Business+ sponsorship: Sponsor @mholt on GitHub Sponsors · GitHub

Unfortunately I can seldom afford to help companies for free these days. (So I can get around to this eventually, but it might involve some guesswork and take a long time.)

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.