I did not see your post until now and it was already locked. I don’t know if you found a solution, but I noticed that your launchctl plist did not have keys to set the user/group of caddy when running. Maybe you deal with that a separate ways, but it can make for unintended consequences if permissions don’t match.
Try adding these keys to your plist and reload it in launchctl:
I saw that your file root was the web server root – so I think that is typically 755 for the _www:_www user. While I also saw that you have caddy running from sbin, I’m assuming you have it starting as root so you can bind privileged ports. If you leave it as root, that might cause hiccups on the python pipes. Maybe not, but it was a thought.
Also, out of my curiosity, how did you build caddy? did you have to chmod +s the binary executable? I seem to be unable to decipher when that is necessary in MacOS, so I thought I’d ask.
Cheers! I don’t know anything about the grpc protocol, but I know that not including those run as user _www:_www keys have given me headaches in the past.
I’ve added the keys you suggested to the launch plist, and confirmed that the caddy server was properly running as user _www after the change. That broke things for me at first, but I tracked that down to it no longer having access to the SSL certificates, since those were in directories/files owned by root. After changing the ownership there, caddy started working again. However, it had no impact on the issue I posted about.
As mentioned in the original note, caddy is working perfectly for both normal web traffic and even for gRPC traffic, but only when I don’t try to interleave the exchange of metadata and the actual gRPC request. If I have the client send both metadata and the regular request before trying to read either the response metadata or the response, Caddy’s forwarding of gRPC works great. However, as soon as I try to read the response metadata before writing the request, the client hangs, presumably because Caddy isn’t actually set up to forward the response metadata from the gRPC server it is forwarding the request to prior to reading the main gRPC request.
I can see from my test program that Caddy is forwarding the client’s metadata through and the server is receiving it and immediately responding with its own metadata. However, that response metadata doesn’t seem to be making it from Caddy back to the client.
Once Caddy has read the regular gRPC request, it correctly forwards both the server’s response metadata and the server’s gRPC response through without a problem. However, if the client blocks trying to read the server metadata before sending its gRPC request, it hangs.
All of this works perfectly fine if I bypass Caddy and have the gRPC client talk directly to the gRPC server.
Regarding building Caddy, I didn’t built it myself from source. I just downloaded the binary from the Caddy web site (version 2.3.0, which was the latest at the time I posted my original report here). I also didn’t chmod the binary to be setuid, and not doing so didn’t seem to be causing any problem (both when running as root and later when running as _www).
Thanks for the tip on the chmod. Glad to know it worked without. I think I had to do it because my old box has Server installed. I’m prepping a new box now and was looking.
Do you use macports? I’m switching to macports and away from that hot mess of homebrew. I just noticed that your paths match the macports schema.
Yeah - I have both MacPorts and homebrew installed here, but I only use homebrew for a handful of packages not available on MacPorts, whereas I have a couple hundred ports installed. MacPorts is working well for me.