Isn’t binding ports but not logging error either

I have been using caddy for two years. I started with the downloadable executable file for Mac, and I have always run Macserver. This set up has worked very well until this past week when I installed the security update 2019-003 for High Sierra

Mac server has always listened on *:80 and *:443
Using bind to the IP4 address has always allowed caddy to bind and listen and serve pages with glee for several vhosts - even with Mac server running on *:443, caddy would bind x.x.x.x:443 and all was well

I start caddy with launchctl plist in the global daemons and have no errors. Just caddy isn’t listening on the IP ports anymore. :face_with_head_bandage:

Caddy is running no problem, but it’s not serving anything and lsof doesn’t show it bound anywhere. System logs don’t have any reports of error. Caddy logs have no error.
If I check on lsof, the * ports are all httpd but no caddy to be seen.

I’m at wits end … if caddy wasn’t allowed to bind x.x.x.x, then there’d be an error in the logs, right?

I’m not sure what other info I can provide… this has worked for years and now it doesn’t but there is no log errors or even hints in the logs. My sysfoo is unable to conjure any other ideas…help!!

Is there anything in your Console.app perhaps? Caddy will write a log to its usual process logs if it is running and has permissions to; otherwise it must be somewhere else in the system, like OS permissions or the launch daemon.

1 Like

Thanks Matt – good idea, but I have the launchd plist route standard out and error to known files in the working folders I have for caddy. All I see is “Activating Privacy Features…” in the console output.

Because it isn’t binding, and caddy can’t listen, if I leave it running long enough, it will eventually throw some errors from trying to renew certs that it failed ALPN acme challenge method… which of course it would because it can’t react to the challenges that are supposed to come on the ports it can’t listen on…

I’ve looked in system log, console, servers logs, and the caddy output logs (that are routed as mentioned above) – there’s nothing. silence.

I’ve looked for entries complaining that caddy tried to listen… nothing. I would have expected either a security log saying it was denied (and why) and I would expect caddy to have at least one line in the caddy.error.log that would say something about denied a port and an echo of the error return. but instead, nobody says anything in any log!

I’ve repeatedly run “sudo lsof” with parameters to look at port 443 and 80. I can see the server app httpd sitting on *:80. Last week, if I ran that command, I would have seen httpd on *:443 and caddy on x.x.x.x:443…

There’s nothing – logs show a normal start up for caddy. caddy is running, has a PID, I can probe it and it’s alive.

it’s just silently not listening and I can’t figure out why.

Is there anything in your $CADDYPATH/locks folder? If so, delete and try again.

Hmmmm. I’ll look and try. Thank you!

I finally have my first clue. I tried deleted the contents of the locks folder. That didn’t work.

So, I thought about a totally clean slate approach. I switched caddy to an entirely new working folder. It started but then launchctl exits (Error 1). I did get one entry in the console during a start-stop parade of the service. console said:

launchctl dumpstate failed w/ exit code 113: Could not find service "com.caddyserver.web" in domain for system

So far, I have no idea what that means. My launch key is not that name, and I don’t have that name in my plist anywhere, so I’m not sure from where it is getting “com.caddyserver.web”

So now I can get it to start, but it fails pretty quickly:
Unable to deactivated authorizations: https://acme-v02.api.letsencrypt.org/acme/authz/Qr3hC_7ATKVi7wfNKoP7OntbqFBjHq1OB9fIJ_QzR

That appears in the caddy.log std output
the caddy error log has this:

2019/06/18 23:49:02 [domain.com] failed to obtain certificate: acme: Error -> One or more domains had a problem:

acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Cannot negotiate ALPN protocol "acme-tls/1" for tls-alpn-01 challenge, url:

Just to uhh, throw it out there… You aren’t behind Cloudflare’s orange cloud, are you?

Good question. One of the domains for one site does have cloudflare DNS. Not caching, just points to the server box.

The other three domains do not. The domain that is throwing the error above is not at all connected to the big orange cloud.

Anything else in the way? What’s the network path look like?

Not able to negotiate ALPN protocol heavily implies there’s some kind of TLS termination happening.

Ok. So that brings me back to the Mac not allowing caddy to listen on ports (bind).

Caddy is chmod +s and owned by root/wheel (not a good thing, I know, but it’s the only way to get Mac BSD to bind a privileged port)… that has worked for two years.

Something must have changed in Mac to cause this. I haven’t been able to find it though.

FYI. Here’s a discussion from two years ago when I started with caddy. I reached back out to that author to see if he’s had trouble too. He gets into the issue of Mac not having setcap for port bindings and running caddy as root

Taking another direction here. I don’t think the problem is that the Mac is preventing caddy. In the past, caddy would give notice its error log that the port was unavailable. That isn’t happening now.

Rather, caddy doesn’t give any errors except for these problems with obtaining cert packages and ALPN. That makes me think it is getting mixed up with other letsencrypt software (is that even possible?).

now I’m mixed up entirely. So, I decided to run the caddy command from the terminal in userspace with sudo just to see what I’d get. I added the -disable-tls-alpn-challenge flag that you suggested to another user with this error on this thread, @WhitestrakeTLS handshake errors out of nowhere

That got me somewhere entirely different. STDOUT had this to say:
Activating privacy features... 2019/06/24 19:30:00 [INFO] [example.com] acme: Obtaining bundled SAN certificate 2019/06/24 19:30:00 [INFO] [example.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz/giE8r629uZA8TQjpG_EXAMPLE 2019/06/24 19:30:00 [INFO] [example.com] acme: Could not find solver for: tls-alpn-01 2019/06/24 19:30:00 [INFO] [example.com] acme: use http-01 solver 2019/06/24 19:30:00 [INFO] [example.com] acme: Trying to solve HTTP-01 2019/06/24 19:38:40 [example.com] failed to obtain certificate: acme: Error -> One or more domains had a problem: [example.com] the server didn't respond to our request

I went to the URL and saw the challenges:
{ "identifier": { "type": "dns", "value": "example.com" }, "status": "pending", "expires": "2019-07-02T00:30:00Z", "challenges": [ { "type": "http-01", "status": "pending",

Then, I tried again, and I got this as the last output:
failed to obtain certificate: acme: Error -> One or more domains had a problem: [example.com] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Invalid response from https://example.com/.well-known/acme-challenge/EzX3lfYwbVlRlBi2mJd-EXAMPLE [199.195.144.46]: "<!DOCTYPE html>\r\n<html lang=\"en-US\">\r\n<head>\r\n\t\t<!--[if lt IE 9]>\r\n\t<script src=\"https://example.com/wp-content/themes/wp-theme-0319/j", url:

That tell me that the backup apache is serving up the wordpress, and wordpress is giving a 404 – it’s the wordpress theme on example.com’s html document coming up as a 404 - but I think it is actually denying the filetype (no extension) under the htaccess rules/wordpress security rewrite rules.

So, I guess I need to either figure out how to get htaccess to pass this request, to figure out what is blocking TLS discussions between caddy and acme. It’s just the weirdest thing to me that caddy never had trouble before for two years. update to 11.5 on homebrew, and this. Now, running 1.0 in a clean HOME for caddy and TLS won’t work here either – but is does for homebrew certbot.

I was not able to find any ACL entries on the caddy executable, and the binary has the following permissions (which should easily enable it to bind to a privileged port:
-rwsr-sr-x@ 1 root staff 20903376 Jun 7 14:44 caddy

I’m totally stumped.

Is there a way to force caddy to use IP ports only?

Was there any changes to caddy around 0.11.5 that would change the way caddy asks to listen on ports? I’m not smart enough to figure it out from the git repo but I’ve been looking.

I need a way to tell caddy to only specify ports on one IP address, not any wildcards.

EDIT: The -host directive set for the IP4 should do it. I’ll find out tonight.

Setting host as IP4 did not solve it. Caddy isn’t listening but there are no errors.

(I haven’t been keeping up with this conversation, but) Try using the bind directive: https://caddyserver.com/docs/bind

Thanks @matt. All the vhosts have bind with the IP4 host address in them. That’s what is driving me up a wall! This has worked fine for two years through countless caddy updates (I was using the homebrew build)

Now, all of a sudden it doesn’t want to work.