Https certificate hang

1. My Caddy version (v2):

2. How I run Caddy:

I am just trying to run a simple reverse proxy to a backend service.
sudo caddy reverse-proxy --from domain --to localhost:8000

a. System environment:

backend service on port:8000 is a docker container. The whole system is on ubuntu 16.04 LTS

3. The problem I’m having:

Caddy hangs at “The server validated our request” when attempting to provision a SSL certificate.

4. Error messages and/or full log output:

2020/03/05 04:29:09.691 WARN admin admin endpoint disabled
2020/03/05 04:29:09.692 INFO http server is listening only on the HTTPS port but has no TLS connection polici
es; adding one to enable TLS {“server_name”: “proxy”, “https_port”: 443}
2020/03/05 04:29:09.692 INFO http enabling automatic HTTP->HTTPS redirects {“server_name”: “proxy”}
2020/03/05 04:29:09.694 INFO http enabling automatic TLS certificate management {“domains”: [“domain”]}
2020/03/05 04:29:09 [INFO][cache:0xc0001a7270] Started certificate maintenance routine
2020/03/05 04:29:09.711 INFO tls cleaned up storage units
2020/03/05 04:29:09.711 INFO autosaved config {“file”: “/home/internetofh2o/.config/caddy/autosave.json”}
Caddy 2 proxying from domain to localhost:8000
2020/03/05 04:29:10 [INFO][domain] Obtain certificate
2020/03/05 04:29:10 [INFO][domain] Obtain: Waiting on rate limiter…
2020/03/05 04:29:10 [INFO][domain] Obtain: Done waiting
2020/03/05 04:29:10 [INFO] [domain] acme: Obtaining bundled SAN certificate
2020/03/05 04:29:10 [INFO] [domain] AuthURL: https://acme-v02.api.letsencrypt.org/acme/aut
hz-v3/3167809221
2020/03/05 04:29:10 [INFO] [domain] acme: Could not find solver for: tls-alpn-01
2020/03/05 04:29:10 [INFO] [domain] acme: use http-01 solver
2020/03/05 04:29:10 [INFO] [domain] acme: Trying to solve HTTP-01
2020/03/05 04:29:11 [INFO][domain] Served key authentication (HTTP challenge)
2020/03/05 04:29:11 [INFO][domain] Served key authentication (HTTP challenge)
2020/03/05 04:29:11 [INFO][domain] Served key authentication (HTTP challenge)
2020/03/05 04:29:11 [INFO][domain] Served key authentication (HTTP challenge)
2020/03/05 04:29:15 [INFO] [domain] The server validated our request

5. What I already tried:

I have tried a variety of different apex domain names that I own. I have tried each domain name on a fresh virtual machine, building caddy from source and starting the docker container from scratch.

Is your server accessible on both ports 80 and 443? Both are needed to successfully complete the HTTP challenge. Check your port forwarding and/or firewall rules.

Yes, ports 80, 443, and 8000 are all open

Could you tell us which build of v2 you’re using?

Also, the rest of the help template is very useful to diagnosing these kinds of issues.

Specifically, 2 b. Command, 2 c. Service/unit/compose file, and 2 d. My complete Caddyfile or JSON config.

Version: path: github.com/caddyserver/caddy/v2/cmd/caddy
main: github.com/caddyserver/caddy/v2 (devel)

Command: sudo caddy reverse-proxy --from frost-import.internetofwater.us --to localhost:8000

for reference, that domain name is correctly pointing to the IP address of the machine.

Compose file - N/A

Caddyfile - N/A, just using the CLI command

1 Like

So I assume you built from source? Would be helpful to know which commit exactly you built from if so.

Otherwise, do you mind trying the linux binary from here: https://dev.azure.com/mholt-dev/Caddy/_build/results?buildId=913&view=artifacts&type=publishedArtifacts this is the latest build on the v2 branch, built from the CI job.

1 Like

I’ll try the binary. The binaries that are linked on the github release pages never worked for me, which is why I built from source in the first place. The version I built from source (which I jsut got from the basic git clone command) would have been 2.0 beta 13

Could you elaborate? What about them didn’t work? Did you forget to chmod +x the file before trying to use it (to give it executable permissions)?

The same thing that happened with this binary jsut now. I proceed through all of the installation instructions, but at the last step sudo systemctl start caddy, when I systemctl status caddy, I get

Mar 05 05:26:45 test systemd[1]: Started Caddy Web Server.
Mar 05 05:26:45 test systemd[1]: caddy.service: Main process exited, code=exited, status=217/USER
Mar 05 05:26:45 test systemd[1]: caddy.service: Unit entered failed state.
Mar 05 05:26:45 test systemd[1]: caddy.service: Failed with result ‘exit-code’.

I wonder if this is similar to V2: Why wasn't cert available after reload + restart, only after reboot?

1 Like

This is telling you that there was a problem with the user your service is trying to run as. The user might not exist or might be an invalid string.

ok. the problem was some stray whitespace when I set up the users. So caddy runs as a service now.

However I get the same problem…

2020/03/05 05:44:14 [INFO] [frost-import.internetofwater.us] acme: Could not find solver for: tls-alpn-01
2020/03/05 05:44:14 [INFO] [frost-import.internetofwater.us] acme: use http-01 solver
2020/03/05 05:44:14 [INFO] [frost-import.internetofwater.us] acme: Trying to solve HTTP-01
2020/03/05 05:44:14 [INFO][frost-import.internetofwater.us] Served key authentication (HTTP challenge)
2020/03/05 05:44:15 [INFO][frost-import.internetofwater.us] Served key authentication (HTTP challenge)
2020/03/05 05:44:15 [INFO][frost-import.internetofwater.us] Served key authentication (HTTP challenge)
2020/03/05 05:44:20 [INFO] [frost-import.internetofwater.us] The server validated our request

Okay… @matt I think this might be a lego issue? That last message comes from there: https://sourcegraph.com/search?q=repo:^github\.com/go-acme/lego%24+The+server+validated+our+request&patternType=literal

3 Likes

Yes, seems that it’s probably a bug in lego or in the backoff lib: https://sourcegraph.com/github.com/go-acme/lego@b5d99c7b1405d23d71cca8bd4d865379e704474a/-/blob/challenge/resolver/solver_manager.go#L135 - if that line returned, we’d see another log message.

Bug filed upstream: https://github.com/go-acme/lego/issues/1075

1 Like

@ksonda Would you be able to upgrade to the latest on the v2 branch (newer than beta 18, will go out with beta 19)? I think I’ve fixed a bug today that was at least related to the hanging, if not the hanging itself. Did you notice that this only happens with the HTTP challenge? If so, it might very well have been fixed now… if not, meh, we’ll see. Let me know after you upgrade if it happens or doesn’t happen again!

After 20 trials I was unable to replicate the issue (which I have experienced before myself since your report).