Sequential challenges

Does this look right? I’m specifically wondering if the fact that everything looks to be sequential, is that correct? Referring to https://twitter.com/mholt6/status/1145839032317800448, @matt seems to think not?

[INFO] Wait for route53 [timeout: 2m0s, interval: 4s]
[INFO] [rs.show.domain.net] acme: Trying to solve DNS-01
[INFO] [rs.show.domain.net] acme: Checking DNS record propagation using [10.96.0.10:53]
[INFO] Wait for propagation [timeout: 2m0s, interval: 4s]
[INFO] [rs.show.domain.net] The server validated our request
[INFO] [rs.show.domain.net] acme: Cleaning DNS-01 challenge
[INFO] Wait for route53 [timeout: 2m0s, interval: 4s]
[INFO] [rs.show.domain.net] acme: Validations succeeded; requesting certificates
[INFO] [rs.show.domain.net] Server responded with a certificate.
[INFO] [rs.beta.domain.net] acme: Obtaining bundled SAN certificate
[INFO] [rs.beta.domain.net] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz/N3xZBRWg4CbSEDSLrCcJ5veAHSMIpPKTXUhV3b3_59c
[INFO] [rs.beta.domain.net] acme: Could not find solver for: tls-alpn-01
[INFO] [rs.beta.domain.net] acme: Could not find solver for: http-01
[INFO] [rs.beta.domain.net] acme: use dns-01 solver
[INFO] [rs.beta.domain.net] acme: Preparing to solve DNS-01
[INFO] Wait for route53 [timeout: 2m0s, interval: 4s]
[INFO] [rs.beta.domain.net] acme: Trying to solve DNS-01
[INFO] [rs.beta.domain.net] acme: Checking DNS record propagation using [10.96.0.10:53]
[INFO] Wait for propagation [timeout: 2m0s, interval: 4s]
[INFO] [rs.beta.domain.net] The server validated our request
[INFO] [rs.beta.domain.net] acme: Cleaning DNS-01 challenge
[INFO] Wait for route53 [timeout: 2m0s, interval: 4s]
[INFO] [rs.beta.domain.net] acme: Validations succeeded; requesting certificates
[INFO] [rs.beta.domain.net] Server responded with a certificate.
[INFO] [kue.rs.beta.domain.net] acme: Obtaining bundled SAN certificate
[INFO] [kue.rs.beta.domain.net] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz/6QeQQLvHaZRc1k0pu0a3UkkOcW3JIogMeM2SYPA302g
[INFO] [kue.rs.beta.domain.net] acme: Could not find solver for: tls-alpn-01
[INFO] [kue.rs.beta.domain.net] acme: Could not find solver for: http-01
[INFO] [kue.rs.beta.domain.net] acme: use dns-01 solver
[INFO] [kue.rs.beta.domain.net] acme: Preparing to solve DNS-01
[INFO] Wait for route53 [timeout: 2m0s, interval: 4s]
[INFO] [kue.rs.beta.domain.net] acme: Trying to solve DNS-01
[INFO] [kue.rs.beta.domain.net] acme: Checking DNS record propagation using [10.96.0.10:53]
[INFO] Wait for propagation [timeout: 2m0s, interval: 4s]
[INFO] Wait for propagation [timeout: 2m0s, interval: 4s]
[INFO] [kue.rs.beta.domain.net] The server validated our request
[INFO] [kue.rs.beta.domain.net] acme: Cleaning DNS-01 challenge
[INFO] Wait for route53 [timeout: 2m0s, interval: 4s]
[INFO] [kue.rs.beta.domain.net] acme: Validations succeeded; requesting certificates
[INFO] [kue.rs.beta.domain.net] Server responded with a certificate.

I’m not sure I understand what you mean. What is sequential and why should it not be?

I asked @matt via a Tweet (linked to above) if Caddy could possibly start serving requests before tidying up the DNS challenges. He said it should already work like that and to file a bug with the upstream project.

I thought I’d just double check here that my logs actually demonstrate that is the case. Everything looks to be sequential, i.e. request a DNS challenge for domain 1 then tidy up the DNS challenge for domain one then request a DNS challenge for domain 2 then tidy up the DNS challenge for domain two then start serving requests.

Do you also think Caddy is behaving incorrectly?

Ah, I thought you were asking if it cleaned up the TXT resource created during the DNS challenges, nothing about serving sites before, after, or during that.

At startup, Caddy ensures it can serve your sites the way you require; if it needs HTTPS and needs a certificate, then it obtains them before serving them.

Yeah, I was just wondering if there is room for optimisation of the process? Sometimes it can take >3m to retrieve just a couple of certs. I understand the long timeouts (i.e. 2m), but perhaps the tidy up portion could be deferred until all certificates are retrieved (or not) and the certs could be tidied up in parallel?

Just curious, why do you need it faster, since your server was not already running? There’s not much we can do to speed up the challenges, unless the CA and all networks in between get faster.

Matt, that is a good question. I suppose it doesn’t need to be faster.

I was just watching it happen in the logs and thought the server doesn’t really need to wait for DNS verification clean-up before starting to serve requests.

But you raise a good point.

You keep saying this but I still don’t know what you mean by this :thinking: because I still think you’re referring to cleaning up the TXT records it creates.

I am talking about cleaning up the TXT records it creates. I’m talking about cleaning them up AFTER it starts responding to requests, at the moment it cleans them up BEFORE.

Ah. Yes, that’s because to Caddy, obtaining certificates is a single function call, which returns after it is finished, not after it’s partially finished. There would have to be some major changes in lego to expose that level of detail, which I’m not sure we want to do. There’s no guarantee or reason to suppose that the cleanup is always going to take a long time relative to the time spent obtaining the certificate.

Okay, thanks for the explanation Matt.