Cannot get ACME certificate

wotno · August 30, 2024, 9:16pm

1. The problem I’m having:

Caddy is unable to complete the automatic lets-encrypt certification. In the error log it also says that :80 and :443 are already in use, but I checked and they are NOT. Futhermore, both are allowed in the firewall correctly.

2. Error messages and/or full log output:

$ /usr/bin/caddy run --config /etc/caddy/Caddyfile 
2024/08/30 20:50:52.987	INFO	using config from file	{"file": "/etc/caddy/Caddyfile"}
2024/08/30 20:50:52.991	INFO	adapted config to JSON	{"adapter": "caddyfile"}
2024/08/30 20:50:52.994	INFO	admin	admin endpoint started	{"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2024/08/30 20:50:52.995	INFO	http.auto_https	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2024/08/30 20:50:52.995	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0xc00066a380"}
2024/08/30 20:50:52.995	INFO	http.auto_https	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2024/08/30 20:50:52.996	INFO	http	enabling HTTP/3 listener	{"addr": ":443"}
2024/08/30 20:50:52.996	INFO	http.log	server running	{"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2024/08/30 20:50:52.996	INFO	http.log	server running	{"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2024/08/30 20:50:52.997	INFO	autosaved config (load with --resume flag)	{"file": "/root/.config/caddy/autosave.json"}
2024/08/30 20:50:52.997	INFO	serving initial configuration
2024/08/30 20:50:52.997	INFO	tls.obtain	acquiring lock	{"identifier": "foo.bar.org"}
2024/08/30 20:50:53.005	INFO	tls	storage cleaning happened too recently; skipping for now	{"storage": "FileStorage:/root/.local/share/caddy", "try_again": "2024/08/31 20:50:53.005", "try_again_in": 86399.999998845}
2024/08/30 20:50:53.005	INFO	tls	finished cleaning storage units
2024/08/30 20:50:53.006	INFO	tls.obtain	lock acquired	{"identifier": "foo.bar.org"}
2024/08/30 20:50:53.006	INFO	tls.obtain	obtaining certificate	{"identifier": "foo.bar.org"}
2024/08/30 20:50:54.199	INFO	tls.issuance.acme	waiting on internal rate limiter	{"identifiers": ["foo.bar.org"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "email@eggs.com"}
2024/08/30 20:50:54.199	INFO	tls.issuance.acme	done waiting on internal rate limiter	{"identifiers": ["foo.bar.org"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "email@eggs.com"}
2024/08/30 20:50:54.199	INFO	tls.issuance.acme	using ACME account	{"account_id": "https://acme-v02.api.letsencrypt.org/acme/acct/redacted", "account_contact": ["mailto:email@eggs.com"]}
2024/08/30 20:50:54.814	INFO	tls.issuance.acme.acme_client	trying to solve challenge	{"identifier": "foo.bar.org", "challenge_type": "tls-alpn-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2024/08/30 20:50:54.816	INFO	[WARNING] listen tcp :443: bind: address already in use - be sure to set the ACMEIssuer.ListenHost field; assuming conflicting listener is correctly configured and continuing
2024/08/30 20:51:05.533	ERROR	tls.issuance.acme.acme_client	challenge failed	{"identifier": "foo.bar.org", "challenge_type": "tls-alpn-01", "problem": {"type": "urn:ietf:params:acme:error:connection", "title": "", "detail": "red.act.ed.ip: Timeout during connect (likely firewall problem)", "instance": "", "subproblems": []}}
2024/08/30 20:51:05.541	ERROR	tls.issuance.acme.acme_client	validating authorization	{"identifier": "foo.bar.org", "problem": {"type": "urn:ietf:params:acme:error:connection", "title": "", "detail": "red.act.ed.ip: Timeout during connect (likely firewall problem)", "instance": "", "subproblems": []}, "order": "https://acme-v02.api.letsencrypt.org/acme/order/redacted/redacted", "attempt": 1, "max_attempts": 3}
2024/08/30 20:51:07.166	INFO	tls.issuance.acme.acme_client	trying to solve challenge	{"identifier": "foo.bar.org", "challenge_type": "http-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2024/08/30 20:51:07.168	INFO	[WARNING] listen tcp :80: bind: address already in use - be sure to set the ACMEIssuer.ListenHost field; assuming conflicting listener is correctly configured and continuing
2024/08/30 20:51:17.941	ERROR	tls.issuance.acme.acme_client	challenge failed	{"identifier": "foo.bar.org", "challenge_type": "http-01", "problem": {"type": "urn:ietf:params:acme:error:connection", "title": "", "detail": "red.act.ed.ip: Fetching http://foo.bar.org/.well-known/acme-challenge/redacted: Timeout during connect (likely firewall problem)", "instance": "", "subproblems": []}}
2024/08/30 20:51:17.941	ERROR	tls.issuance.acme.acme_client	validating authorization	{"identifier": "foo.bar.org", "problem": {"type": "urn:ietf:params:acme:error:connection", "title": "", "detail": "red.act.ed.ip: Fetching http://foo.bar.org/.well-known/acme-challenge/redacted: Timeout during connect (likely firewall problem)", "instance": "", "subproblems": []}, "order": "https://acme-v02.api.letsencrypt.org/acme/order/redacted/redacted", "attempt": 2, "max_attempts": 3}
2024/08/30 20:51:17.941	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "foo.bar.org", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 400 urn:ietf:params:acme:error:connection - red.act.ed.ip: Fetching http://foo.bar.org/.well-known/acme-challenge/redacted: Timeout during connect (likely firewall problem)"}
2024/08/30 20:51:19.467	INFO	http	generated EAB credentials	{"key_id": "redacted"}
2024/08/30 20:51:22.066	INFO	tls.issuance.acme	waiting on internal rate limiter	{"identifiers": ["foo.bar.org"], "ca": "https://acme.zerossl.com/v2/DV90", "account": "email@eggs.com"}
2024/08/30 20:51:22.067	INFO	tls.issuance.acme	done waiting on internal rate limiter	{"identifiers": ["foo.bar.org"], "ca": "https://acme.zerossl.com/v2/DV90", "account": "email@eggs.com"}
2024/08/30 20:51:22.067	INFO	tls.issuance.acme	using ACME account	{"account_id": "https://acme.zerossl.com/v2/DV90/account/redacted", "account_contact": ["mailto:email@eggs.com"]}
2024/08/30 20:51:23.234	INFO	tls.issuance.acme.acme_client	trying to solve challenge	{"identifier": "foo.bar.org", "challenge_type": "http-01", "ca": "https://acme.zerossl.com/v2/DV90"}
2024/08/30 20:51:23.234	INFO	[WARNING] listen tcp :80: bind: address already in use - be sure to set the ACMEIssuer.ListenHost field; assuming conflicting listener is correctly configured and continuing
^C2024/08/30 20:51:49.327	INFO	shutting down	{"signal": "SIGINT"}
2024/08/30 20:51:49.328	WARN	exiting; byeee!! 👋	{"signal": "SIGINT"}
2024/08/30 20:51:49.328	INFO	http	servers shutting down with eternal grace period
2024/08/30 20:51:49.329	INFO	admin	stopped previous server	{"address": "localhost:2019"}
2024/08/30 20:51:49.329	INFO	shutdown complete	{"signal": "SIGINT", "exit_code": 0}

3. Caddy version:

v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy:

go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest
xcaddy build

a. System environment:

OS - Ubuntu
Architecture - x86_64
Command - /usr/bin/caddy run --config /etc/caddy/Caddyfile

b. Command:

/usr/bin/caddy run --config /etc/caddy/Caddyfile

c. My complete Caddy config:

:443, foo.bar.org
tls email@foobar.com
route {
        reverse_proxy https://google.com {
                header_up Host {upstream_hostport}
        }
}

Mikhail_Chudinov · August 30, 2024, 10:07pm

How do you check it?
Did you check at all server ip’s, including localhost ?

francislavoie · August 31, 2024, 5:27am

Make sure you don’t have two instances of Caddy running. Make sure you run Caddy as a systemd service. Don’t use caddy run directly, follow these instructions: Keep Caddy Running — Caddy Documentation

Change your config to this:

foo.bar.org {
	tls email@foobar.com
        reverse_proxy https://google.com {
                header_up Host {upstream_hostport}
        }
}

I’m not sure why you’re proxying to Google, I assume you’re just doing that for example purposes to start.

Let’s Encrypt isn’t able to connect to your server. Make sure DNS is correct, and that your firewall/router has ports 80 and 443 open and forwarded to your server.

wotno · August 31, 2024, 5:36am

I checked with these commands

netstat -lnp | grep :80
netstat -lnp | grep :443
ss -lnp 'sport = :80'
ss -lnp 'sport = :443'

wotno · August 31, 2024, 5:41am

I was using caddy run because it has easy access to logs. Same thing with systemd, which also produces the same error messages.

I also changed my config to the one you gave, but still the errors prevail

Both ports 80/443 are open. Nothing is also listening on these ports, and DNS is correct.

Mikhail_Chudinov · September 2, 2024, 2:50pm

Try to run Caddy from the root, just for the test.
May be your user does not have rights to use privileged ports.

caleb · September 2, 2024, 4:38pm

Are you sure 80 and 443 are actually getting to your server? Being open and listening doesn’t mean the requests are getting to your server. I just ran in to this same error when I was trying to cutover some domains to a new server and was hoping to get caddy running with the new config, but it couldn’t get the SSL cert because it would try to do the LE verification opening port 80 but since my DNS was pointing to the old server still, it threw the above error. Switched DNS to point to my new server and reloaded caddy and it was able to get the cert and reload fine.

wotno · September 2, 2024, 9:31pm

Thanks for the replies. I ran nmap and noticed these ports were showing up as filtered. It seems like somehow, along the way, the iptable rules got mangled after years of (ab)use, so I backed up the iptable rules and did a full reset. Then I allowed the ports I wanted and now it’s working again.

system · October 2, 2024, 9:32pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.