Caddy still trying to hit let's encrypt even with local_certs enabled

1. My Caddy version (caddy version):

v2.0.0-beta.18 h1:N8CR8nO+n9Sk16CBx6ysVZWIZv0hgtU1J84st1bFWgI=

2. How I run Caddy:

a. System environment:

macOS Catalina v 10.15.3

/etc/hosts:


   1   │ ##
   2   │ # Host Database
   3   │ #
   4   │ # localhost is used to configure the loopback interface
   5   │ # when the system is booting.  Do not change this entry.
   6   │ ##
   7   │ 127.0.0.1        localhost sourcegraph.test
   8   │ 255.255.255.255  broadcasthost
   9   │ ::1              localhost
  10   │ # Added by Docker Desktop
  11   │ # To allow the same kube context to work on the host and the container:
  12   │ 127.0.0.1        kubernetes.docker.internal
  13   │ # End of section
  14   │ # Added by Geoffrey
  15   │ 127.0.0.1        ggilmore.nice

b. Command:

env SOURCEGRAPH_HTTPS_DOMAIN=sourcegraph.test SOURCEGRAPH_HTTPS_PORT=3443 caddy run --config=dev/Caddyfile

c. Service/unit/compose file:

N/A

d. My complete Caddyfile or JSON config:

{
	local_certs
}


https://{$SOURCEGRAPH_HTTPS_DOMAIN}:{$SOURCEGRAPH_HTTPS_PORT} {
	reverse_proxy localhost:3080
    tls internal
}

3. The problem I’m having:

I’m trying to use caddy as a suitable https proxy for local development of Sourcegraph. We have the app serving over http at localhost:3080, and we’d like to have caddy reverse proxy that over https://sourcegraph.test:3443 (sourcegraph.test is a fake domain that we put in our own /etc/hosts for local testing purposes).

When I try to run Caddy with this configuration, I get this error message:

4. Error messages and/or full log output:

2020/03/20 19:43:35.113	INFO	using provided configuration	{"config_file": "dev/Caddyfile", "config_adapter": ""}
2020/03/20 19:43:35.115	INFO	admin	admin endpoint started	{"address": "localhost:2019", "enforce_origin": false, "origins": ["localhost:2019"]}
2020/03/20 12:43:35 [INFO][cache:0xc000953e00] Started certificate maintenance routine
2020/03/20 19:43:35.122	INFO	http	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2020/03/20 19:43:35.232	INFO	pki.ca.local	root certificate is already trusted by system	{"path": "storage:pki/authorities/local/root.crt"}
2020/03/20 19:43:35.233	INFO	tls	cleaned up storage units
2020/03/20 19:43:35.233	INFO	http	enabling automatic TLS certificate management	{"domains": ["sourcegraph.test"]}
2020/03/20 19:43:35.234	INFO	autosaved config	{"file": "/Users/ggilmore/Library/Application Support/Caddy/autosave.json"}
2020/03/20 19:43:35.234	INFO	serving initial configuration
2020/03/20 12:43:35 [INFO][sourcegraph.test] Obtain certificate; acquiring lock...
2020/03/20 12:43:35 [INFO][sourcegraph.test] Obtain: Lock acquired; proceeding...
2020/03/20 12:43:35 [INFO][sourcegraph.test] Waiting on rate limiter...
2020/03/20 12:43:35 [INFO][sourcegraph.test] Done waiting
2020/03/20 12:43:35 [INFO] [sourcegraph.test] acme: Obtaining bundled SAN certificate given a CSR
2020/03/20 12:43:35 [ERROR] acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rejectedIdentifier :: Error creating new order :: Cannot issue for "sourcegraph.test": Domain name does not end with a valid public suffix (TLD), url:  (challenge=tls-alpn-01 remaining=[http-01])
2020/03/20 12:43:37 [INFO] [sourcegraph.test] acme: Obtaining bundled SAN certificate given a CSR
2020/03/20 12:43:37 [ERROR] acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rejectedIdentifier :: Error creating new order :: Cannot issue for "sourcegraph.test": Domain name does not end with a valid public suffix (TLD), url:  (challenge=http-01 remaining=[])
2020/03/20 12:43:39 [ERROR] attempt 1: [sourcegraph.test] Obtain: [sourcegraph.test] acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rejectedIdentifier :: Error creating new order :: Cannot issue for "sourcegraph.test": Domain name does not end with a valid public suffix (TLD), url:  - retrying in 1m0s (4.374944404s/720h0m0s elapsed)...
2020/03/20 12:43:48 http: TLS handshake error from 127.0.0.1:55664: no certificate available for 'sourcegraph.test'
2020/03/20 12:43:48 http: TLS handshake error from 127.0.0.1:55665: no certificate available for 'sourcegraph.test'
2020/03/20 12:44:39 [INFO] [sourcegraph.test] acme: Obtaining bundled SAN certificate given a CSR
2020/03/20 12:44:40 [ERROR] acme: error: 400 :: POST :: https://acme-staging-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rejectedIdentifier :: Error creating new order :: Cannot issue for "sourcegraph.test": Domain name does not end with a valid public suffix (TLD), url:  (challenge=tls-alpn-01 remaining=[http-01])
2020/03/20 12:44:42 [INFO] [sourcegraph.test] acme: Obtaining bundled SAN certificate given a CSR
2020/03/20 12:44:42 [ERROR] acme: error: 400 :: POST :: https://acme-staging-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rejectedIdentifier :: Error creating new order :: Cannot issue for "sourcegraph.test": Domain name does not end with a valid public suffix (TLD), url:  (challenge=http-01 remaining=[])
2020/03/20 12:44:44 [ERROR] attempt 2: [sourcegraph.test] Obtain: [sourcegraph.test] acme: error: 400 :: POST :: https://acme-staging-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rejectedIdentifier :: Error creating new order :: Cannot issue for "sourcegraph.test": Domain name does not end with a valid public suffix (TLD), url:  - retrying in 2m0s (1m9.040441757s/720h0m0s elapsed)...
^[[A^[[A^C2020/03/20 19:46:42.220	INFO	shutting down	{"signal": "SIGINT"}
2020/03/20 12:46:42 http: TLS handshake error from 127.0.0.1:55666: read tcp 127.0.0.1:3443->127.0.0.1:55666: use of closed network connection
2020/03/20 12:46:42 [INFO][sourcegraph.test] Obtain: Releasing lock
2020/03/20 12:46:42 [INFO][cache:0xc000953e00] Stopped certificate maintenance routine
2020/03/20 19:46:42.221	INFO	shutdown done	{"signal": "SIGINT"}

What’s strange is that Caddy still seems to be trying to contact Lets Encrypt even though I enabled local_certs in the global options.

Both Chrome and Firefox complain about SSL errors whenever I try to hit https://sourcegraph.test:3443

5. What I already tried:

I’ve tried changing the fake domain to sourcegraph.dev (a real TLD), but I still get SSL errors and Caddy still seems to be trying to hit Let’s Encrypt even though I enabled local_certs in the global options.

Caddy Logs

2020/03/20 19:47:23.655	INFO	using provided configuration	{"config_file": "dev/Caddyfile", "config_adapter": ""}
2020/03/20 19:47:23.657	INFO	admin	admin endpoint started	{"address": "localhost:2019", "enforce_origin": false, "origins": ["localhost:2019"]}
2020/03/20 12:47:23 [INFO][cache:0xc000477770] Started certificate maintenance routine
2020/03/20 19:47:23.664	INFO	http	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2020/03/20 19:47:23.776	INFO	pki.ca.local	root certificate is already trusted by system	{"path": "storage:pki/authorities/local/root.crt"}
2020/03/20 19:47:23.777	INFO	tls	cleaned up storage units
2020/03/20 19:47:23.777	INFO	http	enabling automatic TLS certificate management	{"domains": ["sourcegraph.dev"]}
2020/03/20 19:47:23.777	INFO	autosaved config	{"file": "/Users/ggilmore/Library/Application Support/Caddy/autosave.json"}
2020/03/20 19:47:23.777	INFO	serving initial configuration
2020/03/20 12:47:23 [INFO][sourcegraph.dev] Obtain certificate; acquiring lock...
2020/03/20 12:47:23 [INFO][sourcegraph.dev] Obtain: Lock acquired; proceeding...
2020/03/20 12:47:23 [INFO][sourcegraph.dev] Waiting on rate limiter...
2020/03/20 12:47:23 [INFO][sourcegraph.dev] Done waiting
2020/03/20 12:47:23 [INFO] [sourcegraph.dev] acme: Obtaining bundled SAN certificate given a CSR
2020/03/20 12:47:24 [INFO] [sourcegraph.dev] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/3470502029
2020/03/20 12:47:24 [INFO] [sourcegraph.dev] acme: Could not find solver for: tls-alpn-01
2020/03/20 12:47:24 [INFO] [sourcegraph.dev] acme: use http-01 solver
2020/03/20 12:47:24 [INFO] [sourcegraph.dev] acme: Trying to solve HTTP-01

6. Links to relevant resources:

Here is the PR that I’m working on: https://github.com/sourcegraph/sourcegraph/pull/9050/files

2 Likes

Also, I’m not sure if this is a bug or misconfiguration. LMK if this belongs on a github issue

Thanks for the report!

This is a bug… and an oversight on my part when I redesigned automation policy logic a few days ago.

If you want to open a GitHub issue, that would help me track its progress, since I’ll be working on it today and Monday probably!

Thanks for looking into this. Filed here: https://github.com/caddyserver/caddy/issues/3164

2 Likes

Alrighty, I believe I got this fixed up at https://github.com/caddyserver/caddy/commit/d692d503a3d327d54c82bceab48bb1de07bb3c3d – please take a look and give it a try! Thank you again.

2 Likes