Issue with https connection

jono_lo · November 16, 2021, 1:18pm

1. Caddy version (`caddy version`):

$caddy version
v2.4.6 h1:HGkGICFGvyrodcqOOclHKfvJC0qTU7vny/7FhYp9hNw=

2. How I run Caddy:

I run caddy in my home network, from a local computer with local ip 192.168.1.100.
I have a domain name (my_domain.com below) that is linked to my public ip.
Both ports 443 and 80 are forwarded to 192.168.1.100 on my router.

a. System environment:

Debian 11.
No docker

b. Command:

$ cd folder_with_Caddyfile
$ caddy run

c. Service/unit/compose file:

NA

d. My complete Caddyfile or JSON config:

https://my_domain.com, http://my_domain.com {

respond "Hello, world!"

}

3. The problem I’m having:

http access seems fine from a computer on the same LAN:

$ curl http://my_domain.com
Hello, world

https access doesn’t work from a computer on the same LAN:

$ curl -v https://my_domain.com
*   Trying my_public_ip:443...
* Connected to my_domain.com (my_public_ip) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /home/jonolo/anaconda3/ssl/cacert.pem
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, internal error (592):
* error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
* Closing connection 0
curl: (35) error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error

4. Error messages and/or full log output:

$ caddy run
2021/11/16 12:34:02.476	INFO	using adjacent Caddyfile
2021/11/16 12:34:02.477	WARN	input is not formatted with 'caddy fmt'	{"adapter": "caddyfile", "file": "Caddyfile", "line": 2}
2021/11/16 12:34:02.477	INFO	admin	admin endpoint started	{"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["127.0.0.1:2019", "localhost:2019", "[::1]:2019"]}
2021/11/16 12:34:02.478	INFO	http	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2021/11/16 12:34:02.478	INFO	http	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2021/11/16 12:34:02.478	INFO	http	server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server	{"server_name": "srv1", "http_port": 80}
2021/11/16 12:34:02.478	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0xc000377b20"}
2021/11/16 12:34:02.478	INFO	tls	cleaning storage unit	{"description": "FileStorage:/root/.local/share/caddy"}
2021/11/16 12:34:02.478	INFO	http	enabling automatic TLS certificate management	{"domains": ["my_domain.com"]}
2021/11/16 12:34:02.479	INFO	autosaved config (load with --resume flag)	{"file": "/root/.config/caddy/autosave.json"}
2021/11/16 12:34:02.479	INFO	serving initial configuration
2021/11/16 12:34:02.479	INFO	tls	finished cleaning storage units
2021/11/16 12:34:02.479	INFO	tls.obtain	acquiring lock	{"identifier": "my_domain.com"}
2021/11/16 12:34:02.510	INFO	tls.obtain	lock acquired	{"identifier": "my_domain.com"}
2021/11/16 12:34:02.512	INFO	tls.issuance.acme	waiting on internal rate limiter	{"identifiers": ["my_domain.com"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2021/11/16 12:34:02.512	INFO	tls.issuance.acme	done waiting on internal rate limiter	{"identifiers": ["my_domain.com"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2021/11/16 12:34:03.994	INFO	tls.issuance.acme.acme_client	trying to solve challenge	{"identifier": "my_domain.com", "challenge_type": "http-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2021/11/16 12:34:15.054	ERROR	tls.issuance.acme.acme_client	challenge failed	{"identifier": "my_domain.com", "challenge_type": "http-01", "problem": {"type": "urn:ietf:params:acme:error:connection", "title": "", "detail": "Fetching http://my_domain.com/.well-known/acme-challenge/3n5RaavT_vgwrZjy0pcTZfSykpJ_QsyEGJ64R_m_1U0: Timeout during connect (likely firewall problem)", "instance": "", "subproblems": []}}
2021/11/16 12:34:15.054	ERROR	tls.issuance.acme.acme_client	validating authorization	{"identifier": "my_domain.com", "problem": {"type": "urn:ietf:params:acme:error:connection", "title": "", "detail": "Fetching http://my_domain.com/.well-known/acme-challenge/3n5RaavT_vgwrZjy0pcTZfSykpJ_QsyEGJ64R_m_1U0: Timeout during connect (likely firewall problem)", "instance": "", "subproblems": []}, "order": "https://acme-v02.api.letsencrypt.org/acme/order/281589040/40108454990", "attempt": 1, "max_attempts": 3}
2021/11/16 12:34:16.306	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "my_domain.com", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/"}
2021/11/16 12:34:16.307	WARN	tls.issuance.zerossl	missing email address for ZeroSSL; it is strongly recommended to set one for next time
2021/11/16 12:34:17.653	INFO	tls.issuance.zerossl	generated EAB credentials	{"key_id": "IAE1ZIMYjo4uLCcaG7Ngeg"}
2021/11/16 12:34:19.541	INFO	tls.issuance.acme	waiting on internal rate limiter	{"identifiers": ["my_domain.com"], "ca": "https://acme.zerossl.com/v2/DV90", "account": ""}
2021/11/16 12:34:19.541	INFO	tls.issuance.acme	done waiting on internal rate limiter	{"identifiers": ["my_domain.com"], "ca": "https://acme.zerossl.com/v2/DV90", "account": ""}
2021/11/16 12:34:20.231	INFO	tls.issuance.acme.acme_client	trying to solve challenge	{"identifier": "my_domain.com", "challenge_type": "http-01", "ca": "https://acme.zerossl.com/v2/DV90"}
2021/11/16 12:39:22.015	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "my_domain.com", "issuer": "acme.zerossl.com-v2-DV90", "error": "[my_domain.com] solving challenges: [my_domain.com] authorization took too long (order=https://acme.zerossl.com/v2/DV90/order/0Tf9y6_XybNx-K7erM-nRQ) (ca=https://acme.zerossl.com/v2/DV90)"}
2021/11/16 12:39:22.015	ERROR	tls.obtain	will retry	{"error": "[my_domain.com] Obtain: [my_domain.com] solving challenges: [my_domain.com] authorization took too long (order=https://acme.zerossl.com/v2/DV90/order/0Tf9y6_XybNx-K7erM-nRQ) (ca=https://acme.zerossl.com/v2/DV90)", "attempt": 1, "retrying_in": 60, "elapsed": 319.505131193, "max_duration": 2592000}

5. What I already tried:

Check that DNS is set properly:

curl "https://cloudflare-dns.com/dns-query?name=my_domain.com&type=A" \
  -H "accept: application/dns-json"
{"Status":0,"TC":false,"RD":true,"RA":true,"AD":false,"CD":false,"Question":[{"name":"my_domain.com","type":1}],"Answer":[{"name":"my_domain.com","type":1,"TTL":3600,"data":"my_public_ip"}]}%

my_domain.com is linked to my_public_ip, I think everything looks fine here.

My ISP was blocking 80 and 443 ports by default. I ‘unblocked’ these from their portal a few days ago. However both http and https access don’t work from the outside, I’m not sure if this issue is related.

# From remote machine
curl http://my_domain.com
curl: (28) Failed to connect to my_domain.com.au port 80: Operation timed out
curl https://my_domain.com
curl: (7) Failed to connect to my_domain.com port 443: Operation timed out

francislavoie · November 16, 2021, 5:16pm

That’s because you don’t have a valid certificate yet to complete the TLS handshake, as the logs show:

This means Let’s Encrypt couldn’t reach your server to verify the HTTP challenge, so it couldn’t complete issuance.

There’s clearly still a networking issue between Let’s Encrypt and your Caddy server preventing them from connecting.

jono_lo · November 18, 2021, 2:49am

Thanks a lot for you help.

Yes I think there is a network issue there.
According to my ISP, both 80 and 443 ports should be opened.
The port forwarding rules are properly set up on my router as well. I’m not sure where else I can look.

Do you have any suggestion on how to pinpoint the source of the problem ?

Whitestrake · November 18, 2021, 7:12am

There’s a few steps between the public internet and your Caddy server:

Resolve DNS: ensure your domain publicly resolves to your current public IP address
Connect to your external IP (router): ensure your ISP allows access to your external ports
Forward packets to your Caddy host: ensure that the port forwarding is functioning correctly
Accept packets on your Caddy host: ensure the host’s firewall is configured to allow connections

I won’t cover checking step 1 - I assume you’re familiar with verifying DNS resolution - but it’s important to double check it.

We run into a problem with steps 2 and 3, though. Each step must be checked and verified separately, but these steps are difficult to test independently, and failure on either step looks the same: a connection timeout. If you have the router forwarding properly but your ISP lied, it’s difficult to tell which; if your ISP is telling the truth but your router is not properly forwarding packets, it’s hard to tell.

Depending on your networking hardware/capabilities, you could look into performing a packet capture on the WAN port to look for incoming connection attempts on ports 80/443 whenever you attempt ACME challenges; the absence of these packets, in combination with verifying DNS is correct, points a finger at the ISP for not allowing packets to reach you. On the other hand, if those packets are present but they’re still not reaching Caddy, it means the issue is further down the chain.

Step 4 can be tested pretty easily; craft a CURL to check against the Caddy host over the private network and see if you get any response.

system · December 16, 2021, 1:19pm

This topic was automatically closed after 30 days. New replies are no longer allowed.