Thanks for the help so far. Originally I was doing both. So what I did was , I tested doing the curl commands in the lxc container and then again on a remote computer. The behavior was the same.
But, given your response, I changed the DNS to be towards my pihole (I realized I did this in my old gatewayserver, it was using the pihole as it’s DNS server). What this means is my home router is not the DNS. It’s just the WAN IP that port-forwards 80 and 443 to my Caddy now. When I did that, I did make some progress?
I was able to run it against sites such as SSL Shopper, and SSLlabs. The caddy file showed their outputs
{"level":"debug","ts":"2023/11/26 20:45:42","logger":"events","msg":"event","name":"tls_get_certificate","id":"a202e609-5e9a-4e7d-a02a-3d7fd458c335","origin":"tls","data":{"client_hello":{"CipherSuites":[136,150,156,157,158,159,255,49154,49155,49156,49157,49159,49160,49161,49162,49164,49165,49166,49167,49169,49170,49171,49172,49187,49188,49191,49195,49196,49199,49200,49201,49202],"ServerName":"rs2.12egc.com","SupportedCurves":[14,13,25,11,12,24,9,10,22,23,8,6,7,20,21,4,5,18,19,1,2,3,15,16,17,29],"SupportedPoints":"AA==","SignatureSchemes":[1537,1538,1539,1281,1282,1283,1025,1026,1027,769,770,771,513,514,515],"SupportedProtos":null,"SupportedVersions":[771,770,769],"Conn":{}}}}
{"level":"debug","ts":"2023/11/26 20:45:42","logger":"tls.handshake","msg":"choosing certificate","identifier":"rs2.12egc.com","num_choices":1}
{"level":"debug","ts":"2023/11/26 20:45:42","logger":"tls.handshake","msg":"default certificate selection results","identifier":"rs2.12egc.com","subjects":["rs2.12egc.com"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"c0e0ed30fca585c0635ac258e5f61087efe88a8c383743d70bbfc46e5fed1a80"}
{"level":"debug","ts":"2023/11/26 20:45:42","logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"64.41.200.106","remote_port":"39908","subjects":["rs2.12egc.com"],"managed":true,"expiration":"2024/02/23 14:37:56","hash":"c0e0ed30fca585c0635ac258e5f61087efe88a8c383743d70bbfc46e5fed1a80"}
{"level":"debug","ts":"2023/11/26 20:45:46","logger":"events","msg":"event","name":"tls_get_certificate","id":"5c0e2f90-6a40-4a4c-8c69-a3accfeef453","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"rs2.12egc.com","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"Conn":{}}}}
{"level":"debug","ts":"2023/11/26 20:45:46","logger":"tls.handshake","msg":"choosing certificate","identifier":"rs2.12egc.com","num_choices":1}
{"level":"debug","ts":"2023/11/26 20:45:46","logger":"tls.handshake","msg":"default certificate selection results","identifier":"rs2.12egc.com","subjects":["rs2.12egc.com"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"c0e0ed30fca585c0635ac258e5f61087efe88a8c383743d70bbfc46e5fed1a80"}
{"level":"debug","ts":"2023/11/26 20:45:46","logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"108.48.66.209","remote_port":"45158","subjects":["rs2.12egc.com"],"managed":true,"expiration":"2024/02/23 14:37:56","hash":"c0e0ed30fca585c0635ac258e5f61087efe88a8c383743d70bbfc46e5fed1a80"}
{"level":"debug","ts":"2023/11/26 20:45:50","logger":"http.stdlib","msg":"http: TLS handshake error from 64.41.200.106:39908: EOF"}
{"level":"debug","ts":"2023/11/26 20:45:50","logger":"http.stdlib","msg":"http: TLS handshake error from 64.41.200.106:36136: EOF"}
But the EOF is still there. That said when I re-ran the curl cmd from a remote computer I got this instead, it does seem to take a long time to get to this new failure
curl -v https://rs2.12egc.com/
* Trying 173.66.250.81:443...
* Connected to rs2.12egc.com (173.66.250.81) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=rs2.12egc.com
* start date: Nov 25 19:37:56 2023 GMT
* expire date: Feb 23 19:37:55 2024 GMT
* subjectAltName: host "rs2.12egc.com" matched cert's "rs2.12egc.com"
* issuer: C=US; O=Let's Encrypt; CN=R3
* SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x56337b273e90)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host: rs2.12egc.com
> user-agent: curl/7.81.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* OpenSSL SSL_read: Connection reset by peer, errno 104
* Failed receiving HTTP2 data
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* OpenSSL SSL_write: Broken pipe, errno 32
* Failed sending HTTP2 data
* Connection #0 to host rs2.12egc.com left intact
curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104
When I re-ran it inside my Caddy LXC container, I got a different response, though
curl -v https://rs2.12egc.com
* Trying 173.66.250.81:443...
* Connected to rs2.12egc.com (173.66.250.81) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* SSL connection timeout
* Closing connection 0
curl: (28) SSL connection timeout
EDIT: Perhaps another clue?
I really didn’t change much besides now my caddy has a respond <h1>Hello, World</h1> and now I see that but, inconsistently
curl -v --trace-time https://rs2.12egc.com
21:51:41.834375 * Trying 173.66.250.81:443...
21:51:41.854598 * Connected to rs2.12egc.com (173.66.250.81) port 443 (#0)
21:51:41.855998 * ALPN, offering h2
21:51:41.856019 * ALPN, offering http/1.1
21:51:41.879614 * CAfile: /etc/ssl/certs/ca-certificates.crt
21:51:41.879624 * CApath: /etc/ssl/certs
21:51:41.879705 * TLSv1.0 (OUT), TLS header, Certificate Status (22):
21:51:41.879713 * TLSv1.3 (OUT), TLS handshake, Client hello (1):
21:51:41.894234 * TLSv1.2 (IN), TLS header, Certificate Status (22):
21:51:41.894259 * TLSv1.3 (IN), TLS handshake, Server hello (2):
21:51:41.894432 * TLSv1.2 (IN), TLS header, Finished (20):
21:51:41.894455 * TLSv1.2 (IN), TLS header, Supplemental data (23):
21:51:41.894473 * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
21:51:41.894491 * TLSv1.2 (IN), TLS header, Supplemental data (23):
21:51:41.894523 * TLSv1.3 (IN), TLS handshake, Certificate (11):
21:51:41.895292 * TLSv1.2 (IN), TLS header, Supplemental data (23):
21:51:41.895309 * TLSv1.3 (IN), TLS handshake, CERT verify (15):
21:51:41.895375 * TLSv1.2 (IN), TLS header, Supplemental data (23):
21:51:41.895389 * TLSv1.3 (IN), TLS handshake, Finished (20):
21:51:41.895409 * TLSv1.2 (OUT), TLS header, Finished (20):
21:51:41.895418 * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
21:51:41.895437 * TLSv1.2 (OUT), TLS header, Supplemental data (23):
21:51:41.895449 * TLSv1.3 (OUT), TLS handshake, Finished (20):
21:51:41.895477 * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
21:51:41.895487 * ALPN, server accepted to use h2
21:51:41.895505 * Server certificate:
21:51:41.895515 * subject: CN=rs2.12egc.com
21:51:41.895525 * start date: Nov 25 19:37:56 2023 GMT
21:51:41.895534 * expire date: Feb 23 19:37:55 2024 GMT
21:51:41.895544 * subjectAltName: host "rs2.12egc.com" matched cert's "rs2.12egc.com"
21:51:41.895553 * issuer: C=US; O=Let's Encrypt; CN=R3
21:51:41.895561 * SSL certificate verify ok.
21:51:41.895586 * Using HTTP2, server supports multiplexing
21:51:41.895595 * Connection state changed (HTTP/2 confirmed)
21:51:41.895611 * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
21:51:41.895624 * TLSv1.2 (OUT), TLS header, Supplemental data (23):
21:51:41.895647 * TLSv1.2 (OUT), TLS header, Supplemental data (23):
21:51:41.895664 * TLSv1.2 (OUT), TLS header, Supplemental data (23):
21:51:41.895682 * Using Stream ID: 1 (easy handle 0x565474cd4e90)
21:51:41.895696 * TLSv1.2 (OUT), TLS header, Supplemental data (23):
21:51:41.895709 > GET / HTTP/2
21:51:41.895709 > Host: rs2.12egc.com
21:51:41.895709 > user-agent: curl/7.81.0
21:51:41.895709 > accept: */*
21:51:41.895709 >
21:51:41.895766 * TLSv1.2 (IN), TLS header, Supplemental data (23):
21:51:41.895792 * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
21:51:41.905540 * TLSv1.2 (IN), TLS header, Supplemental data (23):
21:51:41.905558 * Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
21:51:41.905567 * TLSv1.2 (OUT), TLS header, Supplemental data (23):
21:51:41.905584 * TLSv1.2 (IN), TLS header, Supplemental data (23):
21:51:41.905723 * TLSv1.2 (IN), TLS header, Supplemental data (23):
21:51:41.909848 * TLSv1.2 (IN), TLS header, Supplemental data (23):
21:51:41.909879 < HTTP/2 200
21:51:41.909889 < alt-svc: h3=":443"; ma=2592000
21:51:41.909903 < content-type: text/plain; charset=utf-8
21:51:41.909912 < server: Caddy
21:51:41.909918 < content-length: 22
21:51:41.909931 < date: Mon, 27 Nov 2023 02:51:41 GMT
21:51:41.909943 <
21:51:41.909955 * TLSv1.2 (IN), TLS header, Supplemental data (23):
21:51:41.909977 * Connection #0 to host rs2.12egc.com left intact
<h1>Hello, World!</h1>
I ran it twice in two different terminals on a remote computer. In One terminal hangs, while the other does it instantly. But when I re-ran it, it hangs. Thus, it is not a consistent result. The same for my Firefox browser
Edit #2: As of this edit, it’s working suddenly. I don’t know what fixed it. I figured maybe because I was running caddy and my old gateway, there was some fight in the network(?) since I have a reverse proxy rule to go to my old gateway if I use a special subdomain. I turned that off and tried again, but it did not work.
I figured I’d reboot Caddy lxc container, just gave it more cores (4 instead of 2), and noticed it was working suddenly. I turned back on my old gateway to see if that caused all the hanging, and nope, it still works even with that on. I’m not sure how this got resolved. I’ll update again if it breaks magically in the next 24hrs. If someone has an idea why it could be working or has a test I could do to figure out the truth, I’d love to help.
Edit #3 Not sure how to strike through above. It seems it’s an order of operations which was causing the issue. If my Old gateway was on before my caddy was on, TLS handshake gets dropped. If I turn off my old gateway and do not reboot my caddy lxc container, the entire 443 will time out, no TLS even happens. At that point, if I reboot my caddy LXC container while leaving the old gateway off, it works perfectly, no problems. This persists if I turn back on my old gateway. Thus it seems to be as follows, If the old gate way was up before caddy was up, TLS will not work, otherwise if Caddy was up before the old gateway was up, TLS works and the reverse proxy work. This is definitely on my side then!
Not sure if this is for a different thread, and it might be , but, my old gateway has certificates for another domain that I was going to migrate - in phases- to my new provider. Yet it seems it is responding for the TLS traffic or somehow its intercepting or having packets redirecting to it instead of going back to the client straight from Caddy. So, can Caddy be the handler of certificates and dynamic DNS, and be placed in-between a Home Router and a virtualized Gateway service that manages a LAN (I’m using Nethserver) ?
), which i really enjoyed as I could list sub domains on the go, reload the caddy file and it would automatically add that to my list of subdomains and update the IP address in the record.
thanks!