Enabling Auto Renewal of certificates using TLS ALPN challenge

Apoorva · October 24, 2024, 12:19pm

1. The problem I’m having:

I am trying to setup Caddy to auto renew our certificates from Let’s Encrypt. Currently they are Digi Certs certificates, I am doing a POC using Caddy to automate the certificate renewal through Let’s Encrypt. My servers are complex Broadworks server(COTS product from Cisco) , Since our DNS provider doesn’t provide APIs, we are not able to use DNS Challenge. and The HTTP redirection needs custom coding to handle as we have one main FQDN and sub FQDN under that and hostnames behind these sub FQDNs. Hence we opted for Caddy instead of certbot to use TLS APLN challenge.

2. Error messages and/or full log output:

[root@khk26dvf2 caddy]# caddy run
2024/10/23 14:51:35.639 INFO    using adjacent Caddyfile
2024/10/23 14:51:35.639 INFO    adapted config to JSON  {"adapter": "caddyfile"}
2024/10/23 14:51:35.640 WARN    Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies    {"adapter": "caddyfile", "file": "Caddyfile", "line": 11}
2024/10/23 14:51:35.640 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2024/10/23 14:51:35.640 INFO    http.auto_https automatic HTTP->HTTPS redirects are disabled    {"server_name": "srv0"}
2024/10/23 14:51:35.640 INFO    http    enabling HTTP/3 listener        {"addr": ":443"}
2024/10/23 14:51:35.640 INFO    http.log        server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2024/10/23 14:51:35.640 INFO    http    enabling automatic TLS certificate management   {"domains": ["xx.ims.prod.tdc.dk"]}
2024/10/23 14:51:35.641 INFO    autosaved config (load with --resume flag)      {"file": "/root/.config/caddy/autosave.json"}
2024/10/23 14:51:35.641 INFO    serving initial configuration
2024/10/23 14:51:35.641 INFO    tls.obtain      acquiring lock  {"identifier": "xx.ims.prod.tdc.dk"}
2024/10/23 14:51:35.641 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc0005b4c00"}
2024/10/23 14:51:35.643 INFO    tls     storage cleaning happened too recently; skipping for now        {"storage": "FileStorage:/root/.local/share/caddy", "instance": "e8b7a8df-3ed6-48d1-8876-3e309e086bd6", "try_again": "2024/10/24 14:51:35.643", "try_again_in": 86399.999999694}
2024/10/23 14:51:35.644 INFO    tls     finished cleaning storage units
2024/10/23 14:51:35.644 INFO    tls.obtain      lock acquired   {"identifier": "xx.ims.prod.tdc.dk"}
2024/10/23 14:51:35.644 INFO    tls.obtain      obtaining certificate   {"identifier": "xx.ims.prod.tdc.dk"}
2024/10/23 14:51:35.644 INFO    tls.issuance.acme       waiting on internal rate limiter        {"identifiers": ["xx.ims.prod.tdc.dk"], "ca": "https://acme-staging-v02.api.letsencrypt.org/directory", "account": "xx@tdc.dk"}
2024/10/23 14:51:35.644 INFO    tls.issuance.acme       done waiting on internal rate limiter   {"identifiers": ["xxx.ims.prod.tdc.dk"], "ca": "https://acme-staging-v02.api.letsencrypt.org/directory", "account": "xx@tdc.dk"}
2024/10/23 14:51:35.644 INFO    tls.issuance.acme       using ACME account      {"account_id": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/168355333", "account_contact": ["mailto:xx@tdc.dk"]}
2024/10/23 14:51:36.518 INFO    tls.issuance.acme.acme_client   trying to solve challenge       {"identifier": "xx.ims.prod.tdc.dk", "challenge_type": "tls-alpn-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2024/10/23 14:51:37.458 ERROR   tls.issuance.acme.acme_client   challenge failed        {"identifier": "xx.ims.prod.tdc.dk", "challenge_type": "tls-alpn-01", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge", "instance": "", "subproblems": []}}
2024/10/23 14:51:37.458 ERROR   tls.issuance.acme.acme_client   validating authorization        {"identifier": "xx.ims.prod.tdc.dk", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/168355333/19931912543", "attempt": 1, "max_attempts": 3}
2024/10/23 14:51:38.774 INFO    tls.issuance.acme.acme_client   trying to solve challenge       {"identifier": "xx.ims.prod.tdc.dk", "challenge_type": "http-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2024/10/23 14:51:39.709 ERROR   tls.issuance.acme.acme_client   challenge failed        {"identifier": "xx.ims.prod.tdc.dk", "challenge_type": "http-01", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "xx.xx.xx.xx: Invalid response from http://webex.ims.prod.tdc.dk/.well-known/acme-challenge/n3pb-6cd9MXRFseUxhstTJL6oekrGSiFcCyBQi_b51Y: 404", "instance": "", "subproblems": []}}
2024/10/23 14:51:39.709 ERROR   tls.issuance.acme.acme_client   validating authorization        {"identifier": "xx.ims.prod.tdc.dk", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "87.51.89.53: Invalid response from http://xx.ims.prod.tdc.dk/.well-known/acme-challenge/n3pb-6cd9MXRFseUxhstTJL6oekrGSiFcCyBQi_b51Y: 404", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/168355333/19931913163", "attempt": 2, "max_attempts": 3}
2024/10/23 14:51:39.709 ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "xx.ims.prod.tdc.dk", "issuer": "acme-staging-v02.api.letsencrypt.org-directory", "error": "HTTP 403 urn:ietf:params:acme:error:unauthorized - 87.51.89.53: Invalid response from http://xx.ims.prod.tdc.dk/.well-known/acme-challenge/n3pb-6cd9MXRFseUxhstTJL6oekrGSiFcCyBQi_b51Y: 404"}
2024/10/23 14:51:39.709 ERROR   tls.obtain      will retry      {"error": "[xx.ims.prod.tdc.dk] Obtain: [xx.ims.prod.tdc.dk] solving challenge: xx.ims.prod.tdc.dk: [xx.ims.prod.tdc.dk] authorization failed: HTTP 403 urn:ietf:params:acme:error:unauthorized - 87.51.89.53: Invalid response from http://xx.ims.prod.tdc.dk/.well-known/acme-challenge/n3pb-6cd9MXRFseUxhstTJL6oekrGSiFcCyBQi_b51Y: 404 (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 1, "retrying_in": 60, "elapsed": 4.065562122, "max_duration": 2592000}
2024/10/23 14:52:39.713 INFO    tls.obtain      obtaining certificate   {"identifier": "xx.ims.prod.tdc.dk"}
2024/10/23 14:52:39.714 INFO    tls.issuance.acme       using ACME account      {"account_id": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/168355333", "account_contact": ["mailto:xx@tdc.dk"]}
2024/10/23 14:52:40.170 INFO    tls.issuance.acme.acme_client   trying to solve challenge       {"identifier": "xx.ims.prod.tdc.dk", "challenge_type": "http-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2024/10/23 14:52:41.108 ERROR   tls.issuance.acme.acme_client   challenge failed        {"identifier": "xx.ims.prod.tdc.dk", "challenge_type": "http-01", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "87.51.88.54: Invalid response from http://xx.ims.prod.tdc.dk/.well-known/acme-challenge/uzHkRY-ydQPcQu1EQ9sh0ye_thdzOS-vtD4UO2aju9A: 404", "instance": "", "subproblems": []}}
2024/10/23 14:52:41.108 ERROR   tls.issuance.acme.acme_client   validating authorization        {"identifier": "xx.ims.prod.tdc.dk", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "87.51.88.54: Invalid response from http://xx.ims.prod.tdc.dk/.well-known/acme-challenge/uzHkRY-ydQPcQu1EQ9sh0ye_thdzOS-vtD4UO2aju9A: 404", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/168355333/19931925903", "attempt": 1, "max_attempts": 3}

[root@khk26dxxvf2 caddy]# openssl s_client -alpn acme-tls/1 -connect xx.ims.prod.tdc.dk:443
CONNECTED(00000003)
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root G2
verify return:1
depth=1 C = US, O = DigiCert Inc, CN = DigiCert Global G2 TLS RSA SHA256 2020 CA1
verify return:1
depth=0 C = DK, L = K\C3\B8benhavn SV, O = TDC Holding A/S, CN = xx.ims.prod.tdc.dk
verify return:1
---
Certificate chain
 0 s:C = DK, L = K\C3\B8benhavn SV, O = TDC Holding A/S, CN = webex.ims.prod.tdc.dk
   i:C = US, O = DigiCert Inc, CN = DigiCert Global G2 TLS RSA SHA256 2020 CA1
 1 s:C = US, O = DigiCert Inc, CN = DigiCert Global G2 TLS RSA SHA256 2020 CA1
   i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root G2
---
Server certificate
-----BEGIN CERTIFICATE-----

EQRGsuub5g8wggI5BgNVHREEggIwMIICLIIVd2ViZXguaW1zLnByb2QudGRjLmRr
ghZwdy1jdGkuaW1zLnByb2QudGRjLmRrghxwd2ViZXgteHNwMTAuaW1zLnByb2Qu
dGRjLmRrghxwd2ViZXgteHNwMjAuaW1zLnByb2QudGRjLmRrghZwd2ViZXguaW1z
LnByb2QudGRjLmRrghp3LWN0aS14c3AxLmltcy5wcm9kLnRkYy5ka4Iady1jdGkt
eHNwMi5pbXMucHJvZC50ZGMuZGuCGnctY3RpLXhzcDMuaW1zLnByb2QudGRjLmRr
ghp3LWN0aS14c3A0Lmltcy5wcm9kLnRkYy5ka4IVdy1jdGkuaW1zLnByb2QudGRj
LmRrght3ZWJleC14c3AxMC5pbXMucHJvZC50ZGMuZGuCG3dlYmV4LXhzcDExLmlt
cy5wcm9kLnRkYy5ka4Ibd2ViZXgteHNwMTIuaW1zLnByb2QudGRjLmRrght3ZWJl
eC14c3AxMy5pbXMucHJvZC50ZGMuZGuCG3dlYmV4LXhzcDE0Lmltcy5wcm9kLnRk
Yy5ka4Ibd2ViZXgteHNwMjAuaW1zLnByb2QudGRjLmRrght3ZWJleC14c3AyMS5p
bXMucHJvZC50ZGMuZGuCG3dlYmV4LXhzcDIyLmltcy5wcm9kLnRkYy5ka4Ibd2Vi
ZXgteHNwMjMuaW1zLnByb2QudGRjLmRrght3ZWJleC14c3AyNC5pbXMucHJvZC50
ZGMuZGswPgYDVR0gBDcwNTAzBgZngQwBAgIwKTAnBggrBgEFBQcCARYbaHR0cDov
L3d3dy5kaWdpY2VydC5jb20vQ1BTMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAU
BggrBgEFBQcDAQYIKwYBBQUHAwIwgZ8GA1UdHwSBlzCBlDBIoEagRIZCaHR0cDov
L2NybDMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0R2xvYmFsRzJUTFNSU0FTSEEyNTYy
MDIwQ0ExLTEuY3JsMEigRqBEhkJodHRwOi8vY3JsNC5kaWdpY2VydC5jb20vRGln
d
BwEBBHsweTAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuZGlnaWNlcnQuY29tMFEG
CCsGAQUFBzAChkVodHRwOi8vY2FjZXJ0cy5kaWdpY2VydC5jb20vRGlnaUNlcnRH
bG9iYWxHMlRMU1JTQVNIQTI1NjIwMjBDQTEtMS5jcnQwDAYDVR0TAQH/BAIwADCC
AX4GCisGAQQB1nkCBAIEggFuBIIBagFoAHYATnWjJ1yaEMM4W2zU3z9S6x3w4I4b
jWnAsfpksWKaOd8AAAGPyVF0egAABAMARzBFAiEA+howqML2VcQLCg1v1R/UIERM
GRgUDfMgNarl55x1UTECIGP29lH8hSXbb4E9/p9NeXxxb6PBJXCgCmyIMu2HtQmP
AHYAfVkeEuF4KnscYWd8Xv340IdcFKBOlZ65Ay/ZDowuebgAAAGPyVF0ZQAABAMA
RzBFAiAy7RrkhOVcQ+hHdDBnPGKh0oivuiHj9VxvxXJ1oBkO0QIhAI4n0LqoJIW5
86V/AZdeOmftcFqQx9ulYInstFX9mN4hAHYA5tIxY0B3jMEQQQbXcbnOwdJA9paE
hvu6hzId/R43jlAAAAGPyVF0iwAABAMARzBFAiAlnysUWuypVTNktDDORKs5nS6O
h9MUTbjczbzEX/i4IQIhAJzETVbnD32O2G8LBmSAD/50NglWy0zX7aE6aL6MA8JJ
MA0GCSqGSIb3DQEBCwUAA4IBAQBoyIT4JDPGcNnVSJqlzphhNRyBA+WRbVQ4KDJ3
TMcHYppJ6bm3AA/l8cZtGB5k06+D1LDABBVLN0An2lDnTlXvvSPHybXgeCHep3PG
yIx1dNjvN/vQYPUHnCcYfOFskTpn0pUq5djugWGdZ+phI1T5KrijzU9XNsdvkB7I
Z2Sdti/muDY/pRLYeXernD1LW9SKShwNCszNy9NDw7DCgIJtnXTXBYGM/bw3KouO
Qq9TnWmozzqXNHKxuZ/DKGNhCN6VdgcEiI/ZeyadC4S8YOX0QJuRWk5H5L75AJzf
o7BEoq0nX7694rCFnoiCWcvJkEaPeQQzJvvq1DGAykcK1ppK
-----END CERTIFICATE-----
subject=C = DK, L = K\C3\B8benhavn SV, O = TDC Holding A/S, CN = xx.ims.prod.tdc.dk

issuer=C = US, O = DigiCert Inc, CN = DigiCert Global G2 TLS RSA SHA256 2020 CA1

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4005 bytes and written 471 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: BF0AD79B18C06319962EA0E4149335CD3DF8E3AF21925A4280398D546B4605D8
    Session-ID-ctx:
    Master-Key: EFC34388F28F01FCE61F6985EDB3EF7BFB61B6D0B08005B5D0E4C9440B2959F2CDD50F29F7531B2048DF556C5C52DE98
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1729694631
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes

3. Caddy version:

[root@khk26dvxxf2 caddy]# caddy version
v2.8.4

4. How I installed and ran Caddy:

a. System environment:

VM - Linux based
Operating System: Red Hat Enterprise Linux 8.10 (Ootpa)
CPE OS Name: cpe:/o:redhat:enterprise_linux:8::baseos
Kernel: Linux 4.18.0-513.24.1.el8_9.x86_64
Architecture: x86-64

b. Command:

Installed using dnf
  dnf install 'dnf-command(copr)'
  dnf copr enable @caddy/caddy
  dnf install caddy
  which caddy
PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.
caddy run

d. My complete Caddy config:

{
    email xx@tdc.dk
    auto_https disable_redirects
    acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}

xx.ims.prod.tdc.dk:443 {
    root * /usr/local/broadworks/apps/active/WebContainer/conf/
    file_server
    tls {
        alpn acme-tls/1
    }
}

# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile
~

francislavoie · October 24, 2024, 3:10pm

Actually, you still could if you point a CNAME for the _acme-challenge subdomain to a different domain which you do have APIs for, doing what’s called “challenge delegation”. Then LE will follow the CNAME when looking for the TXT record challenge proof.

You don’t need this, Caddy handles HTTP and TLS-ALPN challenges out of the box automatically.

Apoorva · October 25, 2024, 1:16pm

Hi @francislavoie , thank you for your response. Would you be able to share more information on “challenge delegation”. Does this require any modifications at DNS level , because that is something which is totally out of our control.

francislavoie · October 25, 2024, 4:09pm

See tls (Caddyfile directive) — Caddy Documentation and Challenge Types - Let's Encrypt where it talks about CNAME.

Basically you would add a CNAME only once (not via API) to your main domain, then you can have a different domain on another DNS provider which has an API which can be used to prove you have control of the main domain (since the CNAME proves that you “trust” the other domain).

Apoorva · October 28, 2024, 4:08pm

Hi Francis, Let me check once with my DNS team if they can create a subdomain with the requirements, thanks for the info , I will get back to you once I have some lead , so watch this space

system · November 27, 2024, 4:08pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.