Incorrect TXT record with Namecheap

Help
1

1. The problem I’m having:

I am trying to get a new certificate issued for domain mngcogo.com.au. This is not working as I keep getting LE responses of “Incorrect TXT record” even though caddy added the acme challenge record in the first place.

We are already using domain mng.net.au which is correctly obtaining certificates from Let’s Encrypt. We recently purchased the new domain mngcogo.com.au

I’m having extreme difficulties trying to get a new certificate to be issued for mngcogo.com.au. When restarting caddy service, it creates a new acme challenge record in Namecheap DNS. I wait for propagation. Once propagation has occurred, I restart caddy so that it can ask Lets Encrypt for a certificate. But then I get log lines that it has an issue with older TXT records.

I am also finding that every time I restart caddy, it keeps adding a new TXT record, in which case, Let’s Encrypt keeps responding back with “Incorrect TXT record” for the previous TXT record before the new one that got created after restarting. I find myself having to delete that previous record and leaving the most recent one, but now I seem to be in a catch 22 where every time I restart caddy to ask for a new certificate, it keeps adding a new record, causing LE to find the incorrect old record and not seeming to validate on the most recent one.

2. Error messages and/or full log output:

Below are logs when running sudo journalctl -u caddy --no-pager | grep mngcogo.com.au

Oct 30 02:57:26 caddy-rproxy caddy[51357]: {"level":"error","ts":1730257046.8069344,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"*.mngcogo.com.au","challenge_type":"dns-01","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Incorrect TXT record \"9qLuN125KImIlFrr_jDJz8cO9tVIal78X9IVx6kj4Vs\" found at _acme-challenge.mngcogo.com.au","instance":"","subproblems":[]}}
Oct 30 02:57:26 caddy-rproxy caddy[51357]: {"level":"error","ts":1730257046.8075159,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"*.mngcogo.com.au","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Incorrect TXT record \"9qLuN125KImIlFrr_jDJz8cO9tVIal78X9IVx6kj4Vs\" found at _acme-challenge.mngcogo.com.au","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/1911189476/318294744857","attempt":1,"max_attempts":3}
Oct 30 02:57:26 caddy-rproxy caddy[51357]: {"level":"error","ts":1730257046.8075702,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"*.mngcogo.com.au","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"9qLuN125KImIlFrr_jDJz8cO9tVIal78X9IVx6kj4Vs\" found at _acme-challenge.mngcogo.com.au"}
Oct 30 02:57:26 caddy-rproxy caddy[51357]: {"level":"error","ts":1730257046.8076227,"logger":"tls.obtain","msg":"will retry","error":"[*.mngcogo.com.au] Obtain: [*.mngcogo.com.au] solving challenge: *.mngcogo.com.au: [*.mngcogo.com.au] authorization failed: HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"9qLuN125KImIlFrr_jDJz8cO9tVIal78X9IVx6kj4Vs\" found at _acme-challenge.mngcogo.com.au (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":6.984283985,"max_duration":2592000}

3. Caddy version:

v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy:

I installed by following the precise steps in Install — Caddy Documentation for Ubuntu

a. System environment:

Caddy was installed on Ubuntu v 24.04 LTS

b. Command:

sudo systemctl start caddy
sudo systemctl restart caddy

c. Service/unit/compose file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
Environment="NAMECHEAP_API_KEY=[REDACTED]"

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

{
        debug
}

*.mng.net.au {
        tls {
                dns namecheap {
                        api_key {env.NAMECHEAP_API_KEY}
                        user {env.NAMECHEAP_USER}
                        api_endpoint https://api.namecheap.com/xml.response
                        client_ip {env.CADDY_IP}
                }
        }

        @geoserver host geoserver.mng.net.au
        reverse_proxy @geoserver https://geoserver.mng.net.au {
                transport http {
                        tls_insecure_skip_verify
                }
        }

        #@backpack host backpack.mng.net.au
        #reverse_proxy @backpack https://backpack.mng.net.au {
        #    transport http {
        #        tls_insecure_skip_verify
        #    }
        #}

        @cogo host cogo.mng.net.au
        reverse_proxy @cogo https://cogo.mng.net.au {
                transport http {
                        tls_insecure_skip_verify
                }
        }

        #    @api host api.mngcogo.com.au
        #    reverse_proxy @api https://api.mngcogo.com.au {
        #        transport http {
        #            tls_insecure_skip_verify
        #        }
        #    }


        # Add more host matches as needed
        # Example:
        # @anotherapp host anotherapp.mng.net.au
        # reverse_proxy @anotherapp https://anotherapp.mng.net.au {
        #     transport http {
        #         tls_insecure_skip_verify
        #     }
        # }
}

*.mngcogo.com.au {
        tls {
                dns namecheap {
                        api_key {env.NAMECHEAP_API_KEY}
                        user {env.NAMECHEAP_USER}
                        api_endpoint https://api.namecheap.com/xml.response
                        client_ip {env.CADDY_IP}
                }
                propagation_timeout -1
                resolvers 10.99.101.8 8.8.8.8
                #        dns_challenge_override_domain mng.net.au
        }

        @api host api.mngcogo.com.au
        reverse_proxy @api https://api.mngcogo.com.au {
                transport http {
                        tls_insecure_skip_verify
                }
        }
}

5. Links to relevant resources:

2

Ok I forgot to use the command caddy run to run configuration updates and possibly request lets encrypt for a new certificate without adding a new txt record to Namecheap.

I used caddy run but now I am getting some of these following logs from the debug:

Oct 30 05:46:08 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267168.7251008,"logger":"events","msg":"event","name":"tls_get_certificate","id":"5bedba69-fab5-4bf4-91a6-a2f1f473cf1f","origin":"tls","data":{"client_hello":{"CipherSuites":[60138,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"api.mngcogo.com.au","SupportedCurves":[19018,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[35466,772,771],"RemoteAddr":{"IP":"1.146.232.77","Port":1354,"Zone":""},"LocalAddr":{"IP":"10.99.250.75","Port":443,"Zone":""}}}}
Oct 30 05:46:08 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267168.725189,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"api.mngcogo.com.au"}
Oct 30 05:46:08 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267168.725197,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.mngcogo.com.au"}
Oct 30 05:46:08 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267168.7252347,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"1.146.232.77","remote_port":"1354","server_name":"api.mngcogo.com.au","remote":"1.146.232.77:1354","identifier":"api.mngcogo.com.au","cipher_suites":[60138,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"cert_cache_fill":0.0001,"load_or_obtain_if_necessary":true,"on_demand":false}
Oct 30 05:46:08 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267168.7253098,"logger":"http.stdlib","msg":"http: TLS handshake error from 1.146.232.77:1354: no certificate available for 'api.mngcogo.com.au'"}
Oct 30 05:46:08 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267168.733146,"logger":"events","msg":"event","name":"tls_get_certificate","id":"1ba0d769-94af-4d5b-9dd1-0ae917f91be3","origin":"tls","data":{"client_hello":{"CipherSuites":[43690,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"api.mngcogo.com.au","SupportedCurves":[2570,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[2570,772,771],"RemoteAddr":{"IP":"1.146.232.77","Port":1362,"Zone":""},"LocalAddr":{"IP":"10.99.250.75","Port":443,"Zone":""}}}}
Oct 30 05:46:08 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267168.7331743,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"api.mngcogo.com.au"}
Oct 30 05:46:08 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267168.7331848,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.mngcogo.com.au"}
Oct 30 05:46:08 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267168.7332098,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"1.146.232.77","remote_port":"1362","server_name":"api.mngcogo.com.au","remote":"1.146.232.77:1362","identifier":"api.mngcogo.com.au","cipher_suites":[43690,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"cert_cache_fill":0.0001,"load_or_obtain_if_necessary":true,"on_demand":false}
Oct 30 05:46:08 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267168.7332716,"logger":"http.stdlib","msg":"http: TLS handshake error from 1.146.232.77:1362: no certificate available for 'api.mngcogo.com.au'"}
Oct 30 05:46:08 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267168.7830248,"logger":"events","msg":"event","name":"tls_get_certificate","id":"85218c17-3821-485e-9358-a40a0016d794","origin":"tls","data":{"client_hello":{"CipherSuites":[43690,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"api.mngcogo.com.au","SupportedCurves":[19018,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[35466,772,771],"RemoteAddr":{"IP":"1.146.232.77","Port":1370,"Zone":""},"LocalAddr":{"IP":"10.99.250.75","Port":443,"Zone":""}}}}
Oct 30 05:46:08 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267168.7830672,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"api.mngcogo.com.au"}
Oct 30 05:46:08 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267168.7830749,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.mngcogo.com.au"}
Oct 30 05:46:08 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267168.7830968,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"1.146.232.77","remote_port":"1370","server_name":"api.mngcogo.com.au","remote":"1.146.232.77:1370","identifier":"api.mngcogo.com.au","cipher_suites":[43690,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"cert_cache_fill":0.0001,"load_or_obtain_if_necessary":true,"on_demand":false}
Oct 30 05:46:08 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267168.7831485,"logger":"http.stdlib","msg":"http: TLS handshake error from 1.146.232.77:1370: no certificate available for 'api.mngcogo.com.au'"}
Oct 30 05:46:08 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267168.7990801,"logger":"events","msg":"event","name":"tls_get_certificate","id":"39bb0489-64d1-4d93-88cb-359bc91d7e7e","origin":"tls","data":{"client_hello":{"CipherSuites":[23130,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"api.mngcogo.com.au","SupportedCurves":[14906,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[31354,772,771],"RemoteAddr":{"IP":"1.146.232.77","Port":1376,"Zone":""},"LocalAddr":{"IP":"10.99.250.75","Port":443,"Zone":""}}}}
Oct 30 05:46:08 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267168.7991168,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"api.mngcogo.com.au"}
Oct 30 05:46:08 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267168.7991316,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.mngcogo.com.au"}
Oct 30 05:46:08 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267168.799219,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"1.146.232.77","remote_port":"1376","server_name":"api.mngcogo.com.au","remote":"1.146.232.77:1376","identifier":"api.mngcogo.com.au","cipher_suites":[23130,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"cert_cache_fill":0.0001,"load_or_obtain_if_necessary":true,"on_demand":false}
Oct 30 05:46:08 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267168.7993536,"logger":"http.stdlib","msg":"http: TLS handshake error from 1.146.232.77:1376: no certificate available for 'api.mngcogo.com.au'"}
Oct 30 05:46:09 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267169.7970793,"logger":"events","msg":"event","name":"tls_get_certificate","id":"247185a0-dd48-4292-b2ef-bc2aa8548b60","origin":"tls","data":{"client_hello":{"CipherSuites":[31354,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"api.mngcogo.com.au","SupportedCurves":[10794,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[39578,772,771],"RemoteAddr":{"IP":"1.146.232.77","Port":1400,"Zone":""},"LocalAddr":{"IP":"10.99.250.75","Port":443,"Zone":""}}}}
Oct 30 05:46:09 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267169.7971077,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"api.mngcogo.com.au"}
Oct 30 05:46:09 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267169.797123,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.mngcogo.com.au"}
Oct 30 05:46:09 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267169.7971425,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"1.146.232.77","remote_port":"1400","server_name":"api.mngcogo.com.au","remote":"1.146.232.77:1400","identifier":"api.mngcogo.com.au","cipher_suites":[31354,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"cert_cache_fill":0.0001,"load_or_obtain_if_necessary":true,"on_demand":false}
Oct 30 05:46:09 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267169.797187,"logger":"http.stdlib","msg":"http: TLS handshake error from 1.146.232.77:1400: no certificate available for 'api.mngcogo.com.au'"}
Oct 30 05:46:09 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267169.7977781,"logger":"events","msg":"event","name":"tls_get_certificate","id":"33832cac-4613-4f49-b7ef-8852ac2eca76","origin":"tls","data":{"client_hello":{"CipherSuites":[19018,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"api.mngcogo.com.au","SupportedCurves":[56026,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[10794,772,771],"RemoteAddr":{"IP":"1.146.232.77","Port":1386,"Zone":""},"LocalAddr":{"IP":"10.99.250.75","Port":443,"Zone":""}}}}
Oct 30 05:46:09 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267169.7977943,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"api.mngcogo.com.au"}
Oct 30 05:46:09 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267169.7978003,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.mngcogo.com.au"}
Oct 30 05:46:09 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267169.797817,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"1.146.232.77","remote_port":"1386","server_name":"api.mngcogo.com.au","remote":"1.146.232.77:1386","identifier":"api.mngcogo.com.au","cipher_suites":[19018,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"cert_cache_fill":0.0001,"load_or_obtain_if_necessary":true,"on_demand":false}
Oct 30 05:46:09 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267169.7978554,"logger":"http.stdlib","msg":"http: TLS handshake error from 1.146.232.77:1386: no certificate available for 'api.mngcogo.com.au'"}
Oct 30 05:46:09 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267169.8499897,"logger":"events","msg":"event","name":"tls_get_certificate","id":"933c4590-a11c-4b1f-b04b-0a5ca1103b49","origin":"tls","data":{"client_hello":{"CipherSuites":[35466,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"api.mngcogo.com.au","SupportedCurves":[23130,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[64250,772,771],"RemoteAddr":{"IP":"1.146.232.77","Port":1352,"Zone":""},"LocalAddr":{"IP":"10.99.250.75","Port":443,"Zone":""}}}}
Oct 30 05:46:09 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267169.8500204,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"api.mngcogo.com.au"}
Oct 30 05:46:09 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267169.8500278,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.mngcogo.com.au"}
Oct 30 05:46:09 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267169.850049,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"1.146.232.77","remote_port":"1352","server_name":"api.mngcogo.com.au","remote":"1.146.232.77:1352","identifier":"api.mngcogo.com.au","cipher_suites":[35466,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"cert_cache_fill":0.0001,"load_or_obtain_if_necessary":true,"on_demand":false}
Oct 30 05:46:09 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267169.8501995,"logger":"http.stdlib","msg":"http: TLS handshake error from 1.146.232.77:1352: no certificate available for 'api.mngcogo.com.au'"}
Oct 30 05:46:09 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267169.8504932,"logger":"events","msg":"event","name":"tls_get_certificate","id":"d2f97e73-70c3-4980-9109-ae65c1022f58","origin":"tls","data":{"client_hello":{"CipherSuites":[2570,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"api.mngcogo.com.au","SupportedCurves":[14906,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[6682,772,771],"RemoteAddr":{"IP":"1.146.232.77","Port":1360,"Zone":""},"LocalAddr":{"IP":"10.99.250.75","Port":443,"Zone":""}}}}
Oct 30 05:46:09 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267169.8505182,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"api.mngcogo.com.au"}
Oct 30 05:46:09 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267169.8505232,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.mngcogo.com.au"}
Oct 30 05:46:09 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267169.8505416,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"1.146.232.77","remote_port":"1360","server_name":"api.mngcogo.com.au","remote":"1.146.232.77:1360","identifier":"api.mngcogo.com.au","cipher_suites":[2570,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"cert_cache_fill":0.0001,"load_or_obtain_if_necessary":true,"on_demand":false}
Oct 30 05:46:09 caddy-rproxy caddy[51587]: {"level":"debug","ts":1730267169.850796,"logger":"http.stdlib","msg":"http: TLS handshake error from 1.146.232.77:1360: no certificate available for 'api.mngcogo.com.au'"}

It seems I’m still in this situation of no certificate being returned for mngcogo.com.au

I then instead uncommented the configuration parameter dns_challenge_override_domain mng.net.au and created the corresponding CNAME record _dns-challenge.mngocogo.com.au as well as deleted any previously auto-added record from Caddy. I did this to try delegate the ACME challenge to mng.net.au for my page api.mngcogo.com.au

However, I am still receiving the same above debug logs?

3

So I realized those above logs relate to my external requests for the webpage https://api.mngcogo.com.au

I can see certificates are generated for the *.mng.net.au sites:

administrator@caddy-rproxy:/etc/caddy$ sudo ls /var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/
backpack.mng.net.au  cogo.mng.net.au  geoserver.mng.net.au  wildcard_.mng.net.au

I’m really at a loss as to why I am not receiving a certificate for api.mngcogo.com.au?

4

Caddy is exhibiting very strange behavior. There’s something definitively wrong with the ACME challenge process. I deleted all my TXT records - everything.

I allowed enough time for the deletions to propagate.

I restarted caddy with systemctl restart caddy afresh.

It adds one TXT challenge record, and then something like 2 minutes later, it adds another TXT challenge record which stuffs the whole thing up.

Because now when I check the log, Let’s Encrypt continues to respond with “Incorrect TXT record” due to the first TXT record that was added being invalid/no longer relevant. Logs below of what happened when restarting Caddy after deleting all TXT records below:

Oct 31 01:23:56 caddy-rproxy caddy[57440]: {"level":"debug","ts":1730337836.8556263,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{"subjects":["*.mngcogo.com.au"]},{"subjects":["*.mng.net.au"]},{}]}},"http":{"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","tls":{"insecure_skip_verify":true}},"upstreams":[{"dial":"api.mngcogo.com.au:443"}]}],"match":[{"host":["api.mngcogo.com.au"]}]}]}],"terminal":true},{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","tls":{"insecure_skip_verify":true}},"upstreams":[{"dial":"geoserver.mng.net.au:443"}]}],"match":[{"host":["geoserver.mng.net.au"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","tls":{"insecure_skip_verify":true}},"upstreams":[{"dial":"cogo.mng.net.au:443"}]}],"match":[{"host":["cogo.mng.net.au"]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}}
Oct 31 01:23:56 caddy-rproxy caddy[57440]: {"level":"info","ts":1730337836.856818,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["*.mng.net.au","*.mngcogo.com.au"]}
Oct 31 01:23:56 caddy-rproxy caddy[57440]: {"level":"info","ts":1730337836.8605444,"logger":"tls.obtain","msg":"acquiring lock","identifier":"*.mngcogo.com.au"}
Oct 31 01:23:56 caddy-rproxy caddy[57440]: {"level":"info","ts":1730337836.862705,"logger":"tls.obtain","msg":"lock acquired","identifier":"*.mngcogo.com.au"}
Oct 31 01:23:56 caddy-rproxy caddy[57440]: {"level":"info","ts":1730337836.862809,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"*.mngcogo.com.au"}
Oct 31 01:23:56 caddy-rproxy caddy[57440]: {"level":"debug","ts":1730337836.8628876,"logger":"events","msg":"event","name":"cert_obtaining","id":"a959789d-63a3-4a90-bb04-929302f70721","origin":"tls","data":{"identifier":"*.mngcogo.com.au"}}
Oct 31 01:23:56 caddy-rproxy caddy[57440]: {"level":"info","ts":1730337836.8635085,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["*.mngcogo.com.au"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
Oct 31 01:23:56 caddy-rproxy caddy[57440]: {"level":"info","ts":1730337836.8635197,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["*.mngcogo.com.au"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
Oct 31 01:23:57 caddy-rproxy caddy[57440]: {"level":"debug","ts":1730337837.7316322,"logger":"tls.issuance.acme.acme_client","msg":"creating order","account":"https://acme-v02.api.letsencrypt.org/acme/acct/1911189476","identifiers":["*.mngcogo.com.au"]}
Oct 31 01:23:58 caddy-rproxy caddy[57440]: {"level":"info","ts":1730337838.7307904,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"*.mngcogo.com.au","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Oct 31 01:24:01 caddy-rproxy caddy[57440]: {"level":"debug","ts":1730337841.797194,"logger":"tls.issuance.acme.acme_client","msg":"waiting for solver before continuing","identifier":"*.mngcogo.com.au","challenge_type":"dns-01"}
Oct 31 01:24:01 caddy-rproxy caddy[57440]: {"level":"debug","ts":1730337841.797531,"logger":"tls.issuance.acme.acme_client","msg":"done waiting for solver","identifier":"*.mngcogo.com.au","challenge_type":"dns-01"}
Oct 31 01:24:02 caddy-rproxy caddy[57440]: {"level":"debug","ts":1730337842.08719,"logger":"tls.issuance.acme.acme_client","msg":"challenge accepted","identifier":"*.mngcogo.com.au","challenge_type":"dns-01"}
Oct 31 01:24:04 caddy-rproxy caddy[57440]: {"level":"error","ts":1730337844.579165,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"*.mngcogo.com.au","challenge_type":"dns-01","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: NXDOMAIN looking up TXT for _acme-challenge.mngcogo.com.au - check that a DNS record exists for this domain","instance":"","subproblems":[]}}
Oct 31 01:24:04 caddy-rproxy caddy[57440]: {"level":"error","ts":1730337844.5792365,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"*.mngcogo.com.au","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: NXDOMAIN looking up TXT for _acme-challenge.mngcogo.com.au - check that a DNS record exists for this domain","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/1911189476/318576944337","attempt":1,"max_attempts":3}
Oct 31 01:24:04 caddy-rproxy caddy[57440]: {"level":"error","ts":1730337844.5792694,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"*.mngcogo.com.au","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:dns - DNS problem: NXDOMAIN looking up TXT for _acme-challenge.mngcogo.com.au - check that a DNS record exists for this domain"}
Oct 31 01:24:04 caddy-rproxy caddy[57440]: {"level":"debug","ts":1730337844.5793338,"logger":"events","msg":"event","name":"cert_failed","id":"f93d943c-4d21-4fb7-b199-1175e012aa79","origin":"tls","data":{"error":{},"identifier":"*.mngcogo.com.au","issuers":["acme-v02.api.letsencrypt.org-directory"],"renewal":false}}
Oct 31 01:24:04 caddy-rproxy caddy[57440]: {"level":"error","ts":1730337844.5793471,"logger":"tls.obtain","msg":"will retry","error":"[*.mngcogo.com.au] Obtain: [*.mngcogo.com.au] solving challenge: *.mngcogo.com.au: [*.mngcogo.com.au] authorization failed: HTTP 400 urn:ietf:params:acme:error:dns - DNS problem: NXDOMAIN looking up TXT for _acme-challenge.mngcogo.com.au - check that a DNS record exists for this domain (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":7.716615891,"max_duration":2592000}
Oct 31 01:25:04 caddy-rproxy caddy[57440]: {"level":"info","ts":1730337904.580152,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"*.mngcogo.com.au"}
Oct 31 01:25:04 caddy-rproxy caddy[57440]: {"level":"debug","ts":1730337904.5802383,"logger":"events","msg":"event","name":"cert_obtaining","id":"cee2b258-37bd-49be-a05d-1f72c103326d","origin":"tls","data":{"identifier":"*.mngcogo.com.au"}}
Oct 31 01:25:04 caddy-rproxy caddy[57440]: {"level":"debug","ts":1730337904.582401,"logger":"tls.issuance.acme.acme_client","msg":"creating order","account":"https://acme-staging-v02.api.letsencrypt.org/acme/acct/160809493","identifiers":["*.mngcogo.com.au"]}
Oct 31 01:25:05 caddy-rproxy caddy[57440]: {"level":"debug","ts":1730337905.8878,"logger":"tls.issuance.acme.acme_client","msg":"skipping challenge initiation because authorization is not pending","identifier":"*.mngcogo.com.au","authz_status":"valid"}
Oct 31 01:25:05 caddy-rproxy caddy[57440]: {"level":"info","ts":1730337905.8878112,"logger":"tls.issuance.acme.acme_client","msg":"authorization finalized","identifier":"*.mngcogo.com.au","authz_status":"valid"}
Oct 31 01:25:10 caddy-rproxy caddy[57440]: {"level":"debug","ts":1730337910.187947,"logger":"tls.issuance.acme.acme_client","msg":"getting renewal info","names":["*.mngcogo.com.au"]}
Oct 31 01:25:10 caddy-rproxy caddy[57440]: {"level":"info","ts":1730337910.4483182,"logger":"tls.issuance.acme.acme_client","msg":"got renewal info","names":["*.mngcogo.com.au"],"window_start":1735433165.6666667,"window_end":1735605965.6666667,"selected_time":1735489588,"recheck_after":1730359510.4483037,"explanation_url":""}
Oct 31 01:25:10 caddy-rproxy caddy[57440]: {"level":"debug","ts":1730337910.7084205,"logger":"tls.issuance.acme.acme_client","msg":"getting renewal info","names":["*.mngcogo.com.au"]}
Oct 31 01:25:10 caddy-rproxy caddy[57440]: {"level":"info","ts":1730337910.9673846,"logger":"tls.issuance.acme.acme_client","msg":"got renewal info","names":["*.mngcogo.com.au"],"window_start":1735433165.6666667,"window_end":1735605965.6666667,"selected_time":1735596498,"recheck_after":1730359510.9673812,"explanation_url":""}
Oct 31 01:25:10 caddy-rproxy caddy[57440]: {"level":"info","ts":1730337910.967627,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["*.mngcogo.com.au"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
Oct 31 01:25:10 caddy-rproxy caddy[57440]: {"level":"info","ts":1730337910.9676418,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["*.mngcogo.com.au"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
Oct 31 01:25:10 caddy-rproxy caddy[57440]: {"level":"debug","ts":1730337910.9676642,"logger":"tls.issuance.acme.acme_client","msg":"creating order","account":"https://acme-v02.api.letsencrypt.org/acme/acct/1911189476","identifiers":["*.mngcogo.com.au"]}
Oct 31 01:25:12 caddy-rproxy caddy[57440]: {"level":"info","ts":1730337912.0458477,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"*.mngcogo.com.au","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Oct 31 01:25:14 caddy-rproxy caddy[57440]: {"level":"debug","ts":1730337914.7672002,"logger":"tls.issuance.acme.acme_client","msg":"waiting for solver before continuing","identifier":"*.mngcogo.com.au","challenge_type":"dns-01"}
Oct 31 01:25:14 caddy-rproxy caddy[57440]: {"level":"debug","ts":1730337914.767239,"logger":"tls.issuance.acme.acme_client","msg":"done waiting for solver","identifier":"*.mngcogo.com.au","challenge_type":"dns-01"}
Oct 31 01:25:15 caddy-rproxy caddy[57440]: {"level":"debug","ts":1730337915.0704105,"logger":"tls.issuance.acme.acme_client","msg":"challenge accepted","identifier":"*.mngcogo.com.au","challenge_type":"dns-01"}
Oct 31 01:25:17 caddy-rproxy caddy[57440]: {"level":"error","ts":1730337917.6500947,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"*.mngcogo.com.au","challenge_type":"dns-01","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Incorrect TXT record \"NvA4XwL0ew1Kanx0RXTZOGwU3xyQjbMlh_S9td0bUts\" found at _acme-challenge.mngcogo.com.au","instance":"","subproblems":[]}}
Oct 31 01:25:17 caddy-rproxy caddy[57440]: {"level":"error","ts":1730337917.650698,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"*.mngcogo.com.au","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Incorrect TXT record \"NvA4XwL0ew1Kanx0RXTZOGwU3xyQjbMlh_S9td0bUts\" found at _acme-challenge.mngcogo.com.au","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/1911189476/318577215177","attempt":1,"max_attempts":3}
Oct 31 01:25:17 caddy-rproxy caddy[57440]: {"level":"error","ts":1730337917.6507506,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"*.mngcogo.com.au","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"NvA4XwL0ew1Kanx0RXTZOGwU3xyQjbMlh_S9td0bUts\" found at _acme-challenge.mngcogo.com.au"}
Oct 31 01:25:17 caddy-rproxy caddy[57440]: {"level":"debug","ts":1730337917.650822,"logger":"events","msg":"event","name":"cert_failed","id":"e3bba00e-0e26-465c-bf0d-b5ce6b142bfc","origin":"tls","data":{"error":{"Err":{}},"identifier":"*.mngcogo.com.au","issuers":["acme-v02.api.letsencrypt.org-directory"],"renewal":false}}
Oct 31 01:25:17 caddy-rproxy caddy[57440]: {"level":"info","ts":1730337917.6508327,"logger":"tls.obtain","msg":"releasing lock","identifier":"*.mngcogo.com.au"}
Oct 31 01:25:17 caddy-rproxy caddy[57440]: {"level":"error","ts":1730337917.6509666,"logger":"tls","msg":"job failed","error":"*.mngcogo.com.au: obtaining certificate: [*.mngcogo.com.au] Obtain: [*.mngcogo.com.au] solving challenge: *.mngcogo.com.au: [*.mngcogo.com.au] authorization failed: HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"NvA4XwL0ew1Kanx0RXTZOGwU3xyQjbMlh_S9td0bUts\" found at _acme-challenge.mngcogo.com.au (ca=https://acme-v02.a

I’m just going to try serving up a self provided ssl certificate for this mngcogo.com.au site as I’m just not getting anywhere with this ACME challenge process. I don’t know why it’s behaving differently for this domain vs the other domain, and not sure what could be wrong with my Caddyfile to cause this behavior as I have replicated the same configuration from my original domain that works. Weird.

5

My guess is you have multiple Caddy processes (nodes?) running simultaneously and aren’t talking to each other. Are you sure there’s only 1 Caddy process running on that machine? What about other machines?

6

Here is what I see, using https://unboundtest.com/, there are 2 TXT records.
https://unboundtest.com/m/TXT/_acme-challenge.mngcogo.com.au/GGROYYNY

Query results for TXT _acme-challenge.mngcogo.com.au

Response:
;; opcode: QUERY, status: NOERROR, id: 46453
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version 0; flags: do; udp: 1232

;; QUESTION SECTION:
;_acme-challenge.mngcogo.com.au.	IN	 TXT

;; ANSWER SECTION:
_acme-challenge.mngcogo.com.au.	0	IN	TXT	"SGmpup0IgtkRObA3bBIc7dLkBRztnZL7_pBVjbl9zpo"
_acme-challenge.mngcogo.com.au.	0	IN	TXT	"w5RGizu5Dn5SK61yGd7REQQin1lzXwPCOTSLCeqrSzs"

----- Unbound logs -----
7

Here’s your issue:

Why are you restarting it mid-process?? Let Caddy finish its job. It’ll tell Let’s Encrypt to check the record once Caddy sees the propagation.

1 Like
8

Ok thank you for pointing that out. I wasn’t really clear on how/when Caddy requests the certificate from Let’s Encrypt.

From the debug logs it appeared like I had to restart Caddy every time I wanted it to request Let’s Encrypt for a new certificate.

I just cleared both TXT records. Clean slate again. I restarted Caddy, and now I will wait for Caddy to eventually detect that the TXT record is fully propagated. I also have the paramter propagation_timeout -1 enabled. So that should mean it will auto request Let’s Encrypt to validate the record and issue the certificate once it detects the TXT is fully propagated?

…And just as I typed this, I checked mngcogo.com.au DNS and found it added the second TXT record. Let’s Encrypt is now going to have a problem with this. I’m still not sure why this is occurring.

I don’t have any other Caddy server running. Just the one. I am assuming that perhaps it could be due to my Caddyfile having 2x tls configuration blocks that connect to the one Namecheap API (one for each domain). I was beginning to think that maybe Caddy requires one separate API endpoint per domain, however Namecheap only provides one global API endpoint.

I can’t really think what else could be the issue here…

9

No, setting it to -1 disables propagation checks. It means Caddy asks Let’s Encrypt to validate without Caddy checking before hand. Remove this line completely, let Caddy sane defaults run, and let’s observe from there.

10

Ok I wiped out the line propagation_timeout -1, cleared the TXT records and restarted Caddy again.

But now I get these logs:

Nov 01 01:38:52 caddy-rproxy caddy[62480]: {"level":"error","ts":1730425132.0292974,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"*.mngcogo.com.au","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[*.mngcogo.com.au] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-v02.api.letsencrypt.org/acme/order/1911189476/318884414087) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
Nov 01 01:38:52 caddy-rproxy caddy[62480]: {"level":"error","ts":1730425132.0294728,"logger":"tls.obtain","msg":"will retry","error":"[*.mngcogo.com.au] Obtain: [*.mngcogo.com.au] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-v02.api.letsencrypt.org/acme/order/1911189476/318884414087) (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":128.390987824,"max_duration":2592000}

These logs were what made me use the -1 setting so that Caddy doesn’t time out with it’s own verification checks.

Do I perhaps need to extend the timeout? I’m not too sure what the default time out is otherwise.

It has also just added 2x TXT records yet again.

11

Hmm… I think this issue is related to what you’re experiencing

Setting the resolvers might also help, per this issue

1 Like
12

Ah ok, that’s good to know. I believe there is some issue with the Namecheap API at the end of the day. I tried the exact same resolvers config as suggested as well, but keep getting the error solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct

And I still continue to get the duplicate TXT record.

Suppose I’ll just keep trying with my own cert and key now from here. Appreciate what you have said to help me with the issue.

13

Namecheap’s API is notoriously bad. You might want to consider switching DNS providers to another one with a better API.

1 Like
14

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.