Issues with Bunny DNS module

Louen · December 12, 2024, 5:53pm

1. The problem I’m having:

I’ve been using the Bunny CDN DNS module for 5 months now. But I’m not sure exactly when, maybe some days ago, it seems its broken or something, because I have not changed anything in the configs lately.

2. Error messages and/or full log output:

I’m seeing this error in the logs.

{
  "level": "info",
  "ts": 1734024513.6386855,
  "msg": "panic: certificate worker: runtime error: index out of range [0] with length 0\ngoroutine 51 [running]:\ngithub.com/caddyserver/certmagic.(*jobManager).worker.func1()\n\tgithub.com/caddyserver/certmagic@v0.21.3/async.go:58 +0x65\npanic({0x1869600?, 0xc0007c4ed0?})\n\truntime/panic.go:770 +0x132\ngithub.com/libdns/bunny.getZoneID({0x1e767c8, 0xc00077ee40}, {0xc00043e550, 0x48}, {0xc0002f19b0?, 0x0?})\n\tgithub.com/libdns/bunny@v0.1.0/client.go:82 +0x1ac\ngithub.com/libdns/bunny.createRecord({0x1e767c8, 0xc00077ee40}, {0xc00043e550, 0x48}, {0xc0002f19b0, 0x20}, {{0x0, 0x0}, {0x19424f1, 0x3}, ...})\n\tgithub.com/libdns/bunny@v0.1.0/client.go:122 +0x85\ngithub.com/libdns/bunny.(*Provider).AppendRecords(0xc000500b70, {0x1e767c8, 0xc00077ee40}, {0xc0002f19b0, 0x21}, {0xc000712780?, 0x168cc00?, 0x1?})\n\tgithub.com/libdns/bunny@v0.1.0/provider.go:31 +0x26a\ngithub.com/caddyserver/certmagic.(*DNSManager).createRecord(0xc0002c0d20, {0x1e767c8, 0xc00077ee40}, {0xc0003d6f00, 0x20}, {0x19424f1, 0x3}, {0xc0002f18c0, 0x2b})\n\tgithub.com/caddyserver/certmagic@v0.21.3/solvers.go:401 +0x815\ngithub.com/caddyserver/certmagic.(*DNS01Solver).Present(0xc0002c0d20, {0x1e767c8, 0xc00077ee40}, {{0xc0002b1a68, 0x6}, {0xc000736840, 0x54}, {0xc0002b1a70, 0x7}, {0x0, ...}, ...})\n\tgithub.com/caddyserver/certmagic@v0.21.3/solvers.go:265 +0x19e\ngithub.com/caddyserver/certmagic.solverWrapper.Present({{0x1e6d848?, 0xc0002c0d20?}}, {0x1e767c8, 0xc00077ee40}, {{0xc0002b1a68, 0x6}, {0xc000736840, 0x54}, {0xc0002b1a70, 0x7}, ...})\n\tgithub.com/caddyserver/certmagic@v0.21.3/solvers.go:827 +0x22e\ngithub.com/mholt/acmez/v2.(*Client).presentForNextChallenge(0xc0008be2c0, {0x1e767c8, 0xc00077ee40}, 0xc00097b340)\n\tgithub.com/mholt/acmez/v2@v2.0.1/client.go:410 +0x7ab\ngithub.com/mholt/acmez/v2.(*Client).solveChallenges(_, {_, _}, {{0xc000714d38, 0x5}, {0x0, 0x0, 0x0}, 0x1, {0x0, ...}, ...}, ...)\n\tgithub.com/mholt/acmez/v2@v2.0.1/client.go:361 +0x26d\ngithub.com/mholt/acmez/v2.(*Client).ObtainCertificate(0xc0008be2c0, {0x1e767c8, 0xc00077ee40}, {{{0xc000714d38, 0x5}, {0x0, 0x0, 0x0}, 0x1, {0x0, ...}, ...}, ...})\n\tgithub.com/mholt/acmez/v2@v2.0.1/client.go:136 +0x598\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue(0xc0008b7e00, {0x1e767c8, 0xc00077ee40}, 0xc0006bc008, 0x0)\n\tgithub.com/caddyserver/certmagic@v0.21.3/acmeissuer.go:477 +0x648\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue(0xc0008b7e00, {0x1e767c8, 0xc00077ee40}, 0xc0006bc008)\n\tgithub.com/caddyserver/certmagic@v0.21.3/acmeissuer.go:371 +0xa7\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue(0x2a89150?, {0x1e767c8?, 0xc00077ee40?}, 0xc0008be190?)\n\tgithub.com/caddyserver/caddy/v2@v2.8.4/modules/caddytls/acmeissuer.go:248 +0x25\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2({0x1e767c8, 0xc00077ee40})\n\tgithub.com/caddyserver/certmagic@v0.21.3/config.go:620 +0xc84\ngithub.com/caddyserver/certmagic.doWithRetry({0x1e76800, 0xc00040b0e0}, 0xc00071c800, 0xc000787998)\n\tgithub.com/caddyserver/certmagic@v0.21.3/async.go:104 +0x233\ngithub.com/caddyserver/certmagic.(*Config).obtainCert(0xc00080b880, {0x1e76800, 0xc00040b0e0}, {0xc000418258, 0x12}, 0x0)\n\tgithub.com/caddyserver/certmagic@v0.21.3/config.go:694 +0x729\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync(...)\n\tgithub.com/caddyserver/certmagic@v0.21.3/config.go:499\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1()\n\tgithub.com/caddyserver/certmagic@v0.21.3/config.go:409 +0x73\ngithub.com/caddyserver/certmagic.(*jobManager).worker(0x2aa8220)\n\tgithub.com/caddyserver/certmagic@v0.21.3/async.go:73 +0x11b\ncreated by github.com/caddyserver/certmagic.(*jobManager).Submit in goroutine 1\n\tgithub.com/caddyserver/certmagic@v0.21.3/async.go:50 +0x279"
}

After this I see this type of error all the time non-stop:

{
  "level": "error",
  "ts": 1734024513.7185519,
  "logger": "tls.issuance.acme.acme_client",
  "msg": "validating authorization",
  "identifier": "mysite.com",
  "problem": {
    "type": "urn:ietf:params:acme:error:tls",
    "title": "",
    "detail": "84.17.63.178: remote error: tls: no application protocol",
    "instance": "",
    "subproblems": []
  },
  "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/158696983/21239318364",
  "attempt": 1,
  "max_attempts": 3
}

3. Caddy version:

Docker caddy:latest

4. How I installed and ran Caddy:

This is my current Caddyfile config:

(tls) {
        tls {
                dns bunny API_TOKEN
        }
}

*.mysite.com {
        root * /srv/mysite
        file_server
        php_fastcgi unix//run/php/php-fpm.sock {
                root /var/www/html
        }
        import tls
}

5. Links to relevant resources:

Louen · December 13, 2024, 12:50pm

I contacted the Bunny support and they suggested me that this error was because there was an empty DNS Zone so I created a fork in the bunny libdns Github repo to sort this and solve it

{
   "level":"error",
   "ts":1734093679.7992978,
   "logger":"tls.obtain",
   "msg":"could not get certificate from issuer",
   "identifier":"*.mysite.com",
   "issuer":"acme-v02.api.letsencrypt.org-directory",
   "error":"[*.mysite.com] solving challenges: presenting for challenge: adding temporary record for zone \"_acme-challenge.mysite.com.\": zone not found: _acme-challenge.mysite.com (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/158696983/21239318474) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"
}
{
   "level":"error",
   "ts":1734093679.7995076,
   "logger":"tls.obtain",
   "msg":"will retry",
   "error":"[*.mysite.com] Obtain: [*.mysite.com] solving challenges: presenting for challenge: adding temporary record for zone \"_acme-challenge.mysite.com.\": zone not found: _acme-challenge.mysite.com (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/158696983/21239318474) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)",
   "attempt":2,
   "retrying_in":120,
   "elapsed":62.522038877,
   "max_duration":2592000
}
{
   "level":"info",
   "ts":1734093681.371488,
   "logger":"tls.obtain",
   "msg":"obtaining certificate",
   "identifier":"mysite.com"
}
{
   "level":"info",
   "ts":1734093681.3735995,
   "logger":"tls",
   "msg":"using ACME account",
   "account_id":"https://acme-staging-v02.api.letsencrypt.org/acme/acct/158696983",
   "account_contact":[
      
   ]
}
{
   "level":"info",
   "ts":1734093682.2132306,
   "logger":"tls.acme_client",
   "msg":"trying to solve challenge",
   "identifier":"mysite.com",
   "challenge_type":"http-01",
   "ca":"https://acme-staging-v02.api.letsencrypt.org/directory"
}

So basically, there might still be some errors in the Bunny DNS I’m unable to identify.

Louen · December 13, 2024, 3:19pm

More info.

I’ve enabled the Bunny CDN DNS logging and I can clearly see that the acme challenge is being created

So I can say that the problem is not within neither the Bunny DNS modules nor the LibDNS.

Anyway I tried downgrading to 2.8.1 and 2.7.6 and nothing has changed.

I added some extra logging to the LibDNS and allI can see is this error

{
   "level":"error",
   "ts":1734093679.7992978,
   "logger":"tls.obtain",
   "msg":"could not get certificate from issuer",
   "identifier":"*.mysite.com",
   "issuer":"acme-v02.api.letsencrypt.org-directory",
   "error":"[*.mysite.com] solving challenges: presenting for challenge: adding temporary record for zone \"_acme-challenge.mysite.com.\": zone not found: _acme-challenge.mysite.com (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/158696983/21239318474) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"
}

Also Bunny support answered me saying that the endpoints have not changed in the last year.

I’m wondering if those two SOA and A are expected to be created. I was reading the Certbot docs
https://eff-certbot.readthedocs.io/en/latest/using.html#dns-plugins

Theoretically the _acme-challenge must be created as a TXT record, not as an A/SOA record.

So I wonder whats going on. What feels wrong is that I have not touched anything lately in the Caddyfile configs.

From what I can find, there is a call here from here:

github.com

caddyserver/certmagic/blob/d2a983dc776fcaf71587ef5ae1497529ace15d2a/solvers.go#L403


      
          	logger.Debug("creating DNS record",
          		zap.String("dns_name", dnsName),
          		zap.String("zone", zone),
          		zap.String("record_name", rec.Name),
          		zap.String("record_type", rec.Type),
          		zap.String("record_value", rec.Value),
          		zap.Duration("record_ttl", rec.TTL))
          
          	results, err := m.DNSProvider.AppendRecords(ctx, zone, []libdns.Record{rec})
          	if err != nil {
          		return zoneRecord{}, fmt.Errorf("adding temporary record for zone %q: %w", zone, err)
          	}
          	if len(results) != 1 {
          		return zoneRecord{}, fmt.Errorf("expected one record, got %d: %v", len(results), results)
          	}
          
          	return zoneRecord{zone, results[0]}, nil
          }
          
          // wait blocks until the TXT record created in Present() appears in
          // authoritative lookups, i.e. until it has propagated, or until

But I can’t really see these debug part with debug directive enabled:

logger.Debug("creating DNS record",
		zap.String("dns_name", dnsName),
		zap.String("zone", zone),
		zap.String("record_name", rec.Name),
		zap.String("record_type", rec.Type),
		zap.String("record_value", rec.Value),
		zap.Duration("record_ttl", rec.TTL))

Louen · December 20, 2024, 10:56pm

I’m going to try to replicate this with another host, because i’ve had my site using bunny cdn down for one week now and its killing me. Luckily I only did this with one site for now for testing purposes.

Louen · December 21, 2024, 7:56pm

I have rebuit it in a brand new server, brand new domain, same Bunny DNS provider, and it worked.

This is seriously killing me because I have not touched a thing in the past months, and all of sudden it broke.

I’ve tried to take the failing domain to the other server and did this:

(tls) {
        tls {
                dns bunny API_KEY
        }
}

myworkingsite.com, *.myworkingsite.com {
        root * /srv/web
        file_server
        php_fastcgi unix//run/php/www.sock {
                root /var/www/html
        }
        import tls
}

myfailingsite.com, *.myfailingsite.com {
        root * /srv/web
        file_server
        php_fastcgi unix//run/php/www.sock {
                root /var/www/html
        }
        import tls
}

I don’t know if anyone can confirm me, but AFAIK basically, both myworkingsite.com and myfailingsite.com point to the same place, so technically if the former works, the later should also work the same way (both sites are configured in Bunny CDN and pointing to the exact same Caddy server IP address).

After running Caddy, myfailingsite.com continues to fail, so I’m starting to believe that there is something wrong with Bunny CDN that started happening all of a sudden. I’ve contacted their support to see if they can check this

Louen · December 22, 2024, 12:48am

I contacted their support and this is the conclusion:

I do see that we potentially may have made a change 2 months ago that may have changed how searching works, but I think the fix would be that the client doesn’t send _acme-challenge when requesting a wildcard. I can confirm its replicable with any other domain if you try to issue a wildcard, which is why it worked for your other domain.

$ curl "https://api.bunny.net/dnszone?page=1&perPage=5&search=_acme-challenge.mysite.com" -H "AccessKey: KEY"{"Items":[],"CurrentPage":1,"TotalItems":0,"HasMoreItems":false}

$ curl "https://api.bunny.net/dnszone?page=1&perPage=5&search=mysite.com" -H "AccessKey: KEY"{"Items":[{"Id":1234,"Domain":"mysite.com","Records":[{"Id":1234,"Type":2,"Ttl":300,"Value":".................... THE WHOLE LIST

So basically it seems that the plugin maybe? its looking searching through _acme-challenge because, probably Caddy is asking for this and for this reason it returns 0 results, and the panic error

If you search directly through the domain, it returns everything as expected.

There are 2 very weird things:

I thought it was the fault of Bunny, but its not, because I’ve tried using SWAG bunny plugin and it works flawlessly
What it’s OMEGA weird, is that I used another domain, and it worked well I even tried to delete the certificates twice, and it worked again. I don’t really understand what Caddy is finding in my domain that decides to ask for the _acme-challenge while in the other domain it doesnt. The only difference between the two domains is that the one that doesnt work, has a number.

At this point, I’ve run out of options.

francislavoie · December 22, 2024, 11:13am

It should be a TXT record, not an A record.

No, should not. It must be a bug/misunderstanding with the Bunny API, I guess.

Louen · December 22, 2024, 12:32pm

Yes, but its weird, because neither libdns for bunny or the bunny plugin

Doesnt really have any references to _acme-challenge, which seems to be the cause of why Bunny plugin is failing (and maybe they should)

This module only provides just the minimal interface to let Caddy manipulate the Bunny DNS API

So I understand that the logic behind the certmagic is what is conflicting with the Bunny DNS system, when at some point, its trying to retrieve the information by querying with _acme-challenge to check if its there.

Not sure if this is the expected behaviour. AFAIK, the DNS-01 challenge simply needs to add the _acme-challenge with the received ACME token and ask Letsencrypt to query the domain DNS, but here it seems that it tries to do some checks before and it provokes the fatal error since it doesnt receive any records (+ the fact that bunny libdns has a bug because its not catching the no-record scenario, but this is a different story).

Bruce5051 · December 22, 2024, 11:28pm

A link to the DNS-01 challenge of the Challenge Types - Let's Encrypt.

Louen · December 22, 2024, 11:49pm

I know this. What I dont really know is how CertMagic is doing such, checks and everything (I’m not even sure if CertMagic is what handles this).

The thing is that its failing during Bunny DNS management, and after checking the Bunny plugin and the Bunny DNS library, I see that it only offers the functions representation for Caddy (or CertMagic, again not sure), to operate with Bunny DNS API

If I’m sincere I have not reviewed the code (+ I dont know anything about Go), so I’m not even sure where to start digging and if Bunny DNS lib should improve some function, or if there is some checking operation in CertMagic that should be reviewed to improve this.

Bruce5051 · December 22, 2024, 11:53pm

Maybe try posting an issue with them here Issues · caddy-dns/bunny · GitHub?

system · January 21, 2025, 11:53pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.