"illegal base64 data" error with file placeholder for eab secrets

Help
1

1. The problem I’m having:

Hey! I am trying to make caddy read acme eab secrets (both key_id and mac_key) from secrets files. This is what I have in my Caddyfile:

{
    acme_ca <ca_url>
    acme_eab {
        key_id {file./etc/caddy/eab.key_id}
        mac_key {file./etc/caddy/eab.mac_key}
    }
}

The eab.key_id and eab.mac_key files contain exactly the strings that I previously had set directly in the Caddyfile, which worked. But with the file placeholders I get “base64-decoding MAC key: illegal base64 data at input byte 0” errors. I tried to base64 encode the content of the eab.mac_key file as suggested by the error message, even though that isn’t documented anywhere I could find, but it didn’t help either.

Any idea what I am doing wrong, or if this should work and I am encountering a bug?

Also, once this is working, does the file placeholder support relative paths too? All examples I could find seem to use absolute paths only.

Thanks!

2. Error messages and/or full log output:

# docker compose logs reverse-proxy
reverse-proxy-1  | {"level":"info","ts":1767709815.4882739,"msg":"maxprocs: Leaving GOMAXPROCS=12: CPU quota undefined"}
reverse-proxy-1  | {"level":"info","ts":1767709815.4884846,"msg":"GOMEMLIMIT is updated","package":"github.com/KimMachineGun/automemlimit/memlimit","GOMEMLIMIT":121282810675,"previous":9223372036854775807}
reverse-proxy-1  | {"level":"info","ts":1767709815.4885173,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
reverse-proxy-1  | {"level":"info","ts":1767709815.4895983,"msg":"adapted config to JSON","adapter":"caddyfile"}
reverse-proxy-1  | {"level":"warn","ts":1767709815.489615,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
reverse-proxy-1  | {"level":"info","ts":1767709815.4905484,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:2019","//[::1]:2019"]}
reverse-proxy-1  | {"level":"info","ts":1767709815.4907541,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
reverse-proxy-1  | {"level":"info","ts":1767709815.4907691,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
reverse-proxy-1  | {"level":"info","ts":1767709815.490878,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000479e80"}
reverse-proxy-1  | {"level":"warn","ts":1767709815.491007,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"}
reverse-proxy-1  | {"level":"warn","ts":1767709815.4910176,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"}
reverse-proxy-1  | {"level":"info","ts":1767709815.4910216,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
reverse-proxy-1  | {"level":"info","ts":1767709815.491048,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
reverse-proxy-1  | {"level":"info","ts":1767709815.4911294,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
reverse-proxy-1  | {"level":"info","ts":1767709815.4913592,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
reverse-proxy-1  | {"level":"info","ts":1767709815.4914155,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["arrakis.icg.kfa-juelich.de","arrakis.fz-juelich.de"]}
reverse-proxy-1  | {"level":"info","ts":1767709815.4933748,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
reverse-proxy-1  | {"level":"info","ts":1767709815.4934032,"msg":"serving initial configuration"}
reverse-proxy-1  | {"level":"info","ts":1767709815.4939005,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/data/caddy"}
reverse-proxy-1  | {"level":"info","ts":1767709815.4942474,"logger":"tls","msg":"finished cleaning storage units"}
reverse-proxy-1  | {"level":"info","ts":1767709815.4942462,"logger":"tls.obtain","msg":"acquiring lock","identifier":"arrakis.icg.kfa-juelich.de"}
reverse-proxy-1  | {"level":"info","ts":1767709815.4943001,"logger":"tls.obtain","msg":"acquiring lock","identifier":"arrakis.fz-juelich.de"}
reverse-proxy-1  | {"level":"info","ts":1767709815.4946697,"logger":"tls.obtain","msg":"lock acquired","identifier":"arrakis.icg.kfa-juelich.de"}
reverse-proxy-1  | {"level":"info","ts":1767709815.4947612,"logger":"tls.obtain","msg":"lock acquired","identifier":"arrakis.fz-juelich.de"}
reverse-proxy-1  | {"level":"info","ts":1767709815.4947686,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"arrakis.icg.kfa-juelich.de"}
reverse-proxy-1  | {"level":"info","ts":1767709815.4948761,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"arrakis.fz-juelich.de"}
reverse-proxy-1  | {"level":"info","ts":1767709815.4953938,"logger":"http","msg":"creating new account because no account for configured email is known to us","email":"","ca":"https://acme-v02.harica.gr/acme/5554e4fc-46ab-4cb1-91f7-b86a15922369/directory","error":"open /data/caddy/acme/acme-v02.harica.gr-acme-5554e4fc-46ab-4cb1-91f7-b86a15922369-directory/users/default/default.json: no such file or directory"}
reverse-proxy-1  | {"level":"info","ts":1767709815.4954324,"logger":"http","msg":"ACME account has empty status; registering account with ACME server","contact":[],"location":""}
reverse-proxy-1  | {"level":"info","ts":1767709815.4954264,"logger":"http","msg":"creating new account because no account for configured email is known to us","email":"","ca":"https://acme-v02.harica.gr/acme/5554e4fc-46ab-4cb1-91f7-b86a15922369/directory","error":"open /data/caddy/acme/acme-v02.harica.gr-acme-5554e4fc-46ab-4cb1-91f7-b86a15922369-directory/users/default/default.json: no such file or directory"}
reverse-proxy-1  | {"level":"info","ts":1767709815.4954736,"logger":"http","msg":"ACME account has empty status; registering account with ACME server","contact":[],"location":""}
reverse-proxy-1  | {"level":"info","ts":1767709815.4956896,"logger":"http","msg":"creating new account because no account for configured email is known to us","email":"","ca":"https://acme-v02.harica.gr/acme/5554e4fc-46ab-4cb1-91f7-b86a15922369/directory","error":"open /data/caddy/acme/acme-v02.harica.gr-acme-5554e4fc-46ab-4cb1-91f7-b86a15922369-directory/users/default/default.json: no such file or directory"}
reverse-proxy-1  | {"level":"error","ts":1767709815.6390524,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"arrakis.fz-juelich.de","issuer":"acme-v02.harica.gr-acme-5554e4fc-46ab-4cb1-91f7-b86a15922369-directory","error":"base64-decoding MAC key: illegal base64 data at input byte 0"}
reverse-proxy-1  | {"level":"error","ts":1767709815.639143,"logger":"tls.obtain","msg":"will retry","error":"[arrakis.fz-juelich.de] Obtain: base64-decoding MAC key: illegal base64 data at input byte 0","attempt":1,"retrying_in":60,"elapsed":0.144355988,"max_duration":2592000}

These “base64-decoding MAC key: illegal base64 data at input byte 0” errors then repeat on the retries.

3. Caddy version:

v2.10.2 h1:g/gTYjGMD0dec+UgMw8SnfmJ3I9+M2TdvoRL/Ovu6U8=

4. How I installed and ran Caddy:

a. System environment:

Ubuntu 20.04, Docker Compose version v2.35.1, x86_64 (rather standard server hardware)

b. Command:

c. Service/unit/compose file:

services:
  reverse-proxy:
    image: caddy:2
    restart: always
    ports:
      - "134.94.154.152:80:80"
      - "134.94.154.152:443:443"
    volumes:
      - "./Caddyfile:/etc/caddy/Caddyfile:ro"
      - "./eab.key_id:/etc/caddy/eab.key_id:ro"
      - "./eab.mac_key:/etc/caddy/eab.mac_key:ro"
      - "/srv/data/arrakis/caddy/data:/data:rw"
      - "/srv/data/arrakis/caddy/config:/config:rw"

d. My complete Caddy config:

{
	acme_ca <ca_url>
	acme_eab {
		key_id {file./etc/caddy/eab.key_id}
		mac_key {file./etc/caddy/eab.mac_key}
	}
}

arrakis.fz-juelich.de {
	@debug {
		path /debug
		path /debug/*
	}
	respond @debug "Access denied" 403 {
		close
	}
	reverse_proxy {
		to web:3000
	}
}

arrakis.icg.kfa-juelich.de {
	redir https://arrakis.fz-juelich.de{uri}
}

5. Links to relevant resources:

2

This doesn’t look valid according to the documentation. There’s no such directive acme_eab. Are you using some module?

Instead there is eab within a tls block: tls (Caddyfile directive) — Caddy Documentation

acme_ca should also be ca within tls.

3

They are global options, documented here:

And they do work, as long as I don’t use a file placeholder.

FWIW I’ve previously used the ca and eab directives within a tls block, but those weren’t accepted globally so I had to duplicate the configuration for both sites. They didn’t work with file placeholders either.

4

Sorry, I did search for those options but my search engine failed me. Maybe the base64 encoding of the key file was wrong - there are base64 variants for URL encoding, padding etc?

5

I haven’t looked at the code, but could it that the file has extra new line at the end, i.e. \n, or something like that?

6

No, they don’t end in a newline, I made sure of that. But that shouldn’t be an issue since v2.8.5 anyway: replacer: `{file.*}` global placeholder strips trailing newline by steffenbusch · Pull Request #6411 · caddyserver/caddy · GitHub

7

I’ve created the secrets files with printf '<secret>' | base64 > eab.mac_key, but also tried without encoding: printf '<secret>' > eab.mac_key. If there is any expectation of what base64 encoding to use I couldn’t find it in the docs, I couldn’t even find a mention of base64 related to these file placeholders at all.

8

Global placeholders do not necessarily work everywhere in the config, it requires there being a bit of code which invokes the replacer to transform the placeholder into the underlying value.

PRs welcome if you want to investigate, should be easy to find where those config values are used, and whether there’s a caddy.NewReplacer() to do it. See Placeholder Support — Caddy Documentation for some info about how the code would look (in the case of those docs, for plugins, but still the same thing)