No PEM block found

necrogami · January 27, 2021, 5:00am

1. Caddy version (`caddy version`):

v2.3.0 h1:fnrqJLa3G5vfxcxmOH/+kJOcunPLhSBnjgIvjXV/QTA=

2. How I run Caddy:

Brew services and command line

a. System environment:

OSX Big Sur Homebrew installed.

b. Command:

caddy run --config /usr/local/etc/Caddyfile

c. Service/unit/compose file:

d. My complete Caddyfile or JSON config:

*.local.dev {
	root * /Users/necrogami/code/{label1}
	tls internal
	php_fastcgi 127.0.0.1:9000
}

3. The problem I’m having:

For some reason it’s saying no PEM Block found.

4. Error messages and/or full log output:

etc % caddy run --config /usr/local/etc/Caddyfile

2021/01/27 04:44:32.550 INFO using provided configuration {"config_file": "/usr/local/etc/Caddyfile", "config_adapter": ""}

2021/01/27 04:44:32.552 INFO admin admin endpoint started {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["127.0.0.1:2019", "localhost:2019", "[::1]:2019"]}

2021/01/27 04:44:32.552 INFO tls.cache.maintenance started background certificate maintenance {"cache": "0xc000392f50"}

2021/01/27 04:44:32.552 INFO tls.cache.maintenance stopped background certificate maintenance {"cache": "0xc000392f50"}

run: loading initial config: loading new config: loading http app module: provision http: getting tls app: loading tls app module: provision tls: provisioning automation policy 0: loading TLS automation management module: position 0: loading module 'internal': provision tls.issuance.internal: loading pki app module: provision pki: provisioning CA 'local': parsing root certificate PEM: no PEM block found

5. What I already tried:

I’ve run sudo caddy untrust and sudo caddy trust and I’m still getting the same error.

6. Links to relevant resources:

francislavoie · January 27, 2021, 6:09am

{label1} is not a valid placeholder, I think you want {labels.2} here (0 is dev, 1 is local and 2 is whatever your * ends up being)

I’ve never seen that error before though. Something in Caddy’s storage must be broken. Maybe you’ll need to manually clear out the storage and try again.

necrogami · January 27, 2021, 6:42am

Alright, so I’ve changed the caddy file to be {label.2}.

How would I go about clearing out the storage for caddy? there is no folder in /usr/local/etc and the ca cert is stored in the osx keychain.

necrogami · January 27, 2021, 6:51am

So this is the newly generated ca cert currently in my osx keychain

matt · January 27, 2021, 7:06am

Before you clear any files out, I’d like your help to investigate… that error means that the file is empty, or at least, Caddy wasn’t able to read any data from the file.

Can you find the root CA .crt file in your Caddy data folder? (Run caddy environ to see where that is. It varies by platform.) What is in the file, what are its permissions, etc etc?

francislavoie · January 27, 2021, 7:26am

It’s {labels.2} (with the s)

necrogami · January 27, 2021, 7:33am

However the cert is installed in the osx keychain

When it installed this is what showed:

etc % sudo caddy trust  
Password:
2021/01/27 06:49:33.577	WARN	ca.local	installing root certificate (you might be prompted for password)	{"path": "storage:pki/authorities/local/root.crt"}
2021/01/27 01:49:33 not NSS security databases found
2021/01/27 01:49:33 define JAVA_HOME environment variable to use the Java trust
2021/01/27 01:49:33 certificate installed properly in macOS keychain

I found the file in /System/Volumes/Data/Users/necrogami/Library/Application Support/Caddy/pki/authorities/local/root.crt

-rw-------  1 root  staff   627B Jan 26 23:20 /System/Volumes/Data/Users/necrogami/Library/Application Support/Caddy/pki/authorities/local/root.crt

caddy environ

etc % caddy environ
caddy.HomeDir=/Users/necrogami
caddy.AppDataDir=/Users/necrogami/Library/Application Support/Caddy
caddy.AppConfigDir=/Users/necrogami/Library/Application Support/Caddy
caddy.ConfigAutosavePath=/Users/necrogami/Library/Application Support/Caddy/autosave.json
caddy.Version=v2.3.0
runtime.GOOS=darwin
runtime.GOARCH=amd64
runtime.Compiler=gc
runtime.NumCPU=8
runtime.GOMAXPROCS=8
runtime.Version=go1.15.6
os.Getwd=/usr/local/etc

__CFBundleIdentifier=com.apple.Terminal
TMPDIR=/var/folders/1r/3131c4q921l5bcxgqjts1jh00000gn/T/
XPC_FLAGS=0x0
LaunchInstanceID=132092E9-1507-4B68-9F1D-8AEA0238E4C7
TERM=xterm-256color
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.8bBls4i2or/Listeners
SECURITYSESSIONID=186a6
XPC_SERVICE_NAME=0
TERM_PROGRAM=Apple_Terminal
TERM_PROGRAM_VERSION=440
TERM_SESSION_ID=DFD7F34C-D03D-4BD7-A66C-E07C35DDA5A8
SHELL=/bin/zsh
HOME=/Users/necrogami
LOGNAME=necrogami
USER=necrogami
PATH=/Users/necrogami/.composer/vendor/bin:.git/safe/../../bin:/Users/necrogami/.bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/necrogami/.fzf/bin
SHLVL=1
PWD=/usr/local/etc
OLDPWD=/usr/local/etc/php
CLICOLOR=1
VISUAL=code
EDITOR=code
ERL_AFLAGS=-kernel shell_history enabled
HOMEBREW_NO_ANALYTICS=1
LANG=en_US.UTF-8
_=/usr/local/bin/caddy

necrogami · January 27, 2021, 7:34am

Upon further investigation I found that I had to have caddy running as sudo to be able to get access to the file. however without sudo I cannot start caddy.

necrogami · January 27, 2021, 7:37am

However now when I try to start caddy as a brew service:

{"level":"info","ts":1611732938.3037708,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1611732938.303995,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003bed20"}
{"level":"info","ts":1611732938.317109,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0003bed20"}
run: loading initial config: loading new config: loading tls app module: provision tls: provisioning automation policy 0: loading TLS automation management module: position 0: loading module 'internal': provision tls.issuance.internal: loading pki app module: provision pki: provisioning CA 'local': generating root: saving root certificate: mkdir caddy: read-only file system
{"level":"warn","ts":1611732948.339079,"msg":"unable to determine directory for user configuration; falling back to current directory","error":"$HOME is not defined"}
{"level":"info","ts":1611732948.367262,"msg":"using provided configuration","config_file":"/usr/local/etc/Caddyfile","config_adapter":""}
{"level":"info","ts":1611732948.369656,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1611732948.3699808,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000359650"}
{"level":"info","ts":1611732948.3803341,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc000359650"}
run: loading initial config: loading new config: loading http app module: provision http: getting tls app: loading tls app module: provision tls: provisioning automation policy 0: loading TLS automation management module: position 0: loading module 'internal': provision tls.issuance.internal: loading pki app module: provision pki: provisioning CA 'local': generating root: saving root certificate: mkdir caddy: read-only file system

Any idea what it’s talking about with a read only filesystem
I’m running
sudo brew services start caddy so it starts as root

It’s now running when I run sudo caddy run --config /usr/local/etc/Caddyfile

matt · January 27, 2021, 7:42am

Thanks for the details! That’s helpful.

Looks like /System/Volumes/Data is a new read-only folder:

Apparently Caddy does not have write permissions in that folder, which would definitely be a problem.

(Sorry, I do not have a Mac, so my guess would be worse than yours at this point. All I could do is search and point you to links…)

necrogami · January 27, 2021, 7:50am

So I’m guessing caddy is trying to create a directory there for some reason? Would this be classified as a bug?

matt · January 27, 2021, 7:58am

No, it’s not a bug. Caddy’s data directory is defined here: Conventions — Caddy Documentation

For Macs, it’s $HOME/Library/Application Support/Caddy – so apparently at some point you ran Caddy in such a way that its $HOME was /System/Volumes/Data/Users/necrogami. I don’t know what causes that on Mac.

Honestly, I stopped using Macs last year after I noticed Apple trending toward more hostile decisions against developers. My Mac was becoming more useless to me as more obscure, undocumented, obstructive changes occurred. I’d be interested to learn what you find out about this one!

necrogami · January 27, 2021, 8:38am

I opened up a discussion https://github.com/Homebrew/discussions/discussions/618

Lets hope someone with some knowledge can help get this solved. This is a fairly fresh 2015 intel mbp that was just upgraded from Catalina to Big Sur.

I’ve also tried uninstalling cleaning and reinstalling then running the brew service as both sudo and without sudo, neither have worked. I’ll update here more once I have further details from the home-brew community.

necrogami · January 27, 2021, 9:11am

When the log returns:

run: loading initial config: loading new config: loading http app module: provision http: getting tls app: loading tls app module: provision tls: provisioning automation policy 0: loading TLS automation management module: position 0: loading module 'internal': provision tls.issuance.internal: loading pki app module: provision pki: provisioning CA 'local': generating root: saving root certificate: mkdir caddy: read-only file system

is there a way to make that return more data, like what directory is it trying to create? maybe get some environment variables?

necrogami · January 27, 2021, 9:48am

So talking over there I got it to work, I had to hard code the environment variable for HOME to / which allowed caddy to use /Library/Application Support/Caddy as primary directory.

I’ve submitted a bug with home-brew core to see if they can get a fix.

matt · January 27, 2021, 5:00pm

Interesting… is there any reason your $HOME isn’t just /Users/necrogami?

necrogami · January 27, 2021, 6:01pm

So apparently Launchd doesn’t send environment variables not even path. So no environment variables at all are avaialble in launchd/homebrew services.

necrogami · January 27, 2021, 6:13pm

An idea, if no env, no home is found can it maybe instead dump the files relative to the caddyfile? which would put the pki and autoconf stuff in /usr/local/etc not great but not terrible and would get it working out of the box.

matt · January 27, 2021, 6:46pm

The problem is there is often not a Caddyfile. Config is loaded through an API, so we can’t really use the Caddyfile as a reference.

system · February 26, 2021, 5:01am

This topic was automatically closed after 30 days. New replies are no longer allowed.