Caddy is not starting anymore w/ its own user

Help
1

1. The problem I’m having:

I installed a caddy instance by-hand. Actually it was compiled w/ xcaddy. I use only the mercure and vulcain pkgs inside the caddy. And I have the systemd service from the official website.

The host is an Arch linux. (btw) It is my setup for ~5-6 years. Something happened a few weeks ago, and the systemd services does not working anymore.

I think this problem is come from permission issues, but the error message does not help to me.

If I run w/ administrator roles, then it running perfectly.

$ sudo caddy run --environ --config /etc/caddy/Caddyfile

If I run w/ the caddy user, it fails.

$ sudo -H -u caddy caddy run --environ --config /etc/caddy/Caddyfile

2. Error messages and/or full log output:

systemd logs
(The command you wanted does not show anything…)

$ journalctl -u caddy -f 
márc 08 17:34:02 AMBER systemd[1]: Starting Caddy...
márc 08 17:34:03 AMBER caddy[90444]: caddy.HomeDir=/var/lib/caddy
márc 08 17:34:03 AMBER caddy[90444]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
márc 08 17:34:03 AMBER caddy[90444]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
márc 08 17:34:03 AMBER caddy[90444]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
márc 08 17:34:03 AMBER caddy[90444]: caddy.Version=v2.9.1 h1:OEYiZ7DbCzAWVb6TNEkjRcSCRGHVoZsJinoDR/n9oaY=
márc 08 17:34:03 AMBER caddy[90444]: runtime.GOOS=linux
márc 08 17:34:03 AMBER caddy[90444]: runtime.GOARCH=amd64
márc 08 17:34:03 AMBER caddy[90444]: runtime.Compiler=gc
márc 08 17:34:03 AMBER caddy[90444]: runtime.NumCPU=8
márc 08 17:34:03 AMBER caddy[90444]: runtime.GOMAXPROCS=8
márc 08 17:34:03 AMBER caddy[90444]: runtime.Version=go1.24.1
márc 08 17:34:03 AMBER caddy[90444]: os.Getwd=/
márc 08 17:34:03 AMBER caddy[90444]: LANG=en_GB.UTF-8
márc 08 17:34:03 AMBER caddy[90444]: LC_MEASUREMENT=hu_HU.UTF-8
márc 08 17:34:03 AMBER caddy[90444]: LC_MONETARY=hu_HU.UTF-8
márc 08 17:34:03 AMBER caddy[90444]: LC_NUMERIC=hu_HU.UTF-8
márc 08 17:34:03 AMBER caddy[90444]: LC_PAPER=hu_HU.UTF-8
márc 08 17:34:03 AMBER caddy[90444]: LC_TIME=hu_HU.UTF-8
márc 08 17:34:03 AMBER caddy[90444]: PATH=/usr/local/sbin:/usr/local/bin:/usr/bin
márc 08 17:34:03 AMBER caddy[90444]: XDG_DATA_DIRS=/var/lib/flatpak/exports/share:/usr/local/share/:/usr/share/
márc 08 17:34:03 AMBER caddy[90444]: NOTIFY_SOCKET=/run/systemd/notify
márc 08 17:34:03 AMBER caddy[90444]: USER=caddy
márc 08 17:34:03 AMBER caddy[90444]: LOGNAME=caddy
márc 08 17:34:03 AMBER caddy[90444]: HOME=/var/lib/caddy
márc 08 17:34:03 AMBER caddy[90444]: INVOCATION_ID=c3536044013944a9bf4ea2389dd1a131
márc 08 17:34:03 AMBER caddy[90444]: JOURNAL_STREAM=9:311868
márc 08 17:34:03 AMBER caddy[90444]: SYSTEMD_EXEC_PID=90444
márc 08 17:34:03 AMBER caddy[90444]: MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/system.slice/caddy.service/memory.pressure
márc 08 17:34:03 AMBER caddy[90444]: MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA=
márc 08 17:34:03 AMBER caddy[90444]: {"level":"info","ts":1741451643.0744166,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
márc 08 17:34:03 AMBER caddy[90444]: {"level":"info","ts":1741451643.0773396,"msg":"adapted config to JSON","adapter":"caddyfile"}
márc 08 17:34:03 AMBER caddy[90444]: {"level":"warn","ts":1741451643.0773568,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":23}
márc 08 17:34:03 AMBER caddy[90444]: {"level":"info","ts":1741451643.078957,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
márc 08 17:34:03 AMBER caddy[90444]: {"level":"info","ts":1741451643.0791345,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
márc 08 17:34:03 AMBER caddy[90444]: {"level":"info","ts":1741451643.079148,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
márc 08 17:34:03 AMBER caddy[90444]: {"level":"warn","ts":1741451643.0791545,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
márc 08 17:34:03 AMBER caddy[90444]: {"level":"info","ts":1741451643.079191,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000520880"}
márc 08 17:34:03 AMBER caddy[90444]: {"level":"info","ts":1741451643.079315,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc000520880"}
márc 08 17:34:03 AMBER caddy[90444]: Error: loading initial config: loading new config: loading http app module: provision http: loading pki app module: provision pki: provisioning CA 'local': decoding intermediate certificate PEM: input contained more than a single PEM block
márc 08 17:34:03 AMBER systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
márc 08 17:34:03 AMBER systemd[1]: caddy.service: Failed with result 'exit-code'.
márc 08 17:34:03 AMBER systemd[1]: Failed to start Caddy.

caddy logs

$ sudo -H -u caddy caddy run --environ --config /etc/caddy/Caddyfile
caddy.HomeDir=/var/lib/caddy
caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
caddy.AppConfigDir=/var/lib/caddy/.config/caddy
caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
caddy.Version=v2.9.1 h1:OEYiZ7DbCzAWVb6TNEkjRcSCRGHVoZsJinoDR/n9oaY=
runtime.GOOS=linux
runtime.GOARCH=amd64
runtime.Compiler=gc
runtime.NumCPU=8
runtime.GOMAXPROCS=8
runtime.Version=go1.24.1
os.Getwd=/home/system7

LANG=en_GB.UTF-8
XDG_CURRENT_DESKTOP=GNOME
COLORTERM=truecolor
XAUTHORITY=/run/user/1000/.mutter-Xwaylandauth.HIUZ22
LC_MEASUREMENT=hu_HU.UTF-8
LC_NUMERIC=hu_HU.UTF-8
LC_TIME=hu_HU.UTF-8
LC_PAPER=hu_HU.UTF-8
LC_MONETARY=hu_HU.UTF-8
PATH=/home/system7/.local/share/pnpm:/home/system7/bin:/usr/local/bin:/home/system7/.local/share/gem/ruby/3.0.0/bin:/home/system7/.local/bin:/home/system7/.config/composer/vendor/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/var/lib/flatpak/exports/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/opt/rocm/bin:/home/system7/.local/share/JetBrains/Toolbox/scripts:/home/system7/.local/share/JetBrains/Toolbox/scripts:/home/system7/.oh-my-zsh/custom/plugins/fzf-zsh-plugin/bin:/home/system7/.fzf/bin
LC_TELEPHONE=hu_HU.UTF-8
TERM=xterm-256color
LC_ADDRESS=hu_HU.UTF-8
DISPLAY=:0
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.7z=01;31:*.ace=01;31:*.alz=01;31:*.apk=01;31:*.arc=01;31:*.arj=01;31:*.bz=01;31:*.bz2=01;31:*.cab=01;31:*.cpio=01;31:*.crate=01;31:*.deb=01;31:*.drpm=01;31:*.dwm=01;31:*.dz=01;31:*.ear=01;31:*.egg=01;31:*.esd=01;31:*.gz=01;31:*.jar=01;31:*.lha=01;31:*.lrz=01;31:*.lz=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.lzo=01;31:*.pyz=01;31:*.rar=01;31:*.rpm=01;31:*.rz=01;31:*.sar=01;31:*.swm=01;31:*.t7z=01;31:*.tar=01;31:*.taz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tgz=01;31:*.tlz=01;31:*.txz=01;31:*.tz=01;31:*.tzo=01;31:*.tzst=01;31:*.udeb=01;31:*.war=01;31:*.whl=01;31:*.wim=01;31:*.xz=01;31:*.z=01;31:*.zip=01;31:*.zoo=01;31:*.zst=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:
MAIL=/var/mail/caddy
LOGNAME=caddy
USER=caddy
HOME=/var/lib/caddy
SHELL=/usr/sbin/nologin
SUDO_COMMAND=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
SUDO_USER=system7
SUDO_UID=1000
SUDO_GID=984
SUDO_HOME=/home/system7
2025/03/08 16:33:20.167	INFO	using config from file	{"file": "/etc/caddy/Caddyfile"}
2025/03/08 16:33:20.170	INFO	adapted config to JSON	{"adapter": "caddyfile"}
2025/03/08 16:33:20.170	WARN	Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies	{"adapter": "caddyfile", "file": "/etc/caddy/Caddyfile", "line": 23}
2025/03/08 16:33:20.171	INFO	admin	admin endpoint started	{"address": "localhost:2019", "enforce_origin": false, "origins": ["//[::1]:2019", "//127.0.0.1:2019", "//localhost:2019"]}
2025/03/08 16:33:20.172	INFO	http.auto_https	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2025/03/08 16:33:20.172	INFO	http.auto_https	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2025/03/08 16:33:20.172	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0xc0002c7a80"}
2025/03/08 16:33:20.172	WARN	http.auto_https	server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server	{"server_name": "srv1", "http_port": 80}
2025/03/08 16:33:20.172	INFO	tls.cache.maintenance	stopped background certificate maintenance	{"cache": "0xc0002c7a80"}
Error: loading initial config: loading new config: loading http app module: provision http: loading pki app module: provision pki: provisioning CA 'local': decoding intermediate certificate PEM: input contained more than a single PEM block

This is the point:

Error: loading initial config: loading new config: loading http app module: provision http: loading pki app module: provision pki: provisioning CA ‘local’: decoding intermediate certificate PEM: input contained more than a single PEM block

The question is: what does this exactly mean? What I configured wrong?

3. Caddy version:

$ caddy version
v2.9.1 h1:OEYiZ7DbCzAWVb6TNEkjRcSCRGHVoZsJinoDR/n9oaY=

4. How I installed and ran Caddy:

a. System environment:

$ systemctl --version
systemd 257 (257.4-1-arch)
+PAM +AUDIT -SELINUX -APPARMOR -IMA +IPE +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +BTF +XKBCOMMON +UTMP -SYSVINIT +LIBARCHIVE

$ fastfetch
                  -`                     system7@AMBER
                 .o+`                    -------------
                `ooo/                    OS: Arch Linux x86_64
               `+oooo:                   Host: 20XWCT01WW (ThinkPad X1 Carbon Gen 9)
              `+oooooo:                  Kernel: Linux 6.13.5-zen1-1-zen
              -+oooooo+:                 Uptime: 2 hours, 29 mins
            `/:-:++oooo+:                Packages: 1989 (pacman), 47 (flatpak-system), 1 (flatpak-user)
           `/++++/+++++++:               Shell: zsh 5.9
          `/++++++++++++++:              Display (CSO1404): 1920x1200 @ 60 Hz in 14" [Built-in]
         `/+++ooooooooooooo/`            DE: GNOME 47.5
        ./ooosssso++osssssso+`           WM: Mutter (Wayland)
       .oossssso-````/ossssss+`          WM Theme: adw-gtk3-dark
      -osssssso.      :ssssssso.         Theme: adw-gtk3-dark [GTK2/3/4]
     :osssssss/        osssso+++.        Icons: Papirus-Dark [GTK2/3/4]
    /ossssssss/        +ssssooo/-        Font: Inter (11pt) [GTK2/3/4]
  `/ossssso+/:-        -:/+osssso+-      Cursor: Bibata-Modern-Ice (24px)
 `+sso+:-`                 `.-/+oso:     Terminal: guake
`++:.                           `-/+/    CPU: 11th Gen Intel(R) Core(TM) i7-1185G7 (8) @ 4.80 GHz
.`                                 `/    GPU: Intel Iris Xe Graphics @ 1.35 GHz [Integrated]
                                         Memory: 11.30 GiB / 31.05 GiB (36%)
                                         Swap: 0 B / 20.00 GiB (0%)
                                         Disk (/): 128.85 GiB / 245.00 GiB (53%) - ext4
                                         Disk (/home): 1.15 TiB / 1.59 TiB (72%) - ext4
                                         Local IP (wlp0s20f3): 192.168.88.37/24
                                         Battery (5B10W13975): 49% [Discharging]
                                         Locale: en_GB.UTF-8

b. Command:

$ sudo caddy run --environ --config /etc/caddy/Caddyfile
$ sudo -H -u caddy caddy run --environ --config /etc/caddy/Caddyfile

c. Service/unit/compose file:

d. My complete Caddy config:

$ cat /etc/caddy/Caddyfile 
# The Caddyfile is an easy way to configure your Caddy web server.
#
# https://caddyserver.com/docs/caddyfile
#
# The configuration below serves a welcome page over HTTP on port 80.
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace the line below with your
# domain name.
#
# https://caddyserver.com/docs/caddyfile/concepts#addresses
{
	# Restrict the admin interface to a local unix file socket whose directory
	# is restricted to caddy:caddy. By default the TCP socket allows arbitrary
	# modification for any process and user that has access to the local
	# interface. If admin over TCP is turned on one should make sure
	# implications are well understood.
	#admin "unix//run/caddy/admin.socket"

	servers {
		protocols h1 h2 h2c h3
	}
        debug
}

http:// {
	# Set this path to your site's directory.
	root * /usr/share/caddy

	# Enable the static file server.
	file_server

	# Another common task is to set up a reverse proxy:
	# reverse_proxy localhost:8080

	# Or serve a PHP site through php-fpm:
	# php_fastcgi localhost:9000

	# Refer to the directive documentation for more options.
	# https://caddyserver.com/docs/caddyfile/directives
}

# Import additional caddy config files in /etc/caddy/conf.d/
import /etc/caddy/conf.d/*

$ cat /etc/caddy/conf.d/yc.caddy -p
yc-api.local {
    root * /var/www/yc-api/public
    encode gzip zstd
    file_server

    php_fastcgi unix//var/run/php-fpm/php-fpm.sock

    route /.well-known/mercure {
        mercure {
            redir / /.well-known/mercure
            transport_url local://local
            publisher_jwt <key>
            subscriber_jwt <key>
            cors_origins https://yc-api.local http://localhost http://127.0.0.1
            subscriptions
            heartbeat 25s
       }
    }

    log {
        output file /var/log/caddy/yc.access.log {
                roll_size 3MiB
                roll_keep 5
                roll_keep_for 48h
        }
        format console
    }
}
2

Lol, I spent 4.5 hours to figure out, but I solved in the end. Obviously after I post this topic.

So, I removed and re-created the caddy group and user. I removed all caddy’s home folder. (/var/lib/caddy) Last step was the daemon-reload and restart the service. After this I got another error

márc 08 18:21:25 AMBER sudo[109504]:    caddy : user NOT in sudoers ; PWD=/ ; USER=root ; COMMAND=/usr/bin/tee /etc/ca-certificates/trust-source/anchors/Caddy_Local_Authority_-_2025_ECC_Root_2614925>
márc 08 18:21:23 AMBER sudo[109504]: pam_unix(sudo:auth): auth could not identify password for [caddy]
márc 08 18:21:23 AMBER sudo[109504]: pam_unix(sudo:auth): conversation failed
márc 08 18:21:23 AMBER sudo[109504]: pam_unix(sudo:auth): auth could not identify password for [caddy]
márc 08 18:21:23 AMBER sudo[109504]: pam_unix(sudo:auth): conversation failed

So, I removed all caddy related certs here: /etc/ca-certificates/trust-source/anchors/ and run the trust command again:

$ sudo HOME=~caddy caddy trust

Then, it works as expected. The only problem that, I do not know what caused this bunch of error…