Caddy 2.9.0 breaking change?

Help
1

1. The problem I’m having:

upgraded from caddy 2.8.4 to caddy 2.9.0 and caddy won’t start. error logging doesn’t give much of a hint. was there a breaking change between versions regarding config operators?

2. Error messages and/or full log output:

2025/01/01 01:00:10.971	e[34mINFOe[0m	shutting down apps, then terminating	{"signal": "SIGTERM"}
2025/01/01 01:00:10.972	e[33mWARNe[0m	exiting; byeee!! 👋	{"signal": "SIGTERM"}
2025/01/01 01:00:10.973	e[34mINFOe[0m	http	servers shutting down with eternal grace period
2025/01/01 01:00:15.007	e[34mINFOe[0m	http.acme_client	got renewal info	{"names": ["facts.eiphax.tech"], "window_start": "2025/01/11 14:31:28.000", "window_end": "2025/01/13 14:31:28.000", "selected_time": "2025/01/11 23:20:17.000", "recheck_after": "2025/01/01 07:00:15.007", "explanation_url": ""}
2025/01/01 01:00:15.007	e[34mINFOe[0m	tls.cache.maintenance	updated ACME renewal information	{"identifiers": ["facts.eiphax.tech"], "cert_hash": "f170e8365490e6ffb796e40820fc75a57145a92cbca4dbca09893a8bc0604c3a", "ari_unique_id": "nytfzzwhT50Et-0rLMTGcIvS1w0.A1Gv6QgRZHDwPB6sAbZcDl3I", "cert_expiry": "2025/02/11 14:11:58.000", "selected_time": "2025/01/13 08:01:36.000", "next_update": "2025/01/01 07:00:15.007", "explanation_url": ""}
2025/01/01 01:00:15.076	e[34mINFOe[0m	tls.cache.maintenance	advancing OCSP staple	{"identifiers": ["admin.uwu.tax"], "from": "2025/01/04 12:29:58.000", "to": "2025/01/08 00:59:58.000"}

there’s a SIGTERM here but i don’t know where it’s coming from.

3. Caddy version:

2.9.0, downgraded to 2.8.4 now

4. How I installed and ran Caddy:

a. System environment:

ubuntu lts 22.04 via systemd

b. Command:

via service unit

c. Service/unit/compose file:

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target
OnFailure=crashmailserv@%n.service

StartLimitIntervalSec=10
StartLimitBurst=5

[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
ExecStartPost=+/bin/systemctl start upmailserv@%n.service
StandardOutput=file:/var/www/logs/caddy-so.log
StandardError=file:/var/www/logs/caddy-se.log
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE
Environment=DO_AUTH_TOKEN=a52dfe40dbc9288817ab5e134f6ff113891e4cbf7e67a1574c0810bc6b56b461
Restart=always

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

{
	log {
		format console
		output file /var/www/logs/caddy-access.log {
			roll_size 25mb
			roll_keep 20
			roll_keep_for 720h
		}
	}
}

(gen) {
	encode gzip
	uri strip_suffix .html
	try_files {path} {path}.php {path}.html index.php index.html =404
	php_fastcgi unix//var/run/php/php8.3-fpm.sock {
		try_files {path} {path}.php index.php =404
	}
	handle /xmlrpc.php {
		reverse_proxy :6969
	}
	file_server
}

(sticky) {
	handle /wp* {
		reverse_proxy :6969
	}
	handle /.* {
		reverse_proxy :6969
	}
	handle /env* {
		reverse_proxy :6969
	}
	handle /xmlrpc.php {
		reverse_proxy :6969
	}
}

(e-gen) {
	encode gzip
	uri strip_suffix .html
	try_files {path} {path}.php {path}.html index.php index.html
	php_fastcgi unix//var/run/php/php8.3-fpm.sock {
		try_files {path} {path}.php index.php
	}
	file_server
	respond /seed/* "Gone" 410 {
		close
	}
	respond /nh/* "Gone" 410 {
		close
	}
}

(dns) {
	tls {
		dns digitalocean redacted
	}
}

(header-gen) {
	log {
		format console
		output file /var/www/logs/caddy-access.log {
			roll_size 25mb
			roll_keep 20
			roll_keep_for 720h
		}
	}
	header {
		-Server
		Permissions-Policy interest-cohort=()
		Strict-Transport-Security "max-age=31536000; includeSubdomains; preload"
		X-XSS-Protection 1; mode=block
		X-Content-Type-Options nosniff
		X-Frame-Options SAMEORIGIN
		Content-Security-Policy "default-src 'self' 'unsafe-inline' data: *.google-analytics.com *.fontawesome.com *.googleapis.com *.gstatic.com *.jsdelivr.net *.bootstrapcdn.com *.eiphax.tech *.google.com secure.gravatar.com; script-src 'self' 'unsafe-eval' 'unsafe-inline' data: secure.gravatar.com *.eiphax.tech *.google.com *.gstatic.com *.googletagmanager.com *.google-analytics.com *.google-analytics.com *.fontawesome.com *.googleapis.com *.jsdelivr.net *.bootstrapcdn.com;"
	}
}

(bfm-header) {
	header {
		-Server
		Permissions-Policy interest-cohort=()
		Strict-Transport-Security "max-age=31536000; includeSubdomains; preload"
		X-XSS-Protection 1; mode=block
		X-Content-Type-Options nosniff
		X-Frame-Options SAMEORIGIN
		Cache-Control no-cache, must-revalidate
		Content-Security-Policy "default-src 'self' 'unsafe-inline' data: *.fontawesome.com *.googleapis.com *.gstatic.com *.jsdelivr.net *.bootstrapcdn.com *.eiphax.tech *.google.com i.imgur.com; script-src 'self' 'unsafe-eval'"
	}
}

(dead) {
	respond * "410 Gone" 410 {
		close
	}
}

eiphax.tech {
	import sticky
	header {
		-Server
		Permissions-Policy interest-cohort=()
		Strict-Transport-Security "max-age=31536000; preload"
		X-XSS-Protection 1; mode=block
		X-Content-Type-Options nosniff
		X-Frame-Options SAMEORIGIN
		Content-Security-Policy "default-src 'self' 'unsafe-inline' data: *.fontawesome.com *.googleapis.com *.gstatic.com *.jsdelivr.net *.bootstrapcdn.com *.eiphax.tech *.google.com secure.gravatar.com; script-src 'self' 'unsafe-eval' 'unsafe-inline' data: secure.gravatar.com *.eiphax.tech *.google.com *.gstatic.com *.googletagmanager.com *.google-analytics.com *.google-analytics.com *.fontawesome.com *.googleapis.com *.jsdelivr.net *.bootstrapcdn.com;"
	}
	handle {
		import e-gen
		root * /var/www/eipmain/webroot
	}
}

facts.eiphax.tech {
	import header-gen
	import sticky
	handle {
		root * /var/www/eipmain/facts
		import gen
	}
}

laundry.eiphax.tech {
	import header-gen
	import sticky
	handle {
		root * /var/www/eipmain/webroot/laundry
		import gen
	}
}

blog.eiphax.tech {
	import header-gen
	root * /var/www/blog
	import gen
}

bytes.eiphax.tech {
	import header-gen
	root * /var/www/bytes
	import gen
}

album.eiphax.tech {
	import header-gen
	import sticky
	handle {
		root * /var/www/lychee/public
		import gen
	}
}

secrets.eiphax.tech {
	import sticky
	header {
		-Server
		Permissions-Policy interest-cohort=()
		Strict-Transport-Security "max-age=31536000; includeSubdomains; preload"
		X-XSS-Protection 1; mode=block
		X-Content-Type-Options nosniff
		X-Frame-Options SAMEORIGIN
		Content-Security-Policy "default-src 'none'; base-uri 'self'; form-action 'none'; manifest-src 'self'; connect-src * blob:; script-src 'self' 'unsafe-eval'; style-src 'self'; font-src 'self'; frame-ancestors 'none'; img-src 'self' data: blob:; media-src blob:; object-src blob:; sandbox allow-same-origin allow-scripts allow-forms allow-popups allow-modals allow-downloads"
	}
	handle {
		root * /var/www/eipbin
		import gen
	}
}

3ds.eiphax.tech {
	@ytbad {
		header Referer *youtube.com*
	}
	@ytbad2 {
		header Referer *youtu.be*
	}
	rewrite @ytbad /youtube.php
	rewrite @ytbad2 /youtube.php
	import header-gen
	import sticky
	handle {
		root * /var/www/eipmain/3ds
		import gen
	}
}

http://wiiu.eiphax.tech {
	@ytbad {
		header Referer *youtube.com*
	}
	@ytbad2 {
		header Referer *youtu.be*
	}
	rewrite @ytbad /youtube.php
	rewrite @ytbad2 /youtube.php
	import sticky
	handle {
		root * /var/www/eipmain/wiiu
		import gen
	}
}

https://wiiu.eiphax.tech {
	@ytbad {
		header Referer *youtube.com*
	}
	@ytbad2 {
		header Referer *youtu.be*
	}
	rewrite @ytbad /youtube.php
	rewrite @ytbad2 /youtube.php
	import header-gen
	import sticky
	handle {
		root * /var/www/eipmain/wiiu/resources
		import gen
	}
}

nx.eiphax.tech {
	@ytbad {
		header Referer *youtube.com*
	}
	@ytbad2 {
		header Referer *youtu.be*
	}
	rewrite @ytbad /youtube.php
	rewrite @ytbad2 /youtube.php
	import header-gen
	import sticky
	handle {
		root * /var/www/eipmain/nx
		import gen
	}
}

nintendohomebrew.com {
	header {
		Strict-Transport-Security "max-age=31536000; preload"
		X-XSS-Protection "1; mode=block"
		X-Content-Type-Options "nosniff"
		X-Frame-Options "SAMEORIGIN"
	}
	handle_errors {
		rewrite * /{http.error.status_code}
		reverse_proxy https://http.cat {
			header_up Host http.cat
		}
	}
	import sticky
	handle {
		root * /var/www/eipmain/nh
		respond /seed/* "Gone" 410 {
			close
		}
		import gen
	}
}

http://bhax.nintendohomebrew.com {
	import sticky
	handle {
		root * /var/www/bhax/web/nbhax
		try_files {path} {path}.html
		encode gzip
	}
}

bfm.nintendohomebrew.com, seedminer.hacks.guide {
	import bfm-header
	import sticky
	handle {
		root * /var/www/eipmain/nh/seed
		reverse_proxy localhost:8082
		encode gzip
		file_server
	}
}

http://part1dumper.nintendohomebrew.com https://part1dumper.nintendohomebrew.com {
	reverse_proxy localhost:8081
}

shitpost.lol {
	import header-gen
	import sticky
	handle {
		root * /var/www/sp
		import gen
	}
}

hacc.me please.hacc.me {
	import header-gen
	import sticky
	handle {
		root * /var/www/hacc
		import gen
	}
}

friigaemsworld.com {
	import header-gen
	import sticky
	handle {
		root * /var/www/frigam
		import gen
	}
}

uwu.tax {
	reverse_proxy localhost:4444
	php_fastcgi unix//var/run/php/php8.3-fpm.sock {
		try_files {path} {path}.php index.php =404
	}
}

admin.uwu.tax {
	import header-gen
	import sticky
	handle {
		root * /var/www/uwu-admin
		import gen
	}
}

conversation.id {
	import header-gen
	import sticky
	handle {
		root * /var/www/conv
		import gen
	}
}
puebes.com {
	import header-gen
	import sticky
	handle {
		root * /var/www/puebes
		import gen
	}
}

four.family {
	import header-gen
	import sticky
	handle {
		root * /var/www/four
		import gen
	}
}

230421.wedding {
	import header-gen
	import sticky
	handle {
		root * /var/www/wedding
		import gen
	}
}

photos.230421.wedding {
	import header-gen
	import sticky
	handle {
		root * /mnt/wedding/lychee/public
		import gen
	}
}

durriesberg.biz {
	import header-gen
	root * /var/www/dberg
	import gen
}

news.eiphax.tech {
	import header-gen
	root * /var/www/news
	import gen
}

blep.co {
	import dead
}

photos.four.family {
	import header-gen
	import sticky
	handle {
		root * /mnt/charlie/public
		import gen
	}
}

photos.moonaglio.wedding photos.agliomoon.wedding moonaglio.wedding agliomoon.wedding {
	import header-gen
	import sticky
	handle {
		root * /var/www/moonaglio/public
		import gen
	}
}

garden.eiphax.tech {
	import header-gen
	import sticky
	handle {
		redir https://blog.eiphax.tech/?p=269 permanent
	}
}

rules.eiphax.tech {
	import header-gen
	import sticky
	handle {
		root * /var/www/eipmain/webroot/rules
		import gen
	}
}

mkey.nintendohomebrew.com {
	reverse_proxy localhost:5555
}

tarpit.eiphax.tech {
	import header-gen
	import sticky
	handle {
		import gen
		root * /var/www/temp
	}
}

logs.eiphax.tech {
	import header-gen
	import sticky
	handle {
		basic_auth {
			crc $2a$14$JInGtTqZ0d5MneOBIKZAgu61JFiZHP3UlPxVSAOLsvLovgmwii8Hq
		}
		root * /var/www/logs
		@logs {
			path *.log
		}
		header @logs {
			Content-Type text-plain
			Content-Disposition inline
		}
		import gen
	}
}

expertfinancialservices.org {
	import header-gen
	root * /var/www/finance
	import gen
}

totk.eiphax.tech {
	import dead
}

words.eiphax.tech {
	import header-gen
	root * /var/www/story
	import gen
}

story.eiphax.tech {
	import dead
}

store.eiphax.tech {
	import header-gen
	import sticky
	handle {
		rewrite /api /api/
		@api-rewrite path_regexp api ^/api/(.*)
		rewrite @api-rewrite /webservice/dispatcher.php?url={re.api.1}
		@img-rewrite path_regexp img ^/images_ie/?([^/]+)\.(jpe?g|png|gif)$
		rewrite @img-rewrite /js/jquery/plugins/fancybox/images/{re.img.1}.{re.img.2}
		root * /var/www/store
		encode gzip
		uri strip_suffix .php
		php_fastcgi unix//var/run/php/php8.3-fpm.sock {
			try_files {path} {path}.php index.php =404
		}
		handle /xmlrpc.php {
			reverse_proxy :6969
		}
		file_server
	}
}

passvault.eiphax.tech {
	import header-gen
	import sticky
	reverse_proxy localhost:3284 {
		header_up X-Real-IP {remote_host}
	}
}

oycrunk.eiphax.tech {
	import header-gen
	root * /var/www/oycrunk
	import gen
}

fordspencer.au {
	import header-gen
	root * /var/www/fordspencer
	import gen
}

fusee.nintendohomebrew.com {
	import header-gen
	root * /var/www/fusee
	import gen
}

soap.nintendohomebrew.com {
	import header-gen
	import sticky
	reverse_proxy localhost:8085 {
		header_up X-Real-IP {remote_host}
	}
}

fusee.eiphax.tech {
	redir https://fusee.nintendohomebrew.com permanent
}

mkey.eiphax.tech {
	redir https://mkey.nintendohomebrew.com permanent
}

skater.nintendohomebrew.com {
	import header-gen
	root * /var/www/skater/docs
	import gen
}

luma3ds.com {
	import header-gen
	root * /var/www/luma
	import gen
}

bin.eiphax.tech {
	redir https://secrets.eiphax.tech/ permanent
}

ethalism.org {
	import header-gen
	root * /var/www/ethalism
	import gen
}
2

Nothing that’s obvious

The log is redacted. Can you show more of the logs?

3

it hasn’t been redacted by me, this was the most detailed log output i could find. i have my logs set up to redirect to .log files, not stderr/stdout. i also checked journalctl and the only logs there were the systemd logs about restarting, so i don’t see that that would be particularly useful.

4

Hmm… So there’s a wild SIGTERM causing Caddy to shutdown. We haven’t done anything that’d randomly shutdown Caddy. There’s a way to trace the source of a signal. Can you try the methods here?

5

yes, i’ll look into it. i’ve downgraded to 2.8.4 for now and my prod environment is running fine so i may be a while. the only reason i was quick to blame a config breaking change was because the only thing that happened was caddy was upgraded to 2.9.0, nothing about the config was changed.

6

On my end Caddy 2.9.0’s breaking change was that it is now strict about doing a chmod on the log files, and it simply quits if it failed. Previously, my Caddy was able to read & write its logs thanks to ACLs. Now the user Caddy runs at essentially has to be the owner of the log directory, and all the log files. Or I guess maybe the directory doesn’t have to be, as long as it remains writable? In any case, I’d try moving everything inside a dedicated /var/www/logs/caddy subfolder instead of your current /var/www/logs.

7

Thanks for the tip! I guess it’s this PR causing you the trouble.

I’ll investigate what can be done

8

Yes, that was it. As for myself, saying it is causing me trouble is an overstatement. I’d say it was a very minor speed-bump in getting Caddy 2.9.0 running after the upgrade. I changed the ownership of the log files / directory, and moved on with my life. Nothing really needs to be patched, as far as I’m concerned, and I’m thankful for all the enhancements, and the people working on them. Maybe a notice that this could cause startup issues where previously there weren’t any could be helpful.

2 Likes
9

As far as I understand, the change wasn’t meant to be breaking. It was an oversight.

Thank you for the kind words :blush:

1 Like
10

Why does it fail? Do you have any error messages?

11

Quite simple, Caddy now always tries to chmod log files, even if the new mode option was not set (because it effectively defaults to 0600 regardless), and curiously even if the file’s mode is already matching that.
The error message is like this:
Error: loading initial config: loading new config: setting up custom log 'log0': opening log writer using &logging.FileWriter{Filename:"/var/log/caddy/try-and-write-this", Mode:0x0, Roll:(*bool)(nil), RollSizeMB:0, RollCompress:(*bool)(nil), RollLocalTime:false, RollKeep:0, RollKeepDays:0}: chmod /var/log/caddy/try-and-write-this: operation not permitted

In this case, I made /var/log/caddy/try-and-write-this be owned by root, but gave the user Caddy runs as write permissions through ACLs.

sudo mkdir -p /var/log/caddy &&
sudo touch /var/log/caddy/try-and-write-this &&
sudo chown -R 0:0 /var/log/caddy &&
sudo chmod 0600 /var/log/caddy/try-and-write-this

On macOS then do:
sudo chmod +a 'user:caddy allow read,write' /var/log/caddy/try-and-write-this
Or the Linux way:
sudo setfacl -m u:caddy:rw /var/log/caddy/try-and-write-this

Really just make Caddy unable to chmod the file. It used to not care at all about it in previous versions, now it results in an immediate failure. But obviously not a big deal. You just make sure the log files are actually owned by the Caddy user, so that it is allowed to do a chmod on them, and don’t rely on ACLs. Still it’s a bit weird it insists on attempting the chmod even when the file is already the desired 0600.

In my real world situation I obviously wasn’t trying to intentionally screw over Caddy, only while I was building my configuration, I would often sometimes switch between running Caddy as its final intended and separate user as a background daemon, and sometimes in the foreground tweaking the configuration running under my regular user account, so I didn’t have to authenticate all the time. And simply used ACLs to give the necessary rights to either user.

1 Like
12

I see; so an external system permission was set to disallow Caddy’s ability to change the permissions on its file, because it didn’t own its log file, so when we started chmod’ing it in 2.9, it now gets an error.

I guess… I would have expected Caddy to own its log files. I am not sure if that is a breaking change as much as exposing a misconfiguration of the system. Because surely you’d want the more restrictive 0600 permissions. Why disallow Caddy the ability to tighten down its log files? (EDIT: I just realized you said that the file was already 0600. That does seem like an unnecessary chmod, indeed; but now I have to revisit the PR. Why are we chmod’ing the file if it’s already the desired permission?)

1 Like
13

I opened a PR here to only chmod if the permission bits differ… maybe you could double-check my logic:

What do you think?

1 Like
14

Looks great to me, and this alone will once again let you switch back and forth between running Caddy with the same config file under different user accounts, with ACLs set up to allow either user to read/write all the necessary files accordingly. Handy when still learning the syntax and Caddy’s capabilities. Thank you!

And of course I would’ve been fine with Caddy 2.9.0 remaining as is. It was a very minuscule issue.

Now on my perfectionist side, I’ll admit I’d have preferred if this new mode feature was implemented in a way as purely additive, and nothing was changed for users who did not actively opt into using it by actually adding the new mode option to their configs. That is, if it defaulted to an “off” state instead, where as in all former versions, Caddy would default to creating new log files as mode 0600 but did not care to verify or update their mode afterwards. Unless, of course, the user actually expressed their preference to do so by invoking the cool new feature.

Please consider this as a thought experiment only, and don’t spend any more time on it though. :slight_smile: I was, and remain, already happy with Caddy 2.9.0 as is.

And lest we forget: I’m not the OP of this topic, and only chimed in with what broke for me on the update, in case the original poster had the same issue. We haven’t got confirmation that this is the case. Might be entirely unrelated.

1 Like
15

hi everyone, thanks and sorry for all the effort. I haven’t had time to go back and look into it as prod is running fine on 2.8.4 and yesterday was busy AF but I dare say we’ve hit the nail on the head with logs.

my caddy user might not own its own logs because I have them redirected to logs in /var/www/ which every so often gets chown -r www-data:www-data which itself is a kludge to solve an unrelated issue with an unrelated application.

this hasn’t been a problem until now but I suppose it is unexpected behaviour, so not a fault of caddy whatsoever, more another example of how users can screw things up in new and fantastic ways.

I also appreciate the continued work and improvements on caddy and this post wasn’t intended as me having a whinge, was simply trying to help nail down what might have been unintended behaviour/breaking changes that may affect others.

2 Likes
16

Ah, well thanks for being patient with us and for your grace in understanding. :slight_smile:

2 Likes
17

I just had this, my temp workaround to get the server back up ASAP was to disable all logging whilst I did some googling.

I’m on Oracle Linux 8 (Red Hat EL), example solution YMMV

# Set ownership to caddy user and group
sudo chown caddy:caddy /var/log/caddy

# Set appropriate permissions (755 for directory)
sudo chmod 755 /var/log/caddy

# Set correct SELinux context for logs
sudo semanage fcontext -a -t httpd_log_t "/var/log/caddy(/.*)?"
sudo restorecon -Rv /var/log/caddy
3 Likes
18

Hi Matt,

chown is already caddy:caddy and chmod is already 755.

But this is great, as Caddyfile refers to /var/www/log.

Hence I applied the same ownership that you suggested and voilà, it starts filling up.

Than you.
Regards.

1 Like
19

2.9.1 has a patch that handles unexpected permissions more gracefully…

3 Likes