I'm trying to upgrade my V1 Config to a V2 config and not sure how to troubleshoot

1. Caddy version (caddy version):

V2.1

2. How I run Caddy:

I created a DigitalOcean droplet (ubuntu 18) using the Caddy Droplet from the Marketplace

a. System environment:

Ubuntu 18

b. Command:


c. Service/unit/compose file:

paste full file contents here

d. My complete Caddyfile or JSON config:

http://domains.mydomain.com { 
	errors /var/www/caddy_logs/caddy.log 
	fastcgi / 127.0.0.1:9001 php 
	index index.php 
	root /var/www/mydomain/public
    rewrite {
        to {path} {path}/ /index.php?{query}
    }
}

https://mydomain.com {

	email lee@mydomain.com
	tls {
		on_demand
	}
	errors /var/www/caddy_logs/caddy.log 
	fastcgi / 127.0.0.1:9001 php 
	index index.php 
	root /var/www/mydomain/public
    rewrite {
        to {path} {path}/ /index.php?{query}
    }
}


https://*.mydomain.com { 

	email lee@mydomain.com
	tls {
		on_demand
	}


	on_demand_tls {
		ask http://domains.mydomain.com/allowed
		interval 1h
		burst 30
	}



	errors /var/www/caddy_logs/caddy.log 
	fastcgi / 127.0.0.1:9001 php 
	index index.php 
	root /var/www/mydomain/public
    rewrite {
        to {path} {path}/ /index.php?{query}
    }
}

3. The problem I’m having:

I’m getting an error, but it’s not apparent how I can track down further errors to know what portion of my config file is causing the error. I know it’s not correct, I just cannot figure out where I’m wrong to even begin heading down the correct path.

4. Error messages and/or full log output:

/etc/systemd/system# systemctl reload caddy
Job for caddy.service failed because the control process exited with error code.
See "systemctl status caddy.service" and "journalctl -xe" for details.
root@mydomain:/etc/systemd/system# systemctl status caddy.service
● caddy.service - Caddy
   Loaded: loaded (/etc/systemd/system/caddy.service; enabled; vendor preset: enabled)
   Active: active (running) (Result: exit-code) since Mon 2020-08-17 20:51:28 UTC; 2h 28min ago
     Docs: https://caddyserver.com/docs/
  Process: 18563 ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile (code=exited, status=1/FAILURE)
 Main PID: 1286 (caddy)
    Tasks: 10 (limit: 4915)
   CGroup: /system.slice/caddy.service
           └─1286 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

5. What I already tried:

I’ve tried reducing the config file to be just a root definition and base domain, like the base config, and that works, it’s only when I start trying to map out the actions for the domains that I start running into problems.

Basically, I need the root domain to resolve with https, the domains.mydomain.com to return the domain status of 200 and for the *.mydomain.com I need it to ask domains.mydomain.com for the status.

This is a Laravel app that I had running on Caddy V1 and then shelved for a year. This is what I get for putting it on the backburner :smiley:

If there was a way to show me what line an error is on it would help me to debug this on my own, I’ve been bashing my head at it for an hour now and I feel like I’m spinning my wheels.

6. Links to relevant resources:

Use journalctl -u caddy to view the system logs for the caddy process. What do they say?

Also, just so you know, we have a rule to not redact logs or config files (except for credentials like passwords and API keys) because the actual domain names and other values are often important for troubleshooting. Thanks!

No journal files were found.

– No entries –

Also, just so you know, we have a rule to not redact logs or config files (except for credentials like passwords and API keys) because the actual domain names and other values are often important for troubleshooting. Thanks!

I hear ya Matt, I had to make the call as the domain that this is being used on is heavily monitored by a client and their marketing team. They POUR over Ahrefs everyday making sure that all the backlinks and mentions are exactly what they want their brand portraying.

This site is very well linked to and likely indexed often, it would likely be less than 12 hours before I had an unholy amount of emails flooding my inbox about why the second or third result for their domain is now a support forum post… I have permission for the site to be offline tonight while I upgrade, if I have to explain this post too to people with no technical background, my Tuesday is going to be rough.

I had to pick between asking for help and hoping that this rule wasn’t written in stone when there are good reasons and flattening my forehead more.

Have you seen the upgrade guide? Caddy v2 is a complete rewrite from Caddy v1, the configs are not compatible.

For the standard PHP app, this looks like the following in v2:

root * /var/www/mydomain/public
php_fastcgi 127.0.0.1:9001

On-Demand TLS configuration is now in global options (special block at the top of your Caddyfile with no site label):

Ok, to so you put me in the right direction, and I think I’m getting the hang of it.

First, to make our lives all easier, I took an unused domain of mine and spun up a second droplet using the Caddy Ubuntu Image from the market place.

This is my current Caddyfile, with domain of test server :wink:

{
	debug
}

offload.io {

	#root * /usr/share/caddy
	# Set this path to your site's directory.

	root * /var/www/offload-io/public
	php_fastcgi 127.0.0.1:9000

	# Enable the static file server.
	# file_server

}

In my /etc/php/7.2/fpm/pool.d/www.conf file I have

listen = 127.0.0.1:9000

; Set listen(2) backlog.
; Default Value: 511 (-1 on FreeBSD and OpenBSD)
;listen.backlog = 511

; Set permissions for unix socket, if one is used. In Linux, read/write
; permissions must be set in order to allow connections from a web server. Many
; BSD-derived systems allow connections regardless of permissions.
; Default Values: user and group are set as the running user
;                 mode is set to 0660
listen.owner = www-data
listen.group = www-data

This is a laravel app, located at /var/www/offload-io/public

root@offload-io:/var/www/offload-io/public# ls -la
total 64
drwxr-x---  6 www-data www-data 4096 Aug 18 01:51 .
drwxr-x--- 13 www-data www-data 4096 Aug 18 00:52 ..
-rw-r-----  1 www-data www-data 6148 Aug 18 00:15 .DS_Store
-rw-r-----  1 www-data www-data  603 Aug 18 00:15 .htaccess
drwxr-x---  3 www-data www-data 4096 Aug 18 00:15 assets
drwxr-x---  2 www-data www-data 4096 Aug 18 00:15 css
-rw-r-----  1 www-data www-data  434 Aug 18 00:15 favicon-16x16.png
-rw-r-----  1 www-data www-data  846 Aug 18 00:15 favicon-32x32.png
-rw-r-----  1 www-data www-data 5430 Aug 18 00:15 favicon.ico
drwxr-x---  2 www-data www-data 4096 Aug 18 00:15 img
-rw-r-----  1 www-data www-data 1823 Aug 18 01:51 index.php
drwxr-x---  2 www-data www-data 4096 Aug 18 00:15 js
-rw-r-----  1 www-data www-data   71 Aug 18 00:15 mix-manifest.json
-rw-r-----  1 www-data www-data   24 Aug 18 00:15 robots.txt

I can get the site to load now, sort of, it issues a cert and all that, but it still serves up a white screen without anything php being rendered. Even replacing the index.php in public with phpinfo still throws a white page.

At best, the blatantly obvious fix that I’m sure you can spot a mile away will stand to help someone in the future. The only other references I can find the OP deleted their solution :confused: :man_shrugging:

You’ll need file_server enabled to serve static assets (JS/CSS/images). Just uncomment that.

What do your logs say? That config looks correct to me. I think you can run journalctl -u caddy to see the logs.

When I uncomment that line I get a server 403 error (as set now)

No journal files were found.
– No entries –

I’m… confused.

I don’t use the DO droplet so unfortunately I might not totally be of help… but you are running it as a systemd service, right? What does /etc/systemd/system/caddy.service look like? We’ll definitely need to find where the logs are going, that’s pretty important… :thinking:

I don’t know why Caddy would be serving you a 403 if you enable file_server.


# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

1 Like

Oh, actually I have a hunch - I’m not sure if the DO droplet was updated since to make sure the caddy user is added to the www-data group. Could you check that? The 403 might be from a filesystem permissions problem.

You can run id caddy to check.

root@offload-io:/var/www/offload-io/public# id caddy
uid=999(caddy) gid=999(caddy) groups=999(caddy),33(www-data)

:frowning_face:

Well that is correct, so I’m even more confused.

Looping in @Mohammed90 who maintains the DO droplet, he’ll probably have some ideas.

Okay here’s something else you can try. Shut down the service for now with sudo systemctl stop caddy, then in a 2nd terminal session, switch to the caddy user with su caddy, then run Caddy directly with /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile. That way, you should be able to see your logs in stdout for the time being until we figure out what’s going on with the service and journalctl.

No Bueno.

root@offload-io:~# su caddy
This account is currently not available.

Ah, that’s cause the caddy user has nologin as the default shell. This should do:

su -l caddy -s /bin/bash

Got something,

Added this to my caddyfile and started the service again

    log {
        output file /var/www/caddylog.log
        level DEBUG
    }

This is what it output now.

{"level":"error","ts":1597720347.083793,"logger":"http.log.access.log0","msg":"handled request","request":{"method":"GET","uri":"/","proto":"HTTP/2.0","remote_addr":"70.75.165.186:62610","host":"offload.io","headers":{"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Dest":["document"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["en-US,en;q=0.9,la;q=0.8,nl;q=0.7"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"h2","proto_mutual":true,"server_name":"offload.io"}},"common_log":"70.75.165.186 - - [18/Aug/2020:03:12:27 +0000] \"GET / HTTP/2.0\" 502 0","duration":0.001496647,"size":0,"status":502,"resp_headers":{"Server":["Caddy"]}}
1 Like

Right, so 502 means Caddy couldn’t connect to your proxy backend (in this case, PHP FPM). Are you sure it’s running correctly?

Winner winner.

Ok, so I have a laravel error being thrown now, I have no idea what the solution was though, sorry googler…

I think the error was resolved when caddy had somewhere to log things, is the log required???

1 Like

And enabling file_server no longer throws a permission error.

1 Like

It shouldn’t be required, but it’s definitely helpful to have access logs somewhere.

I’m still pretty confused about journalctl though… does that suddenly work now? :joy: