Simple PHP + Caddy Installation : Browser Throws 502

KaK · March 15, 2021, 12:38pm

1. Caddy version (`caddy version`):

v2.3.0

2. How I run Caddy:

#Installed Caddy as 'root' user on Digitalocean Droplet. Simply logged in and typed following commands

apt update && apt upgrade
sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo apt-key add -
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee -a /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

a. System environment:

Ubuntu 20.10

b. Command:

Caddy runs automatically after following above instructions. I use 

caddy reload

after updating the Caddyfile

I’ve also installed PHP 8.0.3 (PHP-FPM) as root:

apt install php
apt install php8.0-common php8.0-fpm php8.0-curl php8.0-mysql php8.0-gd php8.0-mbstring php8.0-zip php8.0-xml php8.0-bcmath

c. Service/unit/compose file:

paste full file contents here

d. My complete Caddyfile or JSON config:

jatra.info {
        root * /var/jatra/public
        php_fastcgi unix//run/php/php7.4-fpm.sock
        file_server
}

3. The problem I’m having:

I’m trying to get PHP to work with Caddy. However, my browser throw 502 error.

4. Error messages and/or full log output:

This is not available. I’m not sure how to get log output.

5. What I already tried:

Checked all the paths are correct in the Caddyfile
Following multiple links that talk about installing php + caddy; but couldn’t get it working.
Tried playing with permissions. I’m not sure what I really did here; but nothing worked.

My guess is that I’ve messed up with permissions somewhere. Is there any tested tutorial I could follow that tells me how to install PHP + Caddy that just works?

6. Links to relevant resources:

Install — Caddy Documentation

francislavoie · March 15, 2021, 1:06pm

If you installed PHP 8.0, that’s not the right path to the unix socket. The example you probably copied from used PHP 7.4. Please check your php-fpm configuration for where it places the unix socket file.

KaK · March 15, 2021, 1:09pm

My mistake. Changed to

jatra.info {
	root * /var/jatra/public
	php_fastcgi unix//run/php/php8.0-fpm.sock
	file_server
}

Now the browser throws 500 error.

francislavoie · March 15, 2021, 1:12pm

A 500 will be because of PHP or application misconfiguration. Check your PHP app logs or php-fpm logs. The 500 is not due to Caddy.

KaK · March 15, 2021, 1:45pm

Yes, I suspect that as well. However, I’m wondering if this is a permissions issue. There could be something that Caddy is unable to access because of permissions.

Would really appreciate if someone can point me to a tutorial that works (PHP-FPM + Caddy).

Addendum:

I see some tutorials say -

Are these steps required?

sudo mkdir /var/www
sudo chown www-data:www-data /var/www
sudo nano /var/www/index.html

Also, port binding for Caddy. Wondering if this is really required-

sudo setcap cap_net_bind_service=+ep /usr/local/bin/caddy

Documentation doesn’t mention this.

What would be the minimum permissions required for Caddy?

francislavoie · March 15, 2021, 2:05pm

Caddy, when running as a systemd service, runs with the caddy user. So that user needs to be able to read the files in your site. You need to make sure php-fpm can also read those files.

That’s only required if you’re running Caddy as a user which cannot bind to ports 80 and 443, which is not the case for you because you’re running as a systemd service which sets that up correctly by default.

But anyways, a 500 error is not a problem with Caddy, that is a repsonse served by your PHP app or php-fpm. Caddy is just proxying that response back to your browser.

KaK · March 15, 2021, 2:43pm

@francislavoie - So my PHP files need to be accessible by caddy or www-data user; is that correct?

francislavoie · March 15, 2021, 2:58pm

Probably both, if php-fpm uses the www-data user. In which case, you can add the caddy user to the www-data group.

KaK · March 16, 2021, 11:58am

@francislavoie - thanks! I got it working, finally. I’ve a question though. I’m unable to figure out which user is Caddy making use of (not sure if this is right question, but asking anyway!) - www-data OR caddy ?

I’m a newbie and processes and permissions have always been my weak point.

Also -

Can this user be changed to say “jack”?
I installed Caddy as root user. I thought this would make caddy access any damn file on the server regardless of who owns it. But that’s not clearly the case.

I had to change the owner of the storage folder to www-data to make it work for Laravel. Which means, caddy had issues writing to the log files.

I tried a few lame tricks like adding my ‘jack’ user to ‘caddy’ and ‘www-data’ groups; but that didn’t do the trick. Also tried adding caddy and www-data to the ‘jack’ group (yes, i’ve a group with the same name). But it doesn’t work.

As of now, my entire laravel installation is owned by jack user.

Would really appreciate some help in figuring this out.

francislavoie · March 16, 2021, 2:51pm

Caddy runs as the caddy user, as configured in the systemd service file, which you can find installed at /lib/systemd/system/caddy.service, and php-fpm will probably be running as the www-data user, i.e. whatever it’s configured to run as in its config.

The user you use to install apt packages is unrelated to how applications run. You typically need to install things as root or as a sudoer (users who can run sudo, based on a group assignment, aka “administrators”), because the apt program needs to be able to write files wherever it needs to properly install packages.

Yeah, you could run sudo systemctl edit caddy and this would present you with an empty editor window that lets you configure overrides for your systemd config. I wouldn’t recommend it though, this is more of an advanced topic, and best to leave editing systemd configs until you’re more comfortable with how it all works.

If you run the commands id caddy and id jack, what do you see now? It should show you your user with its ID, and all the groups they’re assigned to.

It sounds correct to me to add both of those users to the www-data group and change group ownership of all the files in /var/www to www-data, and make sure your storage dir is group also writable (do you understand how permission bits work?)

KaK · March 17, 2021, 6:04am

@francislavoie - That was super helpful! I now have a much better idea of how things work. Definitely helpful in my next step with Caddy.

My next challenge is to have wildcard subdomains working and eventually be able to host multiple top level domains pointed to relevant subdomains.

What I’ve done so far (testing this with a new test domain layoff.wtf) -

Step 1 : Created an A record to point *.layoff.wtf to my server IP

Step 2: Updated my Caddyfile to the following -

{
        on_demand_tls {
                ask https://layoff.wtf/caddy/ask # This currently returns 200 response for everything (it's a test domain) 
                interval 2m
        }

}

https:// {
        tls {
                on_demand
        }
}

layoff.wtf {

        root * /var/www/caddy/public

        file_server

        php_fastcgi unix//run/php/php8.0-fpm.sock
}

I currently have the main layoff.wtf domain working correctly.

However -

If I change layoff.wtf block to layoff.wtf, *.layoff.wtf; browser throws SSL error. Apparently, I need some more configuration to make wildcard subdomain work.

layoff.wtf, *.layoff.wtf {

        root * /var/www/caddy/public

        file_server

        php_fastcgi unix//run/php/php8.0-fpm.sock
}

I checked the data directory $HOME/.local/share/caddy. The caddy directory did not exist. I therefore created it with following permissions -

drwx------ 5 root root 4096 Mar 17 05:16 .
drwxr-xr-x 3 root root 4096 Mar 16 08:05 ..
drwxrwxr-x 2 root root 4096 Mar 17 05:16 caddy
drwxr-xr-x 2 root root 4096 Mar 16 08:21 composer
drwx------ 2 root root 4096 Mar 16 08:05 nano

Am I on the right track with this? What am I missing out on?

francislavoie · March 17, 2021, 6:16am

Yep. You must use the ACME DNS challenge to get a wildcard certificate:

The other option is to use on_demand, but you haven’t configured Caddy such that on_demand is used for *.layoff.wtf.

You should probably do this:

{
	on_demand_tls {
		ask https://layoff.wtf/caddy/ask
	}
}

https:// {
	tls {
		on_demand
	}

	root * /var/www/caddy/public
	php_fastcgi unix//run/php/php8.0-fpm.sock
	file_server
}

This will serve any domain allowed by your ask endpoint to have a certificate issued. It’s very important that you limit the domains to only ones you know, otherwise you open yourself up to DDOS from attackers. They can point a wildcard domain to your IP, then make requests like foo1.example.com, then foo2.example.com and so on continually and your server will keep issuing certificates for each of those domains, unless your ask endpoint denies it. This could fill your storage, or pin you at the rate limits.

The caddy user’s $HOME is /var/lib/caddy. Files will be stored there, not in your user’s $HOME.

KaK · March 17, 2021, 8:55am

Thanks! I’ve updated my caddyfile as follows:

  GNU nano 5.2                                                             Caddyfile                                                                       
{
        on_demand_tls {
                ask https://layoff.wtf/caddy/ask
                interval 2m
        }

}

https:// {

        tls {
                on_demand
        }

        root * /var/www/caddy/public

        file_server

        php_fastcgi unix//run/php/php8.0-fpm.sock
}

So far, only the layoff.wtf is working as expected.

I also updated my endpoint to return 200 response only for check.layoff.wtf and jatra.info.

Next to experiment -

I updated my jatra.info A record to point to my layoff.wtf server IP
Tried creating A record for check.layoff.wtf and a CNAME record for check.layoff.wtf pointing to layoff.wtf IP.

However, I continue to get SSL error. I’ve tried waiting for ~10 minutes (maybe I should wait longer?) but yet, browser throws SSL error when I visit https://check.layoff.wtf and jatra.info.

Is it that I’m going to need ACME Challenge for this setup?

francislavoie · March 17, 2021, 1:05pm

What’s in your logs? Run journalctl -u caddy --no-pager | less to see them, as you’re using running Caddy as a systemd service.

KaK · March 17, 2021, 1:37pm

I’m not sure if this is any useful, but I’ve copied the latest entries -

Mar 17 08:23:57 caddy caddy[597]:         net/http/server.go:2969 +0x36c
Mar 17 08:23:57 caddy caddy[597]: {"level":"info","ts":1615969437.6265073,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000224230"}
Mar 17 08:23:58 caddy caddy[597]: {"level":"info","ts":1615969438.1266766,"logger":"admin","msg":"stopped previous server"}
Mar 17 08:33:34 caddy caddy[597]: {"level":"info","ts":1615970014.4372818,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_addr":"127.0.0.1:42844","headers":{"Accept-Encoding":["gzip"],"Content-Length":["964"],"Content-Type":["application/json"],"Origin":["localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
Mar 17 08:33:34 caddy caddy[597]: {"level":"info","ts":1615970014.4377685,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Mar 17 08:33:34 caddy caddy[597]: 2021/03/17 08:33:34 http: panic serving 127.0.0.1:42844: invalid configuration: maxEvents = 0 and window != 0 would not allow any events
Mar 17 08:33:34 caddy caddy[597]: goroutine 2103 [running]:
Mar 17 08:33:34 caddy caddy[597]: net/http.(*conn).serve.func1(0xc0000f00a0)
Mar 17 08:33:34 caddy caddy[597]:         net/http/server.go:1801 +0x147
Mar 17 08:33:34 caddy caddy[597]: panic(0x1525360, 0x19e5080)
Mar 17 08:33:34 caddy caddy[597]:         runtime/panic.go:975 +0x47a
Mar 17 08:33:34 caddy caddy[597]: github.com/caddyserver/certmagic.(*RingBufferRateLimiter).SetWindow(0xc00003bbd0, 0x1bf08eb000)
Mar 17 08:33:34 caddy caddy[597]:         github.com/caddyserver/certmagic@v0.12.1-0.20201215190346-201f83a06067/ratelimiter.go:206 +0xc7
Mar 17 08:33:34 caddy caddy[597]: github.com/caddyserver/caddy/v2/modules/caddytls.(*TLS).Provision(0xc0007544d0, 0x1a26ac0, 0xc0005b8c40, 0xc0007d4000, 0xc0003586e0, 0x0, 0x0, 0x0, 0xc000628670, 0x547511)
Mar 17 08:33:34 caddy caddy[597]:         github.com/caddyserver/caddy/v2@v2.3.0/modules/caddytls/tls.go:166 +0xe28
Mar 17 08:33:34 caddy caddy[597]: github.com/caddyserver/caddy/v2.Context.LoadModuleByID(0x1a26ac0, 0xc0005b8c40, 0xc0007d4000, 0xc0003586e0, 0x0, 0x0, 0x0, 0x1767e18, 0x3, 0xc00021ca20, ...)
Mar 17 08:33:34 caddy caddy[597]:         github.com/caddyserver/caddy/v2@v2.3.0/context.go:344 +0x66d
Mar 17 08:33:34 caddy caddy[597]: github.com/caddyserver/caddy/v2.Context.App(0x1a26ac0, 0xc0005b8c40, 0xc0007d4000, 0xc0003586e0, 0x0, 0x0, 0x0, 0x1767e18, 0x3, 0x0, ...)
Mar 17 08:33:34 caddy caddy[597]:         github.com/caddyserver/caddy/v2@v2.3.0/context.go:415 +0x168
Mar 17 08:33:34 caddy caddy[597]: github.com/caddyserver/caddy/v2/modules/caddyhttp.(*App).Provision(0xc0007f4750, 0x1a26ac0, 0xc0005b8c40, 0xc0007d4000, 0xc0003586e0, 0x0, 0x0, 0x0, 0xc0007d4030, 0x0)
Mar 17 08:33:34 caddy caddy[597]:         github.com/caddyserver/caddy/v2@v2.3.0/modules/caddyhttp/app.go:139 +0x7f
Mar 17 08:33:34 caddy caddy[597]: github.com/caddyserver/caddy/v2.Context.LoadModuleByID(0x1a26ac0, 0xc0005b8c40, 0xc0007d4000, 0xc0003586e0, 0x0, 0x0, 0x0, 0xc000808370, 0x4, 0xc000505180, ...)
Mar 17 08:33:34 caddy caddy[597]:         github.com/caddyserver/caddy/v2@v2.3.0/context.go:344 +0x66d
Mar 17 08:33:34 caddy caddy[597]: github.com/caddyserver/caddy/v2.Context.App(0x1a26ac0, 0xc0005b8c40, 0xc0007d4000, 0xc0003586e0, 0x0, 0x0, 0x0, 0xc000808370, 0x4, 0x28, ...)
Mar 17 08:33:34 caddy caddy[597]:         github.com/caddyserver/caddy/v2@v2.3.0/context.go:415 +0x168
Mar 17 08:33:34 caddy caddy[597]: github.com/caddyserver/caddy/v2.run.func3(0xc0003586e0, 0x1a26ac0, 0xc0005b8c40, 0xc0007d4000, 0xc0003586e0, 0x0, 0x0, 0x0, 0x0, 0x0)
Mar 17 08:33:34 caddy caddy[597]:         github.com/caddyserver/caddy/v2@v2.3.0/caddy.go:388 +0xdc
Mar 17 08:33:34 caddy caddy[597]: github.com/caddyserver/caddy/v2.run(0xc0003586e0, 0x301, 0x0, 0x0)
Mar 17 08:33:34 caddy caddy[597]:         github.com/caddyserver/caddy/v2@v2.3.0/caddy.go:393 +0x27f
Mar 17 08:33:34 caddy caddy[597]: github.com/caddyserver/caddy/v2.unsyncedDecodeAndRun(0xc00054ec00, 0x3c4, 0x400, 0x7, 0xc000800390)
Mar 17 08:33:34 caddy caddy[597]:         github.com/caddyserver/caddy/v2@v2.3.0/caddy.go:249 +0xf7
Mar 17 08:33:34 caddy caddy[597]: github.com/caddyserver/caddy/v2.changeConfig(0x17689af, 0x4, 0x1772cd5, 0x7, 0xc0004ec000, 0x3c4, 0x600, 0x300, 0x0, 0x0)
Mar 17 08:33:34 caddy caddy[597]:         github.com/caddyserver/caddy/v2@v2.3.0/caddy.go:156 +0x4da
Mar 17 08:33:34 caddy caddy[597]: github.com/caddyserver/caddy/v2.Load(...)
Mar 17 08:33:34 caddy caddy[597]:         github.com/caddyserver/caddy/v2@v2.3.0/caddy.go:102
Mar 17 08:33:34 caddy caddy[597]: github.com/caddyserver/caddy/v2/caddyconfig.adminLoad.handleLoad(0x1a1fa40, 0xc000810180, 0xc000630000, 0x0, 0x0)
Mar 17 08:33:34 caddy caddy[597]:         github.com/caddyserver/caddy/v2@v2.3.0/caddyconfig/load.go:136 +0x508
Mar 17 08:33:34 caddy caddy[597]: github.com/caddyserver/caddy/v2.AdminHandlerFunc.ServeHTTP(0xc0006debd0, 0x1a1fa40, 0xc000810180, 0xc000630000, 0xc0003516a8, 0x40da10)
Mar 17 08:33:34 caddy caddy[597]:         github.com/caddyserver/caddy/v2@v2.3.0/admin.go:801 +0x44
Mar 17 08:33:34 caddy caddy[597]: github.com/caddyserver/caddy/v2.AdminConfig.newAdminHandler.func2.1(0x1a1fa40, 0xc000810180, 0xc000630000)
Mar 17 08:33:34 caddy caddy[597]:         github.com/caddyserver/caddy/v2@v2.3.0/admin.go:123 +0x97
Mar 17 08:33:34 caddy caddy[597]: net/http.HandlerFunc.ServeHTTP(0xc0003b3ce0, 0x1a1fa40, 0xc000810180, 0xc000630000)
Mar 17 08:33:34 caddy caddy[597]:         net/http/server.go:2042 +0x44
Mar 17 08:33:34 caddy caddy[597]: github.com/caddyserver/caddy/v2.instrumentHandlerCounter.func1(0x1a21b40, 0xc000338000, 0xc000630000)
Mar 17 08:33:34 caddy caddy[597]:         github.com/caddyserver/caddy/v2@v2.3.0/metrics.go:46 +0xad
Mar 17 08:33:34 caddy caddy[597]: net/http.HandlerFunc.ServeHTTP(0xc00051fa20, 0x1a21b40, 0xc000338000, 0xc000630000)
Mar 17 08:33:34 caddy caddy[597]:         net/http/server.go:2042 +0x44
Mar 17 08:33:34 caddy caddy[597]: net/http.(*ServeMux).ServeHTTP(0xc000542a40, 0x1a21b40, 0xc000338000, 0xc000630000)
Mar 17 08:33:34 caddy caddy[597]:         net/http/server.go:2417 +0x1ad
Mar 17 08:33:34 caddy caddy[597]: github.com/caddyserver/caddy/v2.adminHandler.serveHTTP(0x100, 0xc0006b15f0, 0x3, 0x3, 0xc000542a40, 0x1a21b40, 0xc000338000, 0xc000630000)
Mar 17 08:33:34 caddy caddy[597]:         github.com/caddyserver/caddy/v2@v2.3.0/admin.go:368 +0xe8
Mar 17 08:33:34 caddy caddy[597]: github.com/caddyserver/caddy/v2.adminHandler.ServeHTTP(0x100, 0xc0006b15f0, 0x3, 0x3, 0xc000542a40, 0x1a21b40, 0xc000338000, 0xc000630000)
Mar 17 08:33:34 caddy caddy[597]:         github.com/caddyserver/caddy/v2@v2.3.0/admin.go:326 +0x650
Mar 17 08:33:34 caddy caddy[597]: net/http.serverHandler.ServeHTTP(0xc000338fc0, 0x1a21b40, 0xc000338000, 0xc000630000)
Mar 17 08:33:34 caddy caddy[597]:         net/http/server.go:2843 +0xa3
Mar 17 08:33:34 caddy caddy[597]: net/http.(*conn).serve(0xc0000f00a0, 0x1a26ac0, 0xc0005b8000)
Mar 17 08:33:34 caddy caddy[597]:         net/http/server.go:1925 +0x8ad
Mar 17 08:33:34 caddy caddy[597]: created by net/http.(*Server).Serve
Mar 17 08:33:34 caddy caddy[597]:         net/http/server.go:2969 +0x36c
Mar 17 08:33:34 caddy caddy[597]: {"level":"info","ts":1615970014.438085,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000224540"}
Mar 17 08:33:34 caddy caddy[597]: {"level":"info","ts":1615970014.9382436,"logger":"admin","msg":"stopped previous server"}

francislavoie · March 18, 2021, 6:09am

I see, the panic is because you set interval but no burst for on_demand_tls. I’d never seen that before, so I needed to dig into it. I’ll make sure there’s a fix in the next version to reject configs with only one but not both.

So make sure to add burst, or remove interval (the defaults should be fine for you), then try again.

KaK · March 18, 2021, 6:24am

WOW!

That worked! Which part of my logs gave you the hint that we need to combine burst and interval? I simply added burst 5 below interval and it works.

Curious to know why did this work even without setting up ACME challenge? I thought I was going to need to reinstall caddy with DigitalOcean addon?

francislavoie · March 18, 2021, 8:58am

The panic’s message let me trace it to the certmagic code (the library that underpins Caddy’s certificate management):

github.com

caddyserver/certmagic/blob/81657a23377cad89ed4c08d3741b40b3cda055c5/ratelimiter.go#L36


      
          // in a sliding window of size window. If maxEvents and window are
          // both 0, or if maxEvents is non-zero and window is 0, rate limiting
          // is disabled. This function panics if maxEvents is less than 0 or
          // if maxEvents is 0 and window is non-zero, which is considered to be
          // an invalid configuration, as it would never allow events.
          func NewRateLimiter(maxEvents int, window time.Duration) *RingBufferRateLimiter {
          	if maxEvents < 0 {
          		panic("maxEvents cannot be less than zero")
          	}
          	if maxEvents == 0 && window != 0 {
          		panic("invalid configuration: maxEvents = 0 and window != 0 would not allow any events")
          	}
          	rbrl := &RingBufferRateLimiter{
          		window:  window,
          		ring:    make([]time.Time, maxEvents),
          		started: make(chan struct{}),
          		stopped: make(chan struct{}),
          		ticket:  make(chan struct{}),
          	}
          	go rbrl.loop()
          	<-rbrl.started // make sure loop is ready to receive before we return

Caddy’s config and certmagic use different terminology here though – “interval” in the Caddyfile is “window” in certmagic, and “burst” is “maxEvents”.

You don’t need to configure the DNS challenge to use On-Demand TLS, but you do need the DNS challenge for wildcard certificates. On-Demand will issue individual certificates for each domain/subdomain.

The reason the DNS challenge is needed for wildcard certs is that they are valid for an infinite amount of domains, so the HTTP or ALPN challenges are not sufficient to prove that the server has control/ownership of the whole zone. The DNS challenge proves that you have access to change DNS records, which is enough to show that you have control and ownership of the whole domain.

KaK · March 18, 2021, 9:31am

This is interesting. What I have is a multi-tenant application. Each user will run their version on their own subdomain; which they should be able to create on the fly.

My current application handles this through the caddy/ask endpoint. If the subdomain is allowed; I’ll issue SSL certificate. That’s my current working setup.

What rate limits will apply to this setup; if I do not configure a wildcard subdomain?

Addendum: I am looking at this - Rate Limits - Let's Encrypt

The main limit is Certificates per Registered Domain (50 per week).

I don’t see myself hitting this limit for the next 6 months at least (unless I’m super lucky!).

However, if I wish to configure ACME challenge, will that be a simple, one-line addition to the existing automatic tls configuration? Viz:

{
	on_demand_tls {
		ask https://layoff.wtf/caddy/ask
		interval 2m
		burst 5
	}

}

https:// {

	tls {
		on_demand
                dns digitalocean ## <------FOR WILDCARD SUBDOMAINS ---------------
	}

	root * /var/www/caddy/public

	file_server

	php_fastcgi unix//run/php/php8.0-fpm.sock
}

francislavoie · March 18, 2021, 9:46am

You’re subject to Let’s Encrypt’s rate limits:

But if you hit rate limits it will fall back to ZeroSSL which has less strict limits: