[Caddy V2] PHP-fpm 502 Error

1. My Caddy version (caddy -version):

v2.0.0-beta10 h1:MkLEoR4kj30H90sZ4YBg9O7sd4Zrl77H0m3mW9gyEmE=

2. How I run Caddy:

a. System environment:

Ubuntu 18.04

b. Command:

Following the caddy/dist/init/linux-systemd guide

sudo cp /path/to/caddy /usr/local/bin
sudo chown root:root /usr/local/bin/caddy
sudo chmod 755 /usr/local/bin/caddy
sudo setcap 'cap_net_bind_service=+ep' /usr/local/bin/caddy
sudo groupadd -g 33 www-data
sudo useradd \
  -g www-data --no-user-group \
  --home-dir /var/www --no-create-home \
  --shell /usr/sbin/nologin \
  --system --uid 33 www-data
sudo mkdir /etc/caddy
sudo chown -R root:root /etc/caddy
sudo mkdir /etc/ssl/caddy
sudo chown -R root:www-data /etc/ssl/caddy
sudo chmod 0770 /etc/ssl/caddy
sudo touch /etc/caddy/Caddyfile
sudo chown root:root /etc/caddy/Caddyfile
sudo chmod 644 /etc/caddy/Caddyfile
sudo mkdir /var/www
sudo chown www-data:www-data /var/www

c. Service file:

[Unit]
Description=Caddy v2
Documentation=https://github.com/caddyserver/caddy/wiki/v2:-Documentation
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service
[Service]
Restart=on-abnormal

User=www-data
Group=www-data

Environment=CADDYPATH=/etc/ssl/caddy
ExecStart=/usr/local/bin/caddy run --config /etc/caddy/Caddyfile --adapter caddyfile
ExecReload=/usr/local/bin/caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile
KillMode=mixed
KillSignal=SIGQUIT
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=4096

[Install]
WantedBy=multi-user.target

d. My Caddyfile:

note.craigary.net {
	root * /var/www/note
	encode gzip
	php_fastcgi unix//run/php/php7.3-fpm.sock
	file_server
	tls i@craigary.net
}

3. The problem I’m having:

PHP Server doesn’t work, return 502 error.

4. Error messages from systemctl status caddy:

no such file or directory error:

Nov 30 15:32:22 craigary.net caddy[831]: 2019/11/30 15:32:22.571        ERROR        http.log.error        dialing backend: dial unix /run/php/php7.3-fpm.sock:0: connect: no such file or directory

5. What I already tried:

  • Make sure /run/php/php7.3-fpm.sock exists.
  • Edit caddyfile php_fastcgi php-fpm:9000, no error show up in service status, but still 502 error.
  • Check PHP-FPM www.conf: user / group is www-data, same user as the one running caddy:
[www]
user = www-data
group = www-data
listen = /run/php/php7.3-fpm.sock
listen.owner = www-data
listen.group = www-data
pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3

6. Links to relevant resources:

In my Caddyfile I have:

fastcgi / /run/php/php7.3-fpm.sock

and not

fastcgi unix//run/php/php7.3-fpm.sock

I tried, but this doesn’t work, seems you are using caddy v1, but v2 Caddyfile is not backward-compatible with it.

I followed the documents in Github, @matt @Whitestrake any ideas :thinking:?

Conventionally, this should be in /var/run, not /run. Are you sure this is the right location? Caddy seems to think it doesn’t exist.

I found it in /run/php/ the terminal, but it also shows up in /var/run/php,

I changed the Caddyfile, no “file not found” error show up, but still got a 502 error.

note.craigary.net {
        root * /var/www/note
        encode gzip
        php_fastcgi unix//var/run/php/php7.3-fpm.sock
        file_server
        tls i@craigary.net

}

What was the error / output from Caddy this time?

Still dialing backend: dial unix /var/run/php/php7.3-fpm.sock:0: connect: no such file or directory:

Dec 01 06:05:11 craigary.net caddy[7748]: 2019/12/01 06:05:11 [INFO][cache:0xc0001f1ef0] Started certificate maintenance routine
Dec 01 06:05:26 craigary.net caddy[7748]: 2019/12/01 06:05:26.901        ERROR        http.log.error        dialing backend: dial unix /var/run/php/php7.3-fpm.sock:0: connect: no such file or directory        {"request": {"method": "GET", "uri": "/", "proto": "HTTP/2.0", "remote_addr": "47.254.94.88:54983", "host": "note.craigary.net", "headers": {"Upgrade-Insecure-Requests": ["1"], "User-Agent": ["Mozilla/5.0 (Ma
Dec 01 06:05:30 craigary.net caddy[7748]: 2019/12/01 06:05:30.650        ERROR        http.log.error        dialing backend: dial unix /var/run/php/php7.3-fpm.sock:0: connect: no such file or directory        {"request": {"method": "GET", "uri": "/", "proto": "HTTP/2.0", "remote_addr": "47.254.94.88:54983", "host": "note.craigary.net", "headers": {"Dnt": ["1"], "Sec-Fetch-User": ["?1"], "Accept": ["text/html,appli
Dec 01 06:05:31 craigary.net caddy[7748]: 2019/12/01 06:05:31.371        ERROR        http.log.error        dialing backend: dial unix /var/run/php/php7.3-fpm.sock:0: connect: no such file or directory        {"request": {"method": "GET", "uri": "/", "proto": "HTTP/2.0", "remote_addr": "47.254.94.88:54983", "host": "note.craigary.net", "headers": {"Upgrade-Insecure-Requests": ["1"], "User-Agent": ["Mozilla/5.0 (Ma
Dec 01 06:05:31 craigary.net caddy[7748]: 2019/12/01 06:05:31.986        ERROR        http.log.error        dialing backend: dial unix /var/run/php/php7.3-fpm.sock:0: connect: no such file or directory        {"request": {"method": "GET", "uri": "/", "proto": "HTTP/2.0", "remote_addr": "47.254.94.88:54983", "host": "note.craigary.net", "headers": {"Upgrade-Insecure-Requests": ["1"], "User-Agent": ["Mozilla/5.0 (Ma

Can you run Caddy from the command line directly instead of as a service, and see if you get the same problem? (Don’t forget to set the CADDYPATH)

sorry for late reply,

execute command: /usr/local/bin/caddy run --config /etc/caddy/Caddyfile --adapter caddyfile

but the error remains the same…

Error(a little bit messy):

2019/12/01 09:29:39 [INFO] [note.craigary.net] The server validated our request
2019/12/01 09:29:39 [INFO] [note.craigary.net] acme: Validations succeeded; requesting certificates
2019/12/01 09:29:40 [INFO] [note.craigary.net] Server responded with a certificate.
2019/12/01 09:29:44.698	ERROR	http.log.error	dialing backend: dial unix /var/run/php/php7.3-fpm.sock:0: connect: no such file or directory	{"request": {"method": "GET", "uri": "/", "proto": "HTTP/2.0", "remote_addr": "47.254.94.88:21837", "host": "note.craigary.net", "headers": {"Sec-Fetch-Site": ["none"], "Sec-Fetch-Mode": ["navigate"], "Accept-Language": ["en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7,zh-TW;q=0.6"], "Dnt": ["1"], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.56 Safari/537.36"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"], "Cookie": ["_ga=GA1.2.1519661375.1574141031; _gid=GA1.2.1225328046.1575127943"], "Upgrade-Insecure-Requests": ["1"], "Sec-Fetch-User": ["?1"], "Accept-Encoding": ["gzip, deflate, br"]}, "tls": {"resumed": false, "version": 772, "resumed": 4865, "proto": "h2", "proto_mutual": true, "server_name": "note.craigary.net"}}, "status": 502, "err_id": "6w2n9h7mk", "err_trace": "reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:330)"}
2019/12/01 09:29:47.867	ERROR	http.log.error	dialing backend: dial unix /var/run/php/php7.3-fpm.sock:0: connect: no such file or directory	{"request": {"method": "GET", "uri": "/", "proto": "HTTP/2.0", "remote_addr": "47.254.94.88:21837", "host": "note.craigary.net", "headers": {"Sec-Fetch-User": ["?1"], "Sec-Fetch-Mode": ["navigate"], "Accept-Language": ["en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7,zh-TW;q=0.6"], "Cookie": ["_ga=GA1.2.1519661375.1574141031; _gid=GA1.2.1225328046.1575127943"], "Accept-Encoding": ["gzip, deflate, br"], "Cache-Control": ["max-age=0"], "Dnt": ["1"], "Upgrade-Insecure-Requests": ["1"], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.56 Safari/537.36"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"], "Sec-Fetch-Site": ["none"]}, "tls": {"resumed": false, "version": 772, "resumed": 4865, "proto": "h2", "proto_mutual": true, "server_name": "note.craigary.net"}}, "status": 502, "err_id": "7wdzcsi30", "err_trace": "reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:330)"}
2019/12/01 09:29:49.593	ERROR	http.log.error	dialing backend: dial unix /var/run/php/php7.3-fpm.sock:0: connect: no such file or directory	{"request": {"method": "GET", "uri": "/", "proto": "HTTP/2.0", "remote_addr": "47.254.94.88:21837", "host": "note.craigary.net", "headers": {"Dnt": ["1"], "Upgrade-Insecure-Requests": ["1"], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.56 Safari/537.36"], "Sec-Fetch-Mode": ["navigate"], "Accept-Encoding": ["gzip, deflate, br"], "Accept-Language": ["en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7,zh-TW;q=0.6"], "Cache-Control": ["max-age=0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"], "Sec-Fetch-Site": ["cross-site"], "Cookie": ["_ga=GA1.2.1519661375.1574141031; _gid=GA1.2.1225328046.1575127943"], "Sec-Fetch-User": ["?1"]}, "tls": {"resumed": false, "version": 772, "resumed": 4865, "proto": "h2", "proto_mutual": true, "server_name": "note.craigary.net"}}, "status": 502, "err_id": "ig0vyssxd", "err_trace": "reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:330)"}

Well, the file is definitely there, and you can see and access it as your user, and you can run Caddy as your user and it thinks the file doesn’t exist. I can’t think of much else to troubleshoot on that front.

I think it’s time to open an issue on the Github repo. If you can give us some information on your setup and some instructions on how to replicate this error as simply as possible, hopefully we can start narrowing it down over there.

Issues · caddyserver/caddy · GitHub

After testing for a while, I realized this is an issue related to my VPS Operating System image, I tried using caddy v1, with the instructions I familiar with, the problem still exists.

Now I am happy with Caddy + Debian :joy::joy::joy:

1 Like

@Whitestrake I have a very similar problem that just popped up after rebooting my Ubuntu server. “php_fastcgi unix//var/run/php/php7.3-fpm.sock” previously worked fine, but now I get a 502 error like:

Dec 06 09:26:55 attenborough caddy[653]: 2019/12/06 14:26:55.029        ERROR        http.log.error        dialing backend: dial unix /var/run/php/php7.3-fpm.sock:0: connect: no such file or directory        {"request": {"method": "GET", "uri": "/", "proto": "HTTP/2.0", "remote_addr": "[2601:42:0:6200:e00c:411a:fdf6:3923]:65006", "host": "ri.sunrisemovement.dev", "headers": {"Accept-Language": ["en-US,en;q=0.5"], "Cookie": ["__cfduid=d6ca04da650bbdc70d21c10be860b5ffa1572436203"], "Te": ["trailers"], "Pragma": ["no-cache"], "Cache-Control": ["no-cache"], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:71.0) Gecko/20100101 Firefox/71.0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"], "Accept-Encoding": ["gzip, deflate, br"], "Dnt": ["1"], "Upgrade-Insecure-Requests": ["1"]}, "tls": {"resumed": true, "version": 772, "resumed": 4865, "proto": "h2", "proto_mutual": true, "server_name": "ri.sunrisemovement.dev"}}, "status": 502, "err_id": "vi2098aeb", "err_trace": "reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:330)"}

I suspect that it’s also a problem with my OS, but I can’t figure out what it might be. @craigary Did you figure out what it was, or did you just change OS without tracking down the problem?

I’d be interested to know if you have this issue with v2 beta 9. With beta 9 Caddy could read my PHP socket, but with a subsequent commit it couldn’t.

I haven’t yet tried again.

That’s weird, AFAIK we didn’t change anything related to unix sockets or proxy dialing in that time (unless you’re using NTLM, which I doubt is the case).

Actually can you both try before and after this commit? I’m curious whether that affected anything.

That commit indeed seems to be the breaking one:

http.log.error dialing backend: dial unix /run/php/php7.3-fpm.sock:0: connect: no such file or directory

The commit before allows for index.php to be served.

Ruh roh, good to know! @Mohammed90 – any ideas off the top of your head what might be breaking unix sockets in that commit? (I just now noticed php7.3-fpm.sock:0: – not sure what the “:0:” is all about…)

To create the host of upstream, we call:

func (l ParsedAddress) JoinHostPort(offset uint) string {
    return net.JoinHostPort(l.Host, strconv.Itoa(int(l.StartPort+offset)))
}

Which for unix networks results in port of :0 appended. We should strip the port portion if the network is unix.

1 Like

Aha, thanks. I will submit a commit to fix it in just a bit.

@BrightEyesD @craigary Would you mind testing against this commit? Don't append port to unix sockets · caddyserver/caddy@33a318d · GitHub

2 Likes

https://cdn.clipart.email/2fc149baa9bfaa25d402cdec1a9f3ac8_our-voice-scooby-doo-in-wheres-my-mummy-2002-dual-audio-brrip-_1024-576.jpeg

:+1: Thanks @matt and @Mohammed90.

1 Like