504 Gateway Timeout PHP

1. The problem I’m having:

I’m getting a 504 Gateway timeout after 60 seconds even though my timeouts are set to 14400 seconds.
PHP max_execution time and input time are both 14400 seconds fpm doesn’t have terminate max time turned on.

2. Error messages and/or full log output:

504 Gateway Timeout

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

Installed via Apt repo

a. System environment:

Ubuntu 22.04, systemd

d. My complete Caddy config:

# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace ":80" below with your
# domain name.
{
        servers {
                timeouts {
                        read_body 24h
                        read_header 24h
                        write 24h
                        idle 24h
                }
        }
}


:80 {
        # Set this path to your site's directory.
        root * /var/www/vhosts/web-app/current
        encode zstd gzip
        log

        # Enable the static file server.
        file_server

        php_fastcgi 127.0.0.1:9000 { # 9000 is PHP8.2 and 9001 is PHP5.6
                resolve_root_symlink
                try_files {path} {path}.php {path}/index.php /index.php
                dial_timeout 14400s
                read_timeout 14400s
                write_timeout 14400s
        }
        # Another common task is to set up a reverse proxy:
        # reverse_proxy localhost:8080

        # Or serve a PHP site through php-fpm:
        # php_fastcgi localhost:9000
}

# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile

5. Links to relevant resources:

Show your Caddy logs. Show an example request with curl -v that produces that behaviour. Just a status code doesn’t give us enough to go on here. We need a lot more detail.

I’ll try and work on a curl command but the page in question uses post, get and cookies so it’s difficult. I managed to get a caddy log line from this it says 200 but the browser says 504:

2024/05/08 16:42:59.151 INFO    http.log.access handled request {"request": {"remote_ip": "10.0.3.231", "remote_port": "24454", "client_ip": "10.0.3.231", "proto": "HTTP/1.1", "method": "GET", "host": "app2.leadsrx.com", "uri": "/performance-report?reportType=trigger&leadID=*&dateRange=custom:2024-04-22:2024-05-08&aModel=first&leadType=all&agg=daily&triggerID=548792", "headers": {"Accept-Language": ["en-US,en;q=0.8"], "Sec-Fetch-Dest": ["document"], "Priority": ["u=0, i"], "Upgrade-Insecure-Requests": ["1"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"], "Sec-Ch-Ua-Mobile": ["?0"], "Sec-Fetch-Site": ["same-origin"], "Sec-Fetch-User": ["?1"], "Cookie": [], "X-Amzn-Trace-Id": ["Root=1-663bab56-021f13a540de89a7780572c4"], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "Sec-Ch-Ua": ["\"Chromium\";v=\"124\", \"Brave\";v=\"124\", \"Not-A.Brand\";v=\"99\""], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"], "Sec-Gpc": ["1"], "Sec-Fetch-Mode": ["navigate"], "Referer": ["https://app2.leadsrx.com/performance-report"], "X-Forwarded-Proto": ["https"], "X-Forwarded-Port": ["443"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Sec-Ch-Ua-Platform": ["\"Windows\""], "X-Forwarded-For": ["68.50.104.25"]}}, "bytes_read": 0, "user_id": "", "duration": 60.391736727, "size": 28868, "status": 200, "resp_headers": {"Content-Type": ["text/html; charset=UTF-8"], "Expires": ["Thu, 19 Nov 1981 08:52:00 GMT"], "Cache-Control": ["no-store, no-cache, must-revalidate"], "Pragma": ["no-cache"], "Content-Encoding": ["zstd"], "Vary": ["Accept-Encoding"], "Server": ["Caddy"], "Set-Cookie": []}}```

So through more testing i’ve determined this is caused by ELB Keepalive timeout. Is there a caddy config for keepalive timeout to prevent the connection from timing out?

I don’t think it’s Caddy that’s timing out though. We’d see an error saying as much.

1 Like

Yeah, i’ve determined that i need a keepalive to prevent the elb from trigging the timeout for long requests. I know how to do this by putting keepalive_timeout 45s; in a nginx server block. How do i do this on caddy. I’m not seeing any references to it.

Looks like our default is 15 seconds:

1 Like

I just realized we forgot to document that for the Caddyfile, but it works there too. It’s not something anyone ever needs to configure because it’s on by default and the default should be good enough for everyone at 15s.

{
	servers {
		keepalive_interval 15s
	}
}
1 Like

I just saw this, the default wasn’t working. Even setting it it’s still not working however using nginx’s keepalive it is working. Not sure what about caddy in this environment is causing awselb to ignore the keepalive.