Caddy reverse proxy error 502 Bad Gateway on Docker

Help
1

Ubuntu: 22
Caddy: v2.8.4

Hi, I’m having a problem with Caddy using reverse proxy to a service inside a docker network. I have a caddy service that listens on 80, a mysql service, and a php service that listens on 9000. This php service is a laravel application that I will use as an Api. The frontend for now is a simple index.php as a test.

Whenever I access the frontend using my domain it works, but whenever I try to access the subdomain reserved for the laravel api, it throws error 502 bad gateway. I really don’t know what to do anymore, tried looking on the internet but nothing I do seems to work.

This is the Caddyfile (ommited the domain and subdomain):

<domain.app> {
    root * /usr/share/caddy
    file_server
}

<api.domain.app> {
    reverse_proxy app-laravel:9000
}

This is the compose file:

name: app

networks:
  appnetwork:
    name: appnetwork

volumes:
  caddy-config:
  caddy-data:
  db-laravel:

services:

  server:
    container_name: server
    image: caddy:2-alpine
    restart: unless-stopped
    networks:
      - appnetwork
    ports:
      - 80:80
      - 443:443
    volumes:
      - caddy-config:/config
      - caddy-data:/data
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./index.html:/usr/share/caddy/index.html

  db-laravel:
    container_name: db-laravel
    image: mysql:8
    restart: unless-stopped
    networks:
      - appnetwork
    ports:
      - 3306:3306
    env_file:
      - ./mysql.env
    volumes:
      - db-laravel:/var/lib/mysql

  app-laravel:
    container_name: app-laravel
    image: <private-app-laravel-image>
    depends_on:
      - db-laravel
      - server
    networks:
      - appnetwork
    ports:
      - 9000:9000
    env_file:
      - ./laravel.env
    volumes:
      - /var/www/html/storage

When I run docker compose up, every container works fine. I can migrate and tinker inside the laravel container and apply the changes to the database.

I can also ping from inside caddy container to laravel container and vice-versa without any problem.

Then inside the laravel container there are no error logs.

But whenever I try to connect to the subdomain, inside the caddy container it throws this error in the logs:

{
    "level": "error",
    "ts": 1724710486.2305286,
    "logger": "http.log.error",
    "msg": "read tcp 172.18.0.2:43388->172.18.0.4:9000: read: connection reset by peer",
    "request": {
        "remote_ip": "189.6.253.153",
        "remote_port": "2469",
        "client_ip": "189.6.253.153",
        "proto": "HTTP/2.0",
        "method": "GET",
        "host": "<api.domain>",
        "uri": "/api/users",
        "headers": {
            "Accept-Language": [
                "pt-BR,pt;q=0.5"
            ],
            "Sec-Fetch-Dest": [
                "document"
            ],
            "Cache-Control": [
                "no-cache"
            ],
            "Sec-Ch-Ua-Platform": [
                "\"Windows\""
            ],
            "Accept": [
                "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"
            ],
            "Sec-Gpc": [
                "1"
            ],
            "Priority": [
                "u=0, i"
            ],
            "Sec-Ch-Ua": [
                "\"Chromium\";v=\"128\", \"Not;A=Brand\";v=\"24\", \"Brave\";v=\"128\""
            ],
            "Sec-Fetch-Site": [
                "none"
            ],
            "Sec-Fetch-Mode": [
                "navigate"
            ],
            "Sec-Fetch-User": [
                "?1"
            ],
            "Pragma": [
                "no-cache"
            ],
            "Sec-Ch-Ua-Mobile": [
                "?0"
            ],
            "User-Agent": [
                "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
            ],
            "Accept-Encoding": [
                "gzip, deflate, br, zstd"
            ],
            "Upgrade-Insecure-Requests": [
                "1"
            ]
        },
        "tls": {
            "resumed": true,
            "version": 772,
            "cipher_suite": 4865,
            "proto": "h2",
            "server_name": "<api.domain>"
        }
    },
    "duration": 0.003893169,
    "status": 502,
    "err_id": "eutw8jhv5",
    "err_trace": "reverseproxy.statusError (reverseproxy.go:1269)"
}

What can I do?

2

Howdy @Jean_da_Silva, welcome to the Caddy community.

This is a pretty bad response from upstream. It’s essentially slamming the door in Caddy’s face when Caddy tries to connect.

Is it just slamming the door in Caddy’s face specifically, or is it slamming the door in anyone’s face?

What happens if you run docker compose exec server wget --server-response --spider app-laravel:9000 ? (Unrelated, but why did you call your Caddy container just “server”? :thinking:) What happens when you visit port 9000 of the host directly on your browser?

What logs appear in your Laravel container?

1 Like
3

Hi, when I execute the suggested command, it returns this message:

docker compose exec server wget --server-response --spider app-laravel:9000
Connecting to app-laravel:9000 (172.18.0.4:9000)
wget: error getting response: Connection reset by peer

When I try to check if there is a laravel.log file it shows that it doesn’t exist:

docker compose exec app-laravel ls -la storage/logs/laravel.log
ls: storage/logs/laravel.log: No such file or directory

Answering your question about the name of the caddy container, I wrote server as a generic name in case I need change it for other thing. It’s possible this application will grow, but until now I am trying to make it work with less complication possible.

When I try to access the subdomain, if I do it only with the subdomain it throws the error 502, but if I try to access with the port 9000 it only shows the error ERR_CONNECTION_RESET

4

Yeah, gotcha. I tend to call the Caddy container caddy or proxy, so as not to confuse it with other server components in the stack, but that’s all personal preference, doesn’t matter as long as you know what’s referring to what and everything is using the right DNS names, it’s all arbitrary.

As for the rest - yeah, so, it’s not just Caddy that your app isn’t talking to. It’s everyone. It doesn’t wanna talk HTTP at all, regardless of client.

Usually when I see port 9000, I assume a PHP FastCGI interface rather than a web server, although not always. Is it possible you’re trying to use some kind of FPM container as a backend? Because that would require a very different setup.

1 Like
5

I was reading about it and trying some new configs. I am using this image as the base for my private image serversideup/php:8.3-fpm-alpine.

So it is using php-fpm. I was taking a look and it seems the config should be something like it:

site {
	root * /srv/public
	encode gzip
	php_fastcgi php:9000
	file_server
}

So I’ve changed my Caddyfile to this:

<domain.app> {
    root * /usr/share/caddy
    file_server
}

<api.domain.app> {
    root * /var/www/html/public
    encode gzip
    php_fastcgi app-laravel:9000
    file_server
}

The image exposes port 9000 and the workdir is /var/www/html. So it seems fine. When I run docker compose up and try to access the subdomain it no longer show error 502 neither ERR_CONNECTION_RESET.

Now it shows HTTP ERROR 404 like if it wasn’t finding the index.php. It’s strange because if I execute docker exec -it app-laravel sh and ls -la inside of it I can find the file /var/www/html/public/index.php, and if I try to check the logs on the container it doesn’t show any errors.

6

That configuration looks correct to me at a glance. (Although as a minor note, you could, if you wanted, simplify root * /usr/share/caddy to root /usr/share/caddy without the glob, and same with the API site, if you wanted.)

I note that in your original Compose file, Caddy does not have /var/www/html/storage. Have you updated this so that Caddy can see the files the FPM container can?

If Caddy can’t see the file, it will return a 404.

Remember that in a traditional setup, Caddy and FPM would be running in the same environment, not split across containers. The assumption is that the PHP process and the web server process can both see the same files, so you’ll need to replicate that in Docker.

As another incidental note, since you have no need to access port 9000 directly on the host - Caddy is connecting to it from within your configured appnetwork network - you could probably ditch ports: - 9000:9000 from your Compose file for app-laravel service. You can probably do the same for 3306:3306 on the db-laravel service unless you have clients outside the Docker network that need it.

As another incidental note, I saw that domain.app doesn’t exist, but someone might register it one day. It’s a good idea - if you must redact your domain name - to use example.com instead, because IANA have reserved it for demonstration and documentation purposes exactly like this.

1 Like
7

Thank you for helping and these suggestions. I will change them. I didn’t know about the /var/www/html/storage for Caddy. Maybe there’s something I don’t know, I will explain about my intentions on the volumes in the composer file, so if you can help me understanding it better, I’ll be glad.

What I know is that volumes are for persisting the data even after the container is destroyed. When I build the php-fpm image containing Laravel, I already copy everything from the app and run compose install, so the image have everything needed for Laravel to run. So I’m adding the volume on /var/www/html/storage for the app to not lose any content inside storage like assets and log files, to persists whenever I need to build the image again.

This is the dockerfile for php-fpm:

FROM serversideup/php:8.3-fpm-alpine

ENV PHP_OPCACHE_ENABLE=1

WORKDIR /var/www/html

USER root

COPY src .

COPY --chown=www-data:www-data src /var/www/html

COPY --chown=www-data:www-data . /var/www/html

USER www-data

RUN composer clear-cache

RUN composer install --no-interaction --optimize-autoloader --no-dev

On my local development I’m using nginx instead of caddy (I will change to caddy to make everything the same on any environment), and there’s a volume on the nginx service for ./src:/var/www/html, but in this case it’s using the content inside my repository that has the Laravel code (inside src). But if I need to do the same for caddy, how can I do it from the app-laravel service? I don’t have the cloned repository inside the server, only the image.

8

So, if the web root is /var/www/html, Caddy will need to see that whole directory. It needs to see the index files and all the file content - it has the same requirements as nginx in this regard.

Docker doesn’t have a very effective tool to help you here, unfortunately - not without converting the site files to a volume in order to share them between containers, which will require you to persist them.

Given that you’re trying to ship site files inside the container, and presumably want to use container updates to handle site file updates, persisting them sounds like it would be the exact opposite of your intention. That means you’ll have to get creative.

You might look into slipping the Caddy binary itself into your FPM container. It’s not a solution I see often, because it will require you then to convert the container into one with some kind of init system or script - as Docker containers only have a single entrypoint, that entrypoint will have to launch and manage both Caddy and PHP-FPM. But it would give Caddy identical file access to FPM.

Another solution I’ve seen before is persisting /var/www/html, but you don’t actually ship your source in that directory. Instead, when building your container, you ship the source in /src or similar, and an entrypoint script copies from /src to /var/www/html at runtime as a first step before handing off to the “normal” entrypoint. That allows you to add the web root as a volume to the Caddy and app containers, and then when the app container starts up, it makes sure the files in place are up to date with the container source.

Either route is going to take a bit of engineering work on your end, though. Maybe someone else has some other suggestions or maybe a silver bullet for you, but that’s all I can think of right now, and we are unfortunately well beyond the scope of Caddy itself and into Docker and stack engineering territory.

1 Like
9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.