Caddy fails to redirect request based on response status code (not in reverse_proxy directive)

1. The problem I’m having:

Hello,

I have a PHP Symfony app (runs using frankenphp) that initially handles all the requests and if it cannot handle some specific ones, it returns a 307 response that needs to be re-routed to another nuxt app.

I used the handle_errors as suggested by the doc (see bellow).
I Used 3 instances of handle_errors in my Caddyfile just to be sure:

  • one in top level
  • the 2nd in the site config
  • the last within handle with named matcher

The nuxt service does NOT receive any 307 request from the Symfony app throught caddy reverse proxy.

I highly suspect that this is an issue related to error status matcher/expression…

2. Error messages and/or full log output:

front-1  | 2024/03/28 21:25:56.224      DEBUG   http.handlers.rewrite   rewrote request {"request": {"remote_ip": "172.20.0.1", "remote_port": "33094", "client_ip": "172.20.0.1", "proto": "HTTP/2.0", "method": "GET", "host": "metro.craft.fr.local:8080", "uri": "/fr_FR/", "headers": {"Cache-Control": ["no-cache"], "Upgrade-Insecure-Requests": ["1"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"], "Sec-Fetch-Site": ["none"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Pragma": ["no-cache"], "Sec-Fetch-Dest": ["document"], "Sec-Fetch-User": ["?1"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"], "Sec-Ch-Ua": ["\"Google Chrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\""], "Sec-Ch-Ua-Platform": ["\"Windows\""], "Sec-Fetch-Mode": ["navigate"], "Accept-Language": ["en-US,en;q=0.9,fr-FR;q=0.8,fr;q=0.7"], "Cookie": [], "Sec-Ch-Ua-Mobile": ["?0"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "metro.craft.fr.local"}}, "method": "GET", "uri": "/index.php"}
front-1  | 2024/03/28 21:25:56.244      DEBUG   request handling finished       {"worker": "/app/public/index.php", "url": "/index.php"}
front-1  | 2024/03/28 21:25:56.244      INFO    http.log.access handled request {"request": {"remote_ip": "172.20.0.1", "remote_port": "33094", "client_ip": "172.20.0.1", "proto": "HTTP/2.0", "method": "GET", "host": "metro.craft.fr.local:8080", "uri": "/fr_FR/", "headers": {"Sec-Fetch-User": ["?1"], "Sec-Fetch-Dest": ["document"], "Sec-Ch-Ua": ["\"Google Chrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\""], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"], "Cookie": [], "Sec-Ch-Ua-Mobile": ["?0"], "Sec-Ch-Ua-Platform": ["\"Windows\""], "Sec-Fetch-Mode": ["navigate"], "Accept-Language": ["en-US,en;q=0.9,fr-FR;q=0.8,fr;q=0.7"], "Sec-Fetch-Site": ["none"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Pragma": ["no-cache"], "Cache-Control": ["no-cache"], "Upgrade-Insecure-Requests": ["1"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "metro.craft.fr.local"}}, "bytes_read": 0, "user_id": "", "duration": 0.020900041, "size": 29, "status": 307, "resp_headers": {"Server": ["Caddy"], "X-Powered-By": ["PHP/8.3.4"], "X-User-Id": ["Anonymous"], "X-Debug-Token": ["297b07"], "Set-Cookie": [], "Expires": ["Thu, 28 Mar 2024 21:25:56 GMT"], "Alt-Svc": ["h3=\":8080\"; ma=2592000"], "X-Route-Name": ["homepage"], "X-Origin": ["-"], "X-Robots-Tag": ["noindex"], "Cache-Control": ["max-age=0, must-revalidate, no-store, private"], "Date": ["Thu, 28 Mar 2024 21:25:56 GMT"], "X-Order-Number": ["-"], "Content-Type": ["text/html; charset=UTF-8"], "X-Debug-Token-Link": ["https://metro.craft.fr.local:8080/_profiler/297b07"]}}

3. Caddy version:

Caddy v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

a. System environment:

FROM dunglas/frankenphp:sha-e7e0dbf-php8.3.4-bookworm

RUN \
    apt-get update && \
    apt-get install -y gnupg2 ca-certificates libgnutls30 && \
    curl -sL https://nodejs.org/download/release/v14.21.3/node-v14.21.3-linux-x64.tar.gz | tar xz -C /usr/local --strip-components=1 && \
    apt-get update && \
    apt-get install -y git

RUN install-php-extensions \
    zip \
    exif \
    intl


WORKDIR /app/

ARG USER="craft-front"

RUN curl -sS https://getcomposer.org/installer | php -- --2.2 && \
    mv composer.phar /usr/local/bin/composer

RUN useradd -ms /bin/bash ${USER} && \
    usermod -u 1000 ${USER};

RUN \
    chown -R ${USER}:${USER} /var/log/ /var/log/* && \
    chmod -R 777 /var/log/ /var/log/ && \
	# Use "adduser -D ${USER}" for alpine based distros
	#useradd -D ${USER}; \
	# Add additional capability to bind to port 80 and 443
	setcap CAP_NET_BIND_SERVICE=+eip /usr/local/bin/frankenphp && \
	# Give write access to /data/caddy and /config/caddy \
    chown -R ${USER}:${USER} /data/caddy && chown -R ${USER}:${USER} /config/caddy &&  \
    chmod +x /usr/local/bin/frankenphp && \
    chown -R ${USER}:${USER} /app/ &&  \
    chmod 755 /app/

USER ${USER}

RUN touch /var/log/php-franken.error.log &&  \
    touch /var/log/php-franken.access.log

ENV FRANKENPHP_CONFIG="worker /app/public/index.php"
ENV APP_RUNTIME='Runtime\FrankenPhpSymfony\Runtime'
ENV PHP_INI_DIR='/usr/local/etc/php'

USER root

RUN usermod -aG sudo ${USER}  && \
    chown -R ${USER}:${USER} /app && \
    chown -R root:root /etc/caddy && \
    chmod -R 777 /app /config /data/caddy/ /config/caddy/ && \
    chmod 775 /etc/caddy


EXPOSE 8080
EXPOSE 443

COPY . /app/
COPY ./devops/docker/frankenphp/php.ini /usr/local/etc/php/

RUN composer install

b. Command:

docker compose up front --force-recreate
# that run the startup command bellow
frankenphp run --watch --config ./devops/docker/caddy/config/Caddyfile --adapter caddyfile

c. Service/unit/compose file:

  front:
    build:
      context: craft-front/.
    cap_add:
      - NET_ADMIN
    command:
      frankenphp run --watch --config ./devops/docker/caddy/config/Caddyfile --adapter caddyfile
    volumes:
      - ./craft-front:/app
      - ./craft-front/devops/docker/caddy/config/Caddyfile:/etc/caddy/Caddyfile
    ports:
      - "9001:9001"
      - "3007:3007"
      - "12444:12444"
      - "8080:8080" # HTTP
      - "443:443" # HTTPS
      - "443:443/udp" # HTTP/3
    tty: true
    extra_hosts:
      host.docker.internal: host-gateway
    networks:
      - network

  nuxt:
    build: craft-front/devops/docker/ssr/.
    container_name: nuxt
    volumes:
      - ./craft-front:/var/www/craft-front:cached
    restart: always
    networks:
      - network
    environment:
      - STACK_ENV=dev
    command:
      bash -c "pm2 start --env local --no-daemon"

d. My complete Caddy config:

{
    debug
    servers {
        protocols h1 h2 h3
    }

    admin off
    local_certs
    frankenphp
    order php_server before file_server
}

handle_errors {
        @307 {
            expression `{http.error.status_code} == '307'`
        }
        handle @307 {
            reverse_proxy * http://nuxt:3000
        }
}


https://*.*.*.*.local:8080,
https://*.*.*.local:8080,
https://*.*.local:8080,
https://*.local:8080 {

    log # enable basic log

    encode gzip zstd # compression

    handle_errors {
        @307 expression {err.status_code} == 307
        handle @307 {
            reverse_proxy http://nuxt:3000
        }
    }

    handle {
        @307err expression {err.status_code} == 307
        handle @307err {
            reverse_proxy http://nuxt:3000
        }
        root * /app/public/
        php_server
    }
    respond 404
}

I read through the documentation, I know that handle_errors is bit special.
Also experimented with the redir to no avail.
I looked at (definitely not exhaustive list) :

Thanks for the help.

Errors are not triggered when an upstream writes a response. You need to use reverse_proxy’s handle_response to intercept the responses either by status code or header, then use the error directive to trigger an error, which you can later handle with handle_errors.

Also, handle_errors is a directive, it must go within a site block. It can’t be top-level in your Caddyfile.

So if understand correctly, I need to set up a dedicated 1st reverse_proxy for the primary front app, then intercept 307 responses and re-route them through a 2nd reverse reverse_proxy?

I changed the Caddyfile to this, but now I get 400 error on all responses

{
    debug
    local_certs
    frankenphp
    order php_server before file_server
}

https://*.*.*.*.local:8080,
https://*.*.*.local:8080,
https://*.*.local:8080,
https://*.local:8080 {

    log # enable basic log
    encode gzip zstd # compression

    reverse_proxy * http://front:8080 {
        @307err status 307
        handle_response @307err {
            error 307
        }
    }

    handle_errors {
            @307 expression {err.status_code} == 307
            handle @307 {
                reverse_proxy http://nuxt:3000
            }
        }

    root * /app/public/
    php_server
}

And what’s in your Caddy logs at this point? Show the debug logs, need to see what the responses from upstream look like.

With that config, I think you’re proxying all requests to front, and none make it to php_server. You probably need to use somekind of matcher to tell Caddy which requests to route where.

So, during investigation, I changed the named matcher from status code to header attribute. The reverse proxy redirection does not work still :

{
    debug
    local_certs
    frankenphp
    order php_server before file_server
}

https://*.*.*.local:8080 {

    root * /app/public/

    log # enable basic log
    encode gzip zstd # compression

    @nuxtreq {
        header !X-Random-Non-Existing-Header
    }

    reverse_proxy @nuxtreq http://nuxt:3000

    php_server
}

with this config I get 400 Error
LOG

front-1  | 2024/04/03 17:16:09.005      DEBUG   http.handlers.reverse_proxy   selected upstream        {"dial": "front:8080", "total_upstreams": 1}
front-1  | 2024/04/03 17:16:09.006      DEBUG   http.handlers.reverse_proxy   upstream roundtrip       {"upstream": "front:8080", "duration": 0.001032488, "request": {"remote_ip": "172.20.0.1", "remote_port": "47450", "client_ip": "172.20.0.1", "proto": "HTTP/2.0", "method": "GET", "host": "metro.craft.fr.local:8080", "uri": "/fr_FR/", "headers": {"Sec-Fetch-Site": ["cross-site"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Sec-Ch-Ua-Mobile": ["?0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"], "Cookie": [], "X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["metro.craft.fr.local:8080"], "Sec-Ch-Ua": ["\"Google Chrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\""], "Accept-Language": ["en-US,en;q=0.9,fr-FR;q=0.8,fr;q=0.7"], "X-Forwarded-For": ["172.20.0.1"], "Sec-Fetch-Mode": ["navigate"], "Sec-Fetch-Dest": ["document"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"], "Sec-Fetch-User": ["?1"], "Cache-Control": ["max-age=0"], "Sec-Ch-Ua-Platform": ["\"Windows\""], "Upgrade-Insecure-Requests": ["1"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "metro.craft.fr.local"}}, "headers": {}, "status": 400}
front-1  | 2024/04/03 17:16:09.006      ERROR   http.handlers.reverse_proxy   reading from backend     {"error": "read tcp 172.20.0.6:52286->172.20.0.6:8080: read: connection reset by peer"}
front-1  | 2024/04/03 17:16:09.006      ERROR   http.handlers.reverse_proxy   aborting with incomplete response        {"upstream": "front:8080", "duration": 0.001032488, "request": {"remote_ip": "172.20.0.1", "remote_port": "47450", "client_ip": "172.20.0.1", "proto": "HTTP/2.0", "method": "GET", "host": "metro.craft.fr.local:8080", "uri": "/fr_FR/", "headers": {"Sec-Fetch-Site": ["cross-site"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Sec-Ch-Ua-Mobile": ["?0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"], "Cookie": [], "X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["metro.craft.fr.local:8080"], "Sec-Ch-Ua": ["\"Google Chrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\""], "Accept-Language": ["en-US,en;q=0.9,fr-FR;q=0.8,fr;q=0.7"], "X-Forwarded-For": ["172.20.0.1"], "Sec-Fetch-Mode": ["navigate"], "Sec-Fetch-Dest": ["document"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"], "Sec-Fetch-User": ["?1"], "Cache-Control": ["max-age=0"], "Sec-Ch-Ua-Platform": ["\"Windows\""], "Upgrade-Insecure-Requests": ["1"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "metro.craft.fr.local"}}, "error": "reading: read tcp 172.20.0.6:52286->172.20.0.6:8080: read: connection reset by peer"}
front-1  | 2024/04/03 17:16:09.006      ERROR   http.log.access handled request{"request": {"remote_ip": "172.20.0.1", "remote_port": "47450", "client_ip": "172.20.0.1", "proto": "HTTP/2.0", "method": "GET", "host": "metro.craft.fr.local:8080", "uri": "/fr_FR/", "headers": {"Upgrade-Insecure-Requests": ["1"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Accept-Language": ["en-US,en;q=0.9,fr-FR;q=0.8,fr;q=0.7"], "Cache-Control": ["max-age=0"], "Sec-Ch-Ua-Platform": ["\"Windows\""], "Sec-Fetch-Site": ["cross-site"], "Sec-Fetch-Mode": ["navigate"], "Sec-Fetch-Dest": ["document"], "Sec-Ch-Ua": ["\"Google Chrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\""], "Sec-Ch-Ua-Mobile": ["?0"], "Sec-Fetch-User": ["?1"], "Cookie": [], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "metro.craft.fr.local"}}, "bytes_read": 0, "user_id": "", "duration": 0, "size": 48, "status": 400, "resp_headers": {"Server": ["Caddy"], "Alt-Svc": ["h3=\":8080\"; ma=2592000"]}}

The only config that seems to work is when using a named matcher with the exclusion operator ! with header i.e :

    @nuxtreq {
        header !X-Random-Non-Existing-Head
    }
    reverse_proxy @nuxtreq http://nuxt:3000

But this approach is the opposite if my end goal.
I am missing somthing here or is there a chance this is an issue with the reverse_proxy it self?

I don’t know what to tell you. You haven’t shown your logs, or example requests with curl, so I don’t understand what the problem is at this point. You didn’t actually answer my questions, you just stated new unrelated information.

I don’t understand your goal at this point.

Oh, I forgot about the logs. I was trying different approach.
I updated my previous response with logs at both configs.

To better explain the situation, I have drawen the my technical setup in the design attached.

My Goal is to redirect specific responses coming from PHP/Symfony (status code 307 or having a specific header) to be handled by other service (nuxt) hilighted by the orange line

I failed so far to reach the nuxt service, because the nuxt logs are empty showing no hit.

I hope this makes it much more clearer.

Okay. I think it’s currently not possible, it would need to be implemented in the php_server handler. I’m in contact with @dunglas about this and he’s planning to work on this soon.

Currently only reverse_proxy (and by proxy php_fastcgi which is just a shortcut for a longer reverse_proxy config) supports handle_response. The php_server directive isn’t built on top of reverse_proxy so it doesn’t have this functionality yet.

Thanks @francislavoie for the reply. That’s more cleared now.

I’ll try to make it work with the php_fastcgi however I don’t know the performance impact, as frankenPHP is way faster.

I’ll post any updates here.

UPDATE : @dunglas just created a PR implementing this very feature called ´response_interceptors´ :