Caddy fails to redirect request based on response status code (not in reverse_proxy directive)

elghailani · March 28, 2024, 9:59pm

1. The problem I’m having:

Hello,

I have a PHP Symfony app (runs using frankenphp) that initially handles all the requests and if it cannot handle some specific ones, it returns a 307 response that needs to be re-routed to another nuxt app.

I used the handle_errors as suggested by the doc (see bellow).
I Used 3 instances of handle_errors in my Caddyfile just to be sure:

one in top level
the 2nd in the site config
the last within handle with named matcher

The nuxt service does NOT receive any 307 request from the Symfony app throught caddy reverse proxy.

I highly suspect that this is an issue related to error status matcher/expression…

2. Error messages and/or full log output:

front-1  | 2024/03/28 21:25:56.224      DEBUG   http.handlers.rewrite   rewrote request {"request": {"remote_ip": "172.20.0.1", "remote_port": "33094", "client_ip": "172.20.0.1", "proto": "HTTP/2.0", "method": "GET", "host": "metro.craft.fr.local:8080", "uri": "/fr_FR/", "headers": {"Cache-Control": ["no-cache"], "Upgrade-Insecure-Requests": ["1"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"], "Sec-Fetch-Site": ["none"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Pragma": ["no-cache"], "Sec-Fetch-Dest": ["document"], "Sec-Fetch-User": ["?1"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"], "Sec-Ch-Ua": ["\"Google Chrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\""], "Sec-Ch-Ua-Platform": ["\"Windows\""], "Sec-Fetch-Mode": ["navigate"], "Accept-Language": ["en-US,en;q=0.9,fr-FR;q=0.8,fr;q=0.7"], "Cookie": [], "Sec-Ch-Ua-Mobile": ["?0"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "metro.craft.fr.local"}}, "method": "GET", "uri": "/index.php"}
front-1  | 2024/03/28 21:25:56.244      DEBUG   request handling finished       {"worker": "/app/public/index.php", "url": "/index.php"}
front-1  | 2024/03/28 21:25:56.244      INFO    http.log.access handled request {"request": {"remote_ip": "172.20.0.1", "remote_port": "33094", "client_ip": "172.20.0.1", "proto": "HTTP/2.0", "method": "GET", "host": "metro.craft.fr.local:8080", "uri": "/fr_FR/", "headers": {"Sec-Fetch-User": ["?1"], "Sec-Fetch-Dest": ["document"], "Sec-Ch-Ua": ["\"Google Chrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\""], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"], "Cookie": [], "Sec-Ch-Ua-Mobile": ["?0"], "Sec-Ch-Ua-Platform": ["\"Windows\""], "Sec-Fetch-Mode": ["navigate"], "Accept-Language": ["en-US,en;q=0.9,fr-FR;q=0.8,fr;q=0.7"], "Sec-Fetch-Site": ["none"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Pragma": ["no-cache"], "Cache-Control": ["no-cache"], "Upgrade-Insecure-Requests": ["1"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "metro.craft.fr.local"}}, "bytes_read": 0, "user_id": "", "duration": 0.020900041, "size": 29, "status": 307, "resp_headers": {"Server": ["Caddy"], "X-Powered-By": ["PHP/8.3.4"], "X-User-Id": ["Anonymous"], "X-Debug-Token": ["297b07"], "Set-Cookie": [], "Expires": ["Thu, 28 Mar 2024 21:25:56 GMT"], "Alt-Svc": ["h3=\":8080\"; ma=2592000"], "X-Route-Name": ["homepage"], "X-Origin": ["-"], "X-Robots-Tag": ["noindex"], "Cache-Control": ["max-age=0, must-revalidate, no-store, private"], "Date": ["Thu, 28 Mar 2024 21:25:56 GMT"], "X-Order-Number": ["-"], "Content-Type": ["text/html; charset=UTF-8"], "X-Debug-Token-Link": ["https://metro.craft.fr.local:8080/_profiler/297b07"]}}

3. Caddy version:

Caddy v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

a. System environment:

FROM dunglas/frankenphp:sha-e7e0dbf-php8.3.4-bookworm

RUN \
    apt-get update && \
    apt-get install -y gnupg2 ca-certificates libgnutls30 && \
    curl -sL https://nodejs.org/download/release/v14.21.3/node-v14.21.3-linux-x64.tar.gz | tar xz -C /usr/local --strip-components=1 && \
    apt-get update && \
    apt-get install -y git

RUN install-php-extensions \
    zip \
    exif \
    intl


WORKDIR /app/

ARG USER="craft-front"

RUN curl -sS https://getcomposer.org/installer | php -- --2.2 && \
    mv composer.phar /usr/local/bin/composer

RUN useradd -ms /bin/bash ${USER} && \
    usermod -u 1000 ${USER};

RUN \
    chown -R ${USER}:${USER} /var/log/ /var/log/* && \
    chmod -R 777 /var/log/ /var/log/ && \
	# Use "adduser -D ${USER}" for alpine based distros
	#useradd -D ${USER}; \
	# Add additional capability to bind to port 80 and 443
	setcap CAP_NET_BIND_SERVICE=+eip /usr/local/bin/frankenphp && \
	# Give write access to /data/caddy and /config/caddy \
    chown -R ${USER}:${USER} /data/caddy && chown -R ${USER}:${USER} /config/caddy &&  \
    chmod +x /usr/local/bin/frankenphp && \
    chown -R ${USER}:${USER} /app/ &&  \
    chmod 755 /app/

USER ${USER}

RUN touch /var/log/php-franken.error.log &&  \
    touch /var/log/php-franken.access.log

ENV FRANKENPHP_CONFIG="worker /app/public/index.php"
ENV APP_RUNTIME='Runtime\FrankenPhpSymfony\Runtime'
ENV PHP_INI_DIR='/usr/local/etc/php'

USER root

RUN usermod -aG sudo ${USER}  && \
    chown -R ${USER}:${USER} /app && \
    chown -R root:root /etc/caddy && \
    chmod -R 777 /app /config /data/caddy/ /config/caddy/ && \
    chmod 775 /etc/caddy


EXPOSE 8080
EXPOSE 443

COPY . /app/
COPY ./devops/docker/frankenphp/php.ini /usr/local/etc/php/

RUN composer install

b. Command:

docker compose up front --force-recreate
# that run the startup command bellow
frankenphp run --watch --config ./devops/docker/caddy/config/Caddyfile --adapter caddyfile

c. Service/unit/compose file:

  front:
    build:
      context: craft-front/.
    cap_add:
      - NET_ADMIN
    command:
      frankenphp run --watch --config ./devops/docker/caddy/config/Caddyfile --adapter caddyfile
    volumes:
      - ./craft-front:/app
      - ./craft-front/devops/docker/caddy/config/Caddyfile:/etc/caddy/Caddyfile
    ports:
      - "9001:9001"
      - "3007:3007"
      - "12444:12444"
      - "8080:8080" # HTTP
      - "443:443" # HTTPS
      - "443:443/udp" # HTTP/3
    tty: true
    extra_hosts:
      host.docker.internal: host-gateway
    networks:
      - network

  nuxt:
    build: craft-front/devops/docker/ssr/.
    container_name: nuxt
    volumes:
      - ./craft-front:/var/www/craft-front:cached
    restart: always
    networks:
      - network
    environment:
      - STACK_ENV=dev
    command:
      bash -c "pm2 start --env local --no-daemon"

d. My complete Caddy config:

{
    debug
    servers {
        protocols h1 h2 h3
    }

    admin off
    local_certs
    frankenphp
    order php_server before file_server
}

handle_errors {
        @307 {
            expression `{http.error.status_code} == '307'`
        }
        handle @307 {
            reverse_proxy * http://nuxt:3000
        }
}


https://*.*.*.*.local:8080,
https://*.*.*.local:8080,
https://*.*.local:8080,
https://*.local:8080 {

    log # enable basic log

    encode gzip zstd # compression

    handle_errors {
        @307 expression {err.status_code} == 307
        handle @307 {
            reverse_proxy http://nuxt:3000
        }
    }

    handle {
        @307err expression {err.status_code} == 307
        handle @307err {
            reverse_proxy http://nuxt:3000
        }
        root * /app/public/
        php_server
    }
    respond 404
}

I read through the documentation, I know that handle_errors is bit special.
Also experimented with the redir to no avail.
I looked at (definitely not exhaustive list) :

Thanks for the help.

francislavoie · March 29, 2024, 8:46pm

Errors are not triggered when an upstream writes a response. You need to use reverse_proxy’s handle_response to intercept the responses either by status code or header, then use the error directive to trigger an error, which you can later handle with handle_errors.

Also, handle_errors is a directive, it must go within a site block. It can’t be top-level in your Caddyfile.

elghailani · March 29, 2024, 11:18pm

So if understand correctly, I need to set up a dedicated 1st reverse_proxy for the primary front app, then intercept 307 responses and re-route them through a 2nd reverse reverse_proxy?

I changed the Caddyfile to this, but now I get 400 error on all responses

{
    debug
    local_certs
    frankenphp
    order php_server before file_server
}

https://*.*.*.*.local:8080,
https://*.*.*.local:8080,
https://*.*.local:8080,
https://*.local:8080 {

    log # enable basic log
    encode gzip zstd # compression

    reverse_proxy * http://front:8080 {
        @307err status 307
        handle_response @307err {
            error 307
        }
    }

    handle_errors {
            @307 expression {err.status_code} == 307
            handle @307 {
                reverse_proxy http://nuxt:3000
            }
        }

    root * /app/public/
    php_server
}

francislavoie · March 31, 2024, 5:22pm

And what’s in your Caddy logs at this point? Show the debug logs, need to see what the responses from upstream look like.

With that config, I think you’re proxying all requests to front, and none make it to php_server. You probably need to use somekind of matcher to tell Caddy which requests to route where.

elghailani · April 3, 2024, 5:29am

So, during investigation, I changed the named matcher from status code to header attribute. The reverse proxy redirection does not work still :

{
    debug
    local_certs
    frankenphp
    order php_server before file_server
}

https://*.*.*.local:8080 {

    root * /app/public/

    log # enable basic log
    encode gzip zstd # compression

    @nuxtreq {
        header !X-Random-Non-Existing-Header
    }

    reverse_proxy @nuxtreq http://nuxt:3000

    php_server
}

with this config I get 400 Error
LOG

front-1  | 2024/04/03 17:16:09.005      DEBUG   http.handlers.reverse_proxy   selected upstream        {"dial": "front:8080", "total_upstreams": 1}
front-1  | 2024/04/03 17:16:09.006      DEBUG   http.handlers.reverse_proxy   upstream roundtrip       {"upstream": "front:8080", "duration": 0.001032488, "request": {"remote_ip": "172.20.0.1", "remote_port": "47450", "client_ip": "172.20.0.1", "proto": "HTTP/2.0", "method": "GET", "host": "metro.craft.fr.local:8080", "uri": "/fr_FR/", "headers": {"Sec-Fetch-Site": ["cross-site"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Sec-Ch-Ua-Mobile": ["?0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"], "Cookie": [], "X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["metro.craft.fr.local:8080"], "Sec-Ch-Ua": ["\"Google Chrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\""], "Accept-Language": ["en-US,en;q=0.9,fr-FR;q=0.8,fr;q=0.7"], "X-Forwarded-For": ["172.20.0.1"], "Sec-Fetch-Mode": ["navigate"], "Sec-Fetch-Dest": ["document"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"], "Sec-Fetch-User": ["?1"], "Cache-Control": ["max-age=0"], "Sec-Ch-Ua-Platform": ["\"Windows\""], "Upgrade-Insecure-Requests": ["1"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "metro.craft.fr.local"}}, "headers": {}, "status": 400}
front-1  | 2024/04/03 17:16:09.006      ERROR   http.handlers.reverse_proxy   reading from backend     {"error": "read tcp 172.20.0.6:52286->172.20.0.6:8080: read: connection reset by peer"}
front-1  | 2024/04/03 17:16:09.006      ERROR   http.handlers.reverse_proxy   aborting with incomplete response        {"upstream": "front:8080", "duration": 0.001032488, "request": {"remote_ip": "172.20.0.1", "remote_port": "47450", "client_ip": "172.20.0.1", "proto": "HTTP/2.0", "method": "GET", "host": "metro.craft.fr.local:8080", "uri": "/fr_FR/", "headers": {"Sec-Fetch-Site": ["cross-site"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Sec-Ch-Ua-Mobile": ["?0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"], "Cookie": [], "X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["metro.craft.fr.local:8080"], "Sec-Ch-Ua": ["\"Google Chrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\""], "Accept-Language": ["en-US,en;q=0.9,fr-FR;q=0.8,fr;q=0.7"], "X-Forwarded-For": ["172.20.0.1"], "Sec-Fetch-Mode": ["navigate"], "Sec-Fetch-Dest": ["document"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"], "Sec-Fetch-User": ["?1"], "Cache-Control": ["max-age=0"], "Sec-Ch-Ua-Platform": ["\"Windows\""], "Upgrade-Insecure-Requests": ["1"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "metro.craft.fr.local"}}, "error": "reading: read tcp 172.20.0.6:52286->172.20.0.6:8080: read: connection reset by peer"}
front-1  | 2024/04/03 17:16:09.006      ERROR   http.log.access handled request{"request": {"remote_ip": "172.20.0.1", "remote_port": "47450", "client_ip": "172.20.0.1", "proto": "HTTP/2.0", "method": "GET", "host": "metro.craft.fr.local:8080", "uri": "/fr_FR/", "headers": {"Upgrade-Insecure-Requests": ["1"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Accept-Language": ["en-US,en;q=0.9,fr-FR;q=0.8,fr;q=0.7"], "Cache-Control": ["max-age=0"], "Sec-Ch-Ua-Platform": ["\"Windows\""], "Sec-Fetch-Site": ["cross-site"], "Sec-Fetch-Mode": ["navigate"], "Sec-Fetch-Dest": ["document"], "Sec-Ch-Ua": ["\"Google Chrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\""], "Sec-Ch-Ua-Mobile": ["?0"], "Sec-Fetch-User": ["?1"], "Cookie": [], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "metro.craft.fr.local"}}, "bytes_read": 0, "user_id": "", "duration": 0, "size": 48, "status": 400, "resp_headers": {"Server": ["Caddy"], "Alt-Svc": ["h3=\":8080\"; ma=2592000"]}}

The only config that seems to work is when using a named matcher with the exclusion operator ! with header i.e :

    @nuxtreq {
        header !X-Random-Non-Existing-Head
    }
    reverse_proxy @nuxtreq http://nuxt:3000

But this approach is the opposite if my end goal.
I am missing somthing here or is there a chance this is an issue with the reverse_proxy it self?

francislavoie · April 3, 2024, 5:43am

I don’t know what to tell you. You haven’t shown your logs, or example requests with curl, so I don’t understand what the problem is at this point. You didn’t actually answer my questions, you just stated new unrelated information.

I don’t understand your goal at this point.

elghailani · April 3, 2024, 5:07pm

Oh, I forgot about the logs. I was trying different approach.
I updated my previous response with logs at both configs.

To better explain the situation, I have drawen the my technical setup in the design attached.

My Goal is to redirect specific responses coming from PHP/Symfony (status code 307 or having a specific header) to be handled by other service (nuxt) hilighted by the orange line

I failed so far to reach the nuxt service, because the nuxt logs are empty showing no hit.

I hope this makes it much more clearer.

francislavoie · April 3, 2024, 6:02pm

Okay. I think it’s currently not possible, it would need to be implemented in the php_server handler. I’m in contact with @dunglas about this and he’s planning to work on this soon.

Currently only reverse_proxy (and by proxy php_fastcgi which is just a shortcut for a longer reverse_proxy config) supports handle_response. The php_server directive isn’t built on top of reverse_proxy so it doesn’t have this functionality yet.

elghailani · April 8, 2024, 3:03pm

Thanks @francislavoie for the reply. That’s more cleared now.

I’ll try to make it work with the php_fastcgi however I don’t know the performance impact, as frankenPHP is way faster.

I’ll post any updates here.

elghailani · April 10, 2024, 1:38pm

UPDATE : @dunglas just created a PR implementing this very feature called ´response_interceptors´ :

github.com/caddyserver/caddy

feat: add generic response interceptors

caddyserver:master ← dunglas:feat/response-interceptors

opened 10:10AM - 10 Apr 24 UTC

dunglas

+599 -0

In proxy mode, Caddy allows [intercepting and modifying responses](https://caddy…server.com/docs/caddyfile/directives/reverse_proxy#intercepting-responses) coming from upstream servers. This is especially useful for implementing [`X-Sendfile`/`X-Accel-Redirect`-like features](https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/). However, the current implementation of this feature is only available in proxy mode and doesn't allow to intercept and modify responses created by modules. This makes it impossible to use features like ` X-Sendfile` with FrankenPHP, and other similar potential modules (LUA module etc). This patch implements a generic implementation of response interceptors that behaves similarly to the `reverse_proxy` `handle_response` subdirective. It currently does not support the `copy_response` and `copy_response_headers` subdirective, but it should be easy to add support for these features in future Pull Requests. It should also be possible, in the future, to replace the current `reverse_proxy` specific implementation with this new more generic one, but it is out of scope for this PR. Closes https://github.com/dunglas/frankenphp/issues/365. Closes https://github.com/dunglas/frankenphp/pull/396.

system · May 10, 2024, 1:38pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.