Caddy not reliably reverse proxying Jenkins anymore

1. Caddy version (caddy version):

v2.5.1 h1:bAWwslD1jNeCzDa+jDCNwb8M3UJ2tPa8UZFFzPVmGKs=
reverted to
v2.4.6 h1:HGkGICFGvyrodcqOOclHKfvJC0qTU7vny/7FhYp9hNw=

2. How I run Caddy:

Ubuntu 20.04, systemd, sudo service caddy start/stop/reload

a. System environment:

Ubuntu 20.04, Jenkins 2.346, Caddy 2.5.1

b. Command:

sudo service caddy start

c. Service/unit/compose file:

Still have no idea where this is

d. My complete Caddyfile or JSON config:

{
        admin off
}
(logging) {
        log {
                output file /var/log/caddy/caddy.log {
                        roll_size 15mb
                        roll_keep 20
                }
        }
}
(errors) {
        handle_errors {
                root * /var/www/internal/errors
                rewrite * /{http.error.status_code}.html
                file_server
        }
}
(php) {
        php_fastcgi unix//run/php/php8.1-fpm.sock
}
www.telesphoreo.me {
        import logging
        redir https://telesphoreo.me{uri}
}
telesphoreo.me {
        import logging
        import errors
        import php
        root * /var/www/telesphoreo.me
        file_server browse
        encode gzip zstd
        @denied path /assets/ /old_html/* /new_html/* /recyclebin/* /nitrogen/ /wave/
        error @denied 403
}
blog.telesphoreo.me {
        import logging
        import php
        root * /var/www/blog.telesphoreo.me
        file_server
        encode gzip
}
db2.telesphoreo.me {
        import logging
        import php
        root * /usr/share/phpmyadmin
        file_server
}
nexus.telesphoreo.me {
        reverse_proxy http://localhost:8082
}
panel.telesphoreo.me {
        import logging
        php_fastcgi unix//run/php/php8.0-fpm.sock
        root * /var/www/pterodactyl/public
        file_server
        header X-Content-Type-Options nosniff
        header X-XSS-Protection "1; mode=block"
        header X-Robots-Tag none
        header Content-Security-Policy "frame-ancestors 'self'"
        header X-Frame-Options DENY
        header Referrer-Policy same-origin
        request_body {
                max_size 100m
        }
        respond /.ht* 403
}
pictochat.telesphoreo.me {
        import logging
        reverse_proxy http://localhost:8080
}
wordle.telesphoreo.me {
        import logging
        root * /var/www/wordle.telesphoreo.me/games/wordle
        file_server
        encode gzip
}
ci.plex.us.org {
        reverse_proxy http://localhost:8081
}
discord.plex.us.org {
        import logging
        redir https://discord.com/invite/HZsdUnsRKc
}
docs.plex.us.org {
        import logging
        redir https://plex.us.org{uri}
}
httpd.plex.us.org {
        import logging
        reverse_proxy 172.18.0.1:27192
}
plan.plex.us.org {
        import logging
        reverse_proxy 172.18.0.1:8804
}
plex.us.org {
        import logging
        root * /var/www/plexus.org/build
        respond /updater/check/ "1.0.3"
        file_server
}
forum.plex.us.org {
        import logging
        import php
        root * /var/www/forum.plexus.org/public
        file_server
        header /assets {
                +Cache-Control "public, must-revalidate, proxy-revalidate"
                +Cache-Control "max-age=25000"
                Pragma "public"
        }
        respond /.ht* 403
}
www.smokes-crystal.rocks {
        import logging
        redir https://smokes-crystal.rocks{uri}
}
smokes-crystal.rocks {
        import logging
        import errors
        root * /var/www/smokes-crystal.rocks
        file_server
}

3. The problem I’m having:

Ever since Caddy v2.5.0, Jenkins has not been working reliably. I run it using the official Jenkins weekly image. Ever since v2.5.0, I’ve been getting “Error 403 No valid crumb submitted in the request”. I was ignoring it at first, but it was happening so frequently it was driving me insane. So I googled it and it appears that it happens whenever you submit a form. There is an option to “better support proxies” which is vague but is less secure. It’s very easy to reproduce. All I have to do is go anywhere there is a form, and submit the data. The error seems to occur if I either switch tabs or am on the form for too long. If I complete the form fast on v2.5.1, it works. I downgraded to Caddy v2.4.6 and the issue is no longer present. I can be on a form for as long as I like, switch tabs as many times as I want, and the form will still go through. I also tried this on v2.5.0 and I got the same problems as v2.5.1.

4. Error messages and/or full log output:

I genuinely have no idea where any errors are. I searched through my caddy.log file but can’t find anything relevant relating to it. There isn’t even a single instance of “ci.plex.us.org” in the file upon further inspection.

5. What I already tried:

Downgrading to Caddy v2.4.6. This is the solution but I don’t want to stay on v2.4.6 of Caddy for the rest of my life. I really don’t have any idea how to approach this issue because I haven’t had any errors until upgrading to v2.5.0. It seems as if it’s a bug so I am not sure.

6. Links to relevant resources:

https://ci.plex.us.org
spinnaker - Jenkins: 403 No valid crumb was included in the request - Stack Overflow ← suggestion to enable enable proxy compatibility, but not recommended. Issue was still happening even after enabling this by the way

This would be your systemd service file; you can see the path to the service file when you run service caddy status, near the top. If you just installed Caddy with the apt repo, then you’re probably using the default, so that’s fine.

You never enabled logging for that one.

Are you running your server behind some other proxy? We made changes to reverse_proxy; it no longer implicitly trusts X-Forwarded-* header values from clients, unless configured to do so. See the release notes for v2.5.0

Turn on the debug global option when testing this, and show your logs.

2 Likes

Right, my apologies, I’ve added logging and added debug. I’ve gone into notepad++ and filtered out the logs to only include lines with “ci.plex.us.org

https://telesphoreo.me/caddy.log

This is on Caddy v2.5.1

I can do the same test for 2.4.6 if you would like

All I am doing is creating a new project called “Module-Test” and selecting an existing project to duplicate it from. I can pretty much get the error anywhere I have to input data though

Edit: to further answer, yes I installed Caddy using the apt repo so it’s the default. I am not running it behind any other proxy that I am aware of. It’s simply the official Jenkins docker image. I think the embedded server is Jetty but I don’t think that’s proxying it.

Here’s a screenshot of the error, but nothing descriptive

From the logs, looks like your have Cloudflare in front of your server. That’s the issue; Caddy’s reverse_proxy no longer implicitly trusts the X-Forwarded-For headers sent by upstreams proxies, to prevent spoofing.

You’ll need to configure the trusted_proxies option with Cloudflare’s IP ranges.

2 Likes

Perhaps I’m doing it wrong but I’m trying to do it for better code reusability like this

(cloudflare) {
        reverse_proxy {
                trusted_proxies 173.245.48.0/20 103.21.244.0/22 103.22.200.0/22 103.31.4.0/22 \
 141.101.64.0/18 108.162.192.0/18 190.93.240.0/20 188.114.96.0/20 \
 197.234.240.0/22 198.41.128.0/17 162.158.0.0/15 104.16.0.0/13 \
                104.24.0.0/14 172.64.0.0/13 131.0.72.0/22
        }
}

and then adding import cloudflare to the ci.plex.us.org section. I get error 502 bad gateway with this. For reference I’m using https://www.cloudflare.com/ips-v4

You removed the upstream address!

I think what you actually want is this:

(cloudflare) {
	trusted_proxies 173.245.48.0/20 103.21.244.0/22 103.22.200.0/22 103.31.4.0/22 141.101.64.0/18 108.162.192.0/18 190.93.240.0/20 188.114.96.0/20 197.234.240.0/22 198.41.128.0/17 162.158.0.0/15 104.16.0.0/13 104.24.0.0/14 172.64.0.0/13 131.0.72.0/22
}

Then later:

reverse_proxy http://localhost:8081 {
	import cloudflare
}

trusted_proxies isn’t recognized as a directive. It was my assumption to wrap it in the reverse_proxy { … } first

It’s a subdirective of reverse_proxy. But you can’t define two reverse_proxy, one with an upstream and one without; that doesn’t make sense.

Alright, I’ll just give up on that idea then. I added the trusted_proxies to just the ci.plex.us.org one. I wasn’t able to immediately reproduce which is a good sign. I’ll keep trying to use it normally and see but I indeed think that was the problem

I just gave you the config you need though.

Can confirm after more testing that was the solution. Thanks.

2 Likes