Problems reaching Calibre-web behind Reverse Proxy

noja · January 26, 2024, 2:25pm

1. The problem I’m having:

I’ve been getting started with Caddy only in the last week, so I apologize for my poor proficiency. When I access the site through the reverse proxy, it is slow to load, doesn’t display all images and clicking any link will kick me immediately to the login screen again.

2. Error messages and/or full log output:

Jan 26 03:20:46 caddy caddy[1182]: {"level":"error","ts":1706239246.7281516,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"172.16.120.12:8083","duration":0.136355727,"request":{"remote_ip":"108.162.241.184","remote_port":"30134","client_ip":"108.162.241.184","proto":"HTTP/2.0","method":"GET","host":"library.hellmouth.ca","uri":"/cover/2020/sm?c=1705106337","headers":{"Sec-Fetch-Mode":["no-cors"],"X-Forwarded-For":["108.162.241.184"],"Accept-Language":["en-US,en;q=0.5"],"Cf-Ray":["84b59e890a7f36b5-YYZ"],"X-Forwarded-Host":["library.hellmouth.ca"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Priority":["u=4"],"X-Forwarded-Proto":["https"],"X-Scheme":["https"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0"],"Cf-Connecting-Ip":["MY.IP.ADDRESS"],"Cf-Ipcountry":["CA"],"Accept-Encoding":["gzip"],"Cookie":[],"Cdn-Loop":["cloudflare"],"Sec-Fetch-Dest":["image"],"Accept":["image/avif,image/webp,*/*"],"Sec-Fetch-Site":["same-origin"],"Referer":["https://library.hellmouth.ca/"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"library.hellmouth.ca"}},"error":"writing: client disconnected"}
Jan 26 03:21:14 caddy caddy[1182]: {"level":"error","ts":1706239274.6168056,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"172.16.120.12:8083","duration":0.065802308,"request":{"remote_ip":"108.162.241.184","remote_port":"29792","client_ip":"108.162.241.184","proto":"HTTP/2.0","method":"GET","host":"library.hellmouth.ca","uri":"/cover/2036/sm?c=1705106385","headers":{"X-Forwarded-Proto":["https"],"Cf-Ipcountry":["CA"],"Cf-Ray":["84b59e88ea3d36b5-YYZ"],"X-Forwarded-Host":["library.hellmouth.ca"],"X-Scheme":["https"],"Cookie":[],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-Mode":["no-cors"],"Accept":["image/avif,image/webp,*/*"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0"],"Cdn-Loop":["cloudflare"],"Referer":["https://library.hellmouth.ca/"],"Accept-Language":["en-US,en;q=0.5"],"Priority":["u=4"],"X-Forwarded-For":["108.162.241.184"],"Accept-Encoding":["gzip"],"Sec-Fetch-Dest":["image"],"Cf-Connecting-Ip":["MY.IP.ADDRESS"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"library.hellmouth.ca"}},"error":"writing: write tcp 172.16.120.90:443->108.162.241.184:29792: write: connection reset by peer"}
Jan 26 03:21:14 caddy caddy[1182]: {"level":"error","ts":1706239274.617087,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"172.16.120.12:8083","duration":0.390973112,"request":{"remote_ip":"108.162.241.184","remote_port":"30312","client_ip":"108.162.241.184","proto":"HTTP/2.0","method":"GET","host":"library.hellmouth.ca","uri":"/cover/1990/sm?c=1705106230","headers":{"Sec-Fetch-Mode":["no-cors"],"X-Forwarded-Host":["library.hellmouth.ca"],"Cdn-Loop":["cloudflare"],"Accept":["image/avif,image/webp,*/*"],"Sec-Fetch-Dest":["image"],"Cf-Connecting-Ip":["MY.IP.ADDRESS"],"Cf-Ipcountry":["CA"],"Accept-Language":["en-US,en;q=0.5"],"X-Scheme":["https"],"Sec-Fetch-Site":["same-origin"],"Cf-Ray":["84b59e892acb36b5-YYZ"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0"],"Cookie":[],"Accept-Encoding":["gzip"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"X-Forwarded-Proto":["https"],"Priority":["u=4"],"X-Forwarded-For":["108.162.241.184"],"Referer":["https://library.hellmouth.ca/"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"library.hellmouth.ca"}},"error":"writing: client disconnected"}
Jan 26 03:21:14 caddy caddy[1182]: {"level":"error","ts":1706239274.6171277,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"172.16.120.12:8083","duration":0.454309529,"request":{"remote_ip":"108.162.241.184","remote_port":"30260","client_ip":"108.162.241.184","proto":"HTTP/2.0","method":"GET","host":"library.hellmouth.ca","uri":"/cover/1995/sm?c=1705106258","headers":{"Sec-Fetch-Dest":["image"],"X-Forwarded-For":["108.162.241.184"],"Cf-Ipcountry":["CA"],"Priority":["u=4"],"Cf-Connecting-Ip":["MY.IP.ADDRESS"],"X-Scheme":["https"],"Sec-Fetch-Mode":["no-cors"],"X-Forwarded-Proto":["https"],"Accept":["image/avif,image/webp,*/*"],"Cf-Ray":["84b59e890ab336b5-YYZ"],"X-Forwarded-Host":["library.hellmouth.ca"],"Accept-Language":["en-US,en;q=0.5"],"Referer":["https://library.hellmouth.ca/"],"Sec-Fetch-Site":["same-origin"],"Cookie":[],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cdn-Loop":["cloudflare"],"Accept-Encoding":["gzip"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"library.hellmouth.ca"}},"error":"writing: write tcp 172.16.120.90:443->108.162.241.184:30260: write: connection reset by peer"}
Jan 26 03:21:15 caddy caddy[1182]: {"level":"error","ts":1706239275.1286507,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"172.16.120.12:8083","duration":0.098117157,"request":{"remote_ip":"108.162.241.184","remote_port":"29812","client_ip":"108.162.241.184","proto":"HTTP/2.0","method":"GET","host":"library.hellmouth.ca","uri":"/cover/2030/sm?c=1705106370","headers":{"Accept-Language":["en-US,en;q=0.5"],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-Mode":["no-cors"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Connecting-Ip":["MY.IP.ADDRESS"],"Referer":["https://library.hellmouth.ca/"],"Cf-Ray":["84b59e88ea4836b5-YYZ"],"X-Forwarded-For":["108.162.241.184"],"X-Scheme":["https"],"Accept-Encoding":["gzip"],"Cookie":[],"Priority":["u=4"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0"],"Cf-Ipcountry":["CA"],"Cdn-Loop":["cloudflare"],"X-Forwarded-Host":["library.hellmouth.ca"],"Sec-Fetch-Dest":["image"],"Accept":["image/avif,image/webp,*/*"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"library.hellmouth.ca"}},"error":"writing: client disconnected"}

3. Caddy version:

v2.7.6

4. How I installed and ran Caddy:

a. System environment:

I also edited the

systemd on Ubuntu server 22.04.3 - virtualized on Proxmox

b. Command:

systemctl start caddy

I have also edited the systemctl file to add the Cloudflare API token

[Service]
Environment="CF_API_TOKEN=I actually put my token in here"

c. Service/unit/compose file:

d. My complete Caddy config:

food.hellmouth.ca {
        reverse_proxy 172.16.120.12:3026
}

library.hellmouth.ca {
        reverse_proxy 172.16.120.12:8083 {
                header_up X-Scheme https
}

5. Links to relevant resources:

https://caddy.community/t/calibre-web-reverse-proxy-help/12023

https://caddy.community/t/calibre-web-giving-502-error-behind-caddy/19801

noja · January 26, 2024, 6:47pm

I think I figured it out. First I realized that I needed to allow Cloudflare into Caddy.

If someone from Caddy sees this, you may want to update your documentation as it took a random 3rd party site to learn how to install the Cloudflare module. Since I’m just using Caddy installed in an Ubuntu VM, there wasn’t much out there nor was it easy to find.

sudo caddy add-package github.com/caddy-dns/cloudflare

That’s very important.

Then I did what I already knew how to do and create a Cloudflare token which I insert into the ‘systemctl edit caddy’ file.

Following all of that, Calibre-web was still having the same issues.

What finally worked was setting the trusted proxies within the reverse proxy section for calibre web.

library.domain.tld {
        reverse_proxy 172.16.120.12:8083 {
                header_up X-Scheme https
		trusted_proxies 173.245.48.0/20 103.21.244.0/22 103.22.200.0/22 103.31.4.0/22 141.101.64.0/18 108.162.192.0/18 190.93.240.0/20 188.114.96.0/20 197.234.240.0/22 198.41.128.0/17 162.158.0.0/15 104.16.0.0/13 104.24.0.0/14 172.64.0.0/13 131.0.72.0/22
        }
}

I don’t know if the Cloudflare module was fully required or now, but it’s working now after adding the trusted proxies section.

For us homelabbers only, being able to set trusted proxies as a global option might be nice. I’m sure once I’ve used Caddy for a long time, I’ll learn the better way to go, but for now, it’s been really difficult as a beginner.

francislavoie · January 26, 2024, 7:19pm

If you didn’t use dns cloudflare in your config, then the plugin had no effect.

I strongly recommend using the servers > trusted_proxies global option instead, it enables a few extra features like having the correct client_ip in your access logs etc. See Global options (Caddyfile) — Caddy Documentation

noja · January 26, 2024, 9:28pm

Thanks for the look! I did also add the Cloudflare configuration to the Caddyfile

{
        acme_dns cloudflare {env.CF_API_TOKEN}
}

However, when I went to add the trusted_proxies as a global config like this:

{
        acme_dns cloudflare {env.CF_API_TOKEN}
}

{
	servers {
		trusted_proxies 173.245.48.0/20 103.21.244.0/22 103.22.200.0/22 103.31.4.0/22 141.101.64.0/18 108.162.192.0/18 190.93.240.0/20 188.114.96.0/20 197.234.240.0/22 198.41.128.0/17 162.158.0.0/15 104.16.0.0/13 104.24.0.0/14 172.64.0.0/13 131.0.72.0/22
	}
}

Caddy wouldn’t work after with “systemctl reload caddy” - the error being


Jan 26 16:18:11 caddy systemd[1]: Reloading Caddy...
Jan 26 16:18:11 caddy caddy[1272]: {"level":"info","ts":1706303891.2053142,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Jan 26 16:18:11 caddy caddy[1272]: Error: adapting config using caddyfile: parsing caddyfile tokens for 'servers': getting module named 'http.ip_sources.173.245.48.0/20': module not registered: http.i>
Jan 26 16:18:11 caddy systemd[1]: caddy.service: Control process exited, code=exited, status=1/FAILURE
Jan 26 16:18:11 caddy systemd[1]: Reload failed for Caddy.
Jan 26 16:19:49 caddy systemd[1]: Reloading Caddy...
Jan 26 16:19:49 caddy caddy[1289]: {"level":"info","ts":1706303989.7982037,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Jan 26 16:19:49 caddy caddy[1289]: Error: adapting config using caddyfile: parsing caddyfile tokens for 'servers': unrecognized servers option 'acme_dns', at /etc/caddy/Caddyfile:29
Jan 26 16:19:49 caddy systemd[1]: caddy.service: Control process exited, code=exited, status=1/FAILURE
Jan 26 16:19:49 caddy systemd[1]: Reload failed for Caddy.

Funny enough, I can’t find it now but searching through other posts on trusted proxies I think you were the one that mentioned that it had to be under the reverse_proxy setting. Was probably an old version though.

Any idea then where I’m going wrong on my global settings?

noja · January 26, 2024, 9:34pm

Never mind! I got there through trial and error.

Here’s my new global config with working trusted_proxies and dns

{
	acme_dns cloudflare {env.CF_API_TOKEN}
	servers {
		trusted_proxies static 173.245.48.0/20 103.21.244.0/22 103.22.200.0/22 103.31.4.0/22 141.101.64.0/18 108.162.192.0/18 190.93.240.0/20 188.114.96.0/20 197.234.240.0/22 198.41.128.0/17 162.158.0.0/15 104.16.0.0/13 104.24.0.0/14 172.64.0.0/13 131.0.72.0/22
	}
}

francislavoie · January 27, 2024, 8:13am

Since you’re using the Cloudflare DNS plugin, you could also use the GitHub - WeidiDeng/caddy-cloudflare-ip plugin to replace that list of static IPs for trusted proxies.

system · February 26, 2024, 8:13am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.