Reverse proxy - permission denied - upstream reachable

1. Caddy version (caddy version):

v2.4.3 h1:Y1FaV2N4WO3rBqxSYA8UZsZTQdN+PwcoOcAiZTM8C0I=

2. How I run Caddy:

a. System environment:

systemd

c. Service file:


[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace ":80" below with your
# domain name.
{
	admin off
	email "frank@somedomain.com"
}

git.kumro.io {
	reverse_proxy http://trinity:3000
}
# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile

3. The problem I’m having:

I set up a reverse proxy from a ramnode VPS to my home lab which is connected via tailscale. From the VPS running Caddy, I can curl the address listed in the error message and have gitea reply properly. When I access the site from outside the VPS, through Caddy, I get a 502 and the error is posted below.

4. Error messages and/or full log output:

Jan 16 00:43:34 matrix.kumro.io caddy[4536]: {"level":"error","ts":1642293814.5707703,"logger":"http.log.error","msg":"dial tcp 100.90.111.16:3000: connect: permission denied","request":{"remote_addr":"69.207.23.160:37854","proto":"HTTP/2.0","method":"GET","host":"git.kumro.io","uri":"/","headers":{"Dnt":["1"],"Te":["trailers"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"],"Accept-Language":["en-US,en;q=0.5"],"Cookie":["i_like_gitea=9701c0690ce5e9e7; gitea_awesome=frank; gitea_incredible=e7fde55c658f8472ff0115b0e3ca4e8a7a35a3112e8f0cdd93693b7f62df313c62; lang=en-US; _csrf=Cf9co1yhhbNaa_e1B3z4IRRMKdk6MTY0MjI4MzYzOTU3MzcwNzM3OQ"],"Sec-Fetch-Dest":["document"],"Sec-Gpc":["1"],"Sec-Fetch-User":["?1"],"User-Agent":["Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:96.0) Gecko/20100101 Firefox/96.0"],"Accept-Encoding":["gzip, deflate, br"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"git.kumro.io"}},"duration":0.005676763,"status":502,"err_id":"j1bm253zu","err_trace":"reverseproxy.statusError (reverseproxy.go:857)"}

5. What I already tried:

Curl / wget / ping the server in my home lab from the VPS running Caddy, it works. This was working before I rebooted the VPS and added a port forward for port 2222. Tailscale is running and I can reach the machine in my home lab from the VPS, I tried to rule out tailscale.

From the VPS running Caddy.

 curl -v -s trinity:3000
* Rebuilt URL to: trinity:3000/
*   Trying 100.90.111.16...
* TCP_NODELAY set
* Connected to trinity (100.90.111.16) port 3000 (#0)
> GET / HTTP/1.1
> Host: trinity:3000
> User-Agent: curl/7.61.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: text/html; charset=UTF-8
< Set-Cookie: i_like_gitea=0cb48a79a13cf3aa; Path=/; HttpOnly; SameSite=Lax
< Set-Cookie: _csrf=tQVk0tBR5Ifo_iulp-jAoRuoxV46MTY0MjI5MzU1ODQzMDM5OTc1Mw; Path=/; Expires=Mon, 17 Jan 2022 00:39:18 GMT; HttpOnly; SameSite=Lax
< Set-Cookie: macaron_flash=; Path=/; Max-Age=0; HttpOnly; SameSite=Lax
< X-Frame-Options: SAMEORIGIN
< Date: Sun, 16 Jan 2022 00:39:18 GMT
< Transfer-Encoding: chunked
< 

To fix this I had to uninstall caddy (dnf remove caddy) and then re-install it (dnf install caddy). After copying my Caddyfile back into /etc/caddy, everything worked. Not sure why…

Please upgrade to v2.4.6!

Do you have any particular reason to turn this off? Turning off the admin endpoint makes it impossible to reload Caddy with zero downtime.

Glad you figured it out :+1:

I’ve never seen a permission denied error for reverse proxy. Must be something related to SELinux or something? Not sure.

Now having read that you reinstalled, I figure you’re on the latest version of Caddy now :+1:

I don’t use the API, I manage everything via Caddyfile using Ansible and reload via systemd. If the admin endpoint is off, even reloading via systemd won’t work as intended? If so, I’ll remove that line.

I was thinking systemd related, that’s why I figured I would re-install so it can do whatever it needed to do. First jaunt into RockyLinux as a long time Ubuntu server user.

That’s right, reloads are done by pushing the new config to the admin endpoint to perform an in-memory config swap.

This topic was automatically closed after 30 days. New replies are no longer allowed.