Caddy 2 reverse proxy - 502 for PiHole (and nothing else)

Peter_Ten_Eyck · April 22, 2024, 7:04pm

1. The problem I’m having:

Caddy reverse proxy works fine for most things on my local network, but gives a 502 error for my PiHole.

2. Error messages and/or full log output:

2024/04/22 18:41:14.429 ERROR   http.log.error  dial tcp: lookup dns.direct: i/o timeout        {"request": {"remote_ip": "192.168.86.26", "remote_port": "58600", "client_ip": "192.168.86.26", "proto": "HTTP/1.1", "method": "GET", "host": "dns.here", "uri": "/admin/cname_records.php", "headers": {"Cookie": [], "Connection": ["keep-alive"], "Upgrade-Insecure-Requests": ["1"], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"], "Accept-Encoding": ["gzip, deflate"], "Accept-Language": ["en-US,en;q=0.9,he-IL;q=0.8,he;q=0.7"]}}, "duration": 3.003471011, "status": 502, "err_id": "7r391sk8i", "err_trace": "reverseproxy.statusError (reverseproxy.go:1267)"}

(piped through jq)

2024/04/22 18:41:14.429 ERROR   http.log.error  dial tcp: lookup dns.direct: i/o timeout
{
  "request": {
    "remote_ip": "192.168.86.26",
    "remote_port": "58600",
    "client_ip": "192.168.86.26",
    "proto": "HTTP/1.1",
    "method": "GET",
    "host": "dns.here",
    "uri": "/admin/cname_records.php",
    "headers": {
      "Cookie": [],
      "Connection": [
        "keep-alive"
      ],
      "Upgrade-Insecure-Requests": [
        "1"
      ],
      "User-Agent": [
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"
      ],
      "Accept": [
        "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
      ],
      "Accept-Encoding": [
        "gzip, deflate"
      ],
      "Accept-Language": [
        "en-US,en;q=0.9,he-IL;q=0.8,he;q=0.7"
      ]
    }
  },
  "duration": 3.003471011,
  "status": 502,
  "err_id": "7r391sk8i",
  "err_trace": "reverseproxy.statusError (reverseproxy.go:1267)"
}

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

Installed stable version by following Debian/Ubuntu/Raspbian instructions

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy
caddy run

a. System environment:

Armbian 24.2.1 Jammy (Linux 6.6.16-current-meson64)

b. Command:

caddy run

c. Service/unit/compose file:

N/A

d. My complete Caddy config:

{
        servers {
                timeouts {
                        read_body 900s
                        read_header 900s
                }
        }
}

http://dns.here {
        handle {
                reverse_proxy dns.direct:80
        }
}

5. Links to relevant resources:

Not links, but other network info:

dns.here resolves to the Caddy host
dns.direct resolves to the host running PiHole. I can resolve this DNS successfully from the Caddy host using dig:

caddy $ dig dns.direct
...
;; ANSWER SECTION:
dns.direct.             0       IN      A       192.168.86.37
...

If I convert the request from the Caddy log into a curl command, I get the (expected) 302 response:

caddy $ curl -I -X GET \
  'http://dns.direct/admin/cname_records.php' \
  -H 'Connection: keep-alive' \
  -H 'Upgrade-Insecure-Requests: 1' \
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36' \
  -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7' \
  -H 'Accept-Encoding: gzip, deflate' \
  -H 'Accept-Language: en-US,en;q=0.9,he-IL;q=0.8,he;q=0.7'
...
HTTP/1.1 302 Found
Expires: Thu, 19 Nov 1981 08:52:00 GMT
...

Apparently that page expires in 1981…does Caddy’s reverse proxy implementation check page expiration date?
That 302 response takes ~15 seconds (for some reason), which lead me to think that Caddy might be terminating the request due to a timeout somewhere, since the Caddy error messages all show a "duration" of about 3 seconds.
Caddy’s Timeouts documentation states that the read_body and read_header timeout configurations default to “no timeout”, but I tried increasing them to 900 seconds anyway. After caddy reload, that didn’t seem to affect the issue.

francislavoie · April 23, 2024, 9:06am

That’s just to bust browser caches, I think. I don’t think it’s relevant.

That is quite long. There might be somekind of messed up TCP-layer config going on with your pihole.

Caddy’s reverse_proxy defaults to 3s dial_timeout (you can tune that) because it really shouldn’t take long to establish a TCP connection to a proxy upstream, especially if it’s in the same network.

Monviech · April 23, 2024, 12:28pm

I’m also running a pihole and could successfully reverse proxy it with this Caddyfile:

pihole.example.com {
        handle {
                reverse_proxy 10.1.1.250 {
                }
        }
}

Peter_Ten_Eyck · April 23, 2024, 1:16pm

I considered increasing the dial_timeout config to get this to work, but, as you pointed out, that would likely leave some underlying problem unaddressed:

So I instead tried the working approach described above:

Monviech:

I’m also running a pihole and could successfully reverse proxy it with this Caddyfile:
pihole.example.com {
        handle {
                reverse_proxy 10.1.1.250 {
                }
        }
}

The only difference is the use of the IP address instead of the DNS name. I swapped out dns.direct for the IP address of the pihole and that “solved” the problem.

I supposed I’m still leaving the root issue in place, but I have a working reverse proxy that doesn’t take 15 seconds to load the pihole login page, so I’m happy.

Thanks to both of you!

francislavoie · April 23, 2024, 4:55pm

Alright, so it’s clear that you have somekind of issue with DNS resolving.

Btw, you don’t need handle around your proxy, you can just make it a reverse_proxy one-liner inside your site block.