Reverse Proxy not working when listening on port 80

Firnin · September 8, 2021, 8:22pm

1. Caddy version (`caddy version`):

v2.4.4 h1:QBsN1jXEsCqRpKPBb8ebVnBNgPxwL50HINWWTuZ7evU=

2. How I run Caddy:

I installed caddy via apt-get. I run it via the caddy command sudo caddy start from the path where my Caddyfile is located (/etc/caddy/).

a. System environment:

Raspberry Pi 4 (4 GB)
Output from lsb_release -d:
Raspbian GNU/Linux 10 (buster)

b. Command:

sudo caddy start

d. My complete Caddyfile or JSON config:

{
debug
}

kilians-pi4:80 {
        tls internal
        reverse_proxy localhost:7080
        log {
                output file /etc/caddy/caddy.log {
                        roll_disabled
                }
        }
}

3. The problem I’m having:

I want to use caddy as a reverse proxy for my bitwarden container I am running on my pi in Docker. Before making it available external I want to get it up and running in my local network first.
When I specify caddy to listen to port 80 in my Caddyfile, it does not forward the connection to port 7080 (port of my Bitwarden Container) when I try to access the site from my browser (tried several: Chrome, Firefox, Edge - all on the latest version).
Via curl -v kilians-pi4:80 I get the desired output.

4. Error messages and/or full log output:

Browser error message:
ERR_CONNECTION_REFUSED
Output from curl -v kilians-pi4:80

* Expire in 0 ms for 6 (transfer 0x1cd38b0)
* Expire in 1 ms for 1 (transfer 0x1cd38b0)
* Expire in 0 ms for 1 (transfer 0x1cd38b0)
* Expire in 1 ms for 1 (transfer 0x1cd38b0)
* Expire in 0 ms for 1 (transfer 0x1cd38b0)
* Expire in 0 ms for 1 (transfer 0x1cd38b0)
* Expire in 1 ms for 1 (transfer 0x1cd38b0)
* Expire in 0 ms for 1 (transfer 0x1cd38b0)
* Expire in 0 ms for 1 (transfer 0x1cd38b0)
* Expire in 1 ms for 1 (transfer 0x1cd38b0)
* Expire in 0 ms for 1 (transfer 0x1cd38b0)
* Expire in 0 ms for 1 (transfer 0x1cd38b0)
* Expire in 2 ms for 1 (transfer 0x1cd38b0)
* Expire in 0 ms for 1 (transfer 0x1cd38b0)
* Expire in 0 ms for 1 (transfer 0x1cd38b0)
* Expire in 2 ms for 1 (transfer 0x1cd38b0)
* Expire in 0 ms for 1 (transfer 0x1cd38b0)
* Expire in 0 ms for 1 (transfer 0x1cd38b0)
* Expire in 2 ms for 1 (transfer 0x1cd38b0)
* Expire in 0 ms for 1 (transfer 0x1cd38b0)
* Expire in 0 ms for 1 (transfer 0x1cd38b0)
* Expire in 0 ms for 1 (transfer 0x1cd38b0)
*   Trying 127.0.1.1...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x1cd38b0)
* Connected to kilians-pi4 (127.0.1.1) port 80 (#0)
> GET / HTTP/1.1
> Host: kilians-pi4
> User-Agent: curl/7.64.0
> Accept: */*
>
2021/09/08 20:08:55.843 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "localhost:7080", "request": {"remote_addr": "127.0.0.1:34362", "proto": "HTTP/1.1", "method": "GET", "host": "kilians-pi4", "uri": "/", "headers": {"X-Forwarded-For": ["127.0.0.1"], "User-Agent": ["curl/7.64.0"], "Accept": ["*/*"], "X-Forwarded-Proto": ["http"]}}, "headers": {"Content-Length": ["1121"], "Date": ["Wed, 08 Sep 2021 20:08:55 GMT"], "Server": ["Rocket"], "Feature-Policy": ["accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'none'; camera 'none'; encrypted-media 'none'; fullscreen 'none'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; midi 'none'; payment 'none'; picture-in-picture 'none'; sync-xhr 'self' https://haveibeenpwned.com https://2fa.directory; usb 'none'; vr 'none'"], "Referrer-Policy": ["same-origin"], "X-Xss-Protection": ["1; mode=block"], "Content-Security-Policy": ["frame-ancestors 'self' chrome-extension://nngceckbapebfimnlniiiahkandclblb chrome-extension://jbkfoedolllekgbhcbcoahefnbanhhlh moz-extension://* ;"], "Content-Type": ["text/html; charset=utf-8"], "Cache-Control": ["public, max-age=600"], "X-Frame-Options": ["SAMEORIGIN"], "X-Content-Type-Options": ["nosniff"]}, "status": 200}
2021/09/08 20:08:55.845 INFO    http.log.access.log0    handled request {"request": {"remote_addr": "127.0.0.1:34362", "proto": "HTTP/1.1", "method": "GET", "host": "kilians-pi4", "uri": "/", "headers": {"User-Agent": ["curl/7.64.0"], "Accept": ["*/*"]}}, "common_log": "127.0.0.1 - - [08/Sep/2021:22:08:55 +0200] \"GET / HTTP/1.1\" 200 1121", "user_id": "", "duration": 0.004460477, "size": 1121, "status": 200, "resp_headers": {"Content-Length": ["1121"], "Date": ["Wed, 08 Sep 2021 20:08:55 GMT"], "Feature-Policy": ["accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'none'; camera 'none'; encrypted-media 'none'; fullscreen 'none'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; midi 'none'; payment 'none'; picture-in-picture 'none'; sync-xhr 'self' https://haveibeenpwned.com https://2fa.directory; usb 'none'; vr 'none'"], "Content-Security-Policy": ["frame-ancestors 'self' chrome-extension://nngceckbapebfimnlniiiahkandclblb chrome-extension://jbkfoedolllekgbhcbcoahefnbanhhlh moz-extension://* ;"], "Server": ["Caddy", "Rocket"], "Content-Type": ["text/html; charset=utf-8"], "Cache-Control": ["public, max-age=600"], "X-Xss-Protection": ["1; mode=block"], "X-Frame-Options": ["SAMEORIGIN"], "X-Content-Type-Options": ["nosniff"], "Referrer-Policy": ["same-origin"]}}
< HTTP/1.1 200 OK
< Cache-Control: public, max-age=600
< Content-Length: 1121
< Content-Security-Policy: frame-ancestors 'self' chrome-extension://nngceckbapebfimnlniiiahkandclblb chrome-extension://jbkfoedolllekgbhcbcoahefnbanhhlh moz-extension://* ;
< Content-Type: text/html; charset=utf-8
< Date: Wed, 08 Sep 2021 20:08:55 GMT
< Feature-Policy: accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'none'; camera 'none'; encrypted-media 'none'; fullscreen 'none'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; midi 'none'; payment 'none'; picture-in-picture 'none'; sync-xhr 'self' https://haveibeenpwned.com https://2fa.directory; usb 'none'; vr 'none'
< Referrer-Policy: same-origin
< Server: Caddy
< Server: Rocket
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Xss-Protection: 1; mode=block
<
<!doctype html><html><head><meta charset="utf-8"><meta name="viewport" content="width=1010"><meta name="theme-color" content="#175DDC"><title page-title>Bitwarden Web Vault</title><link rel="apple-touch-icon" sizes="180x180" href="images/icons/apple-touch-icon.png"><link rel="icon" type="image/png" sizes="32x32" href="images/icons/favicon-32x32.png"><link rel="icon" type="image/png" sizes="16x16" href="images/icons/favicon-16x16.png"><link rel="mask-icon" href="images/icons/safari-pinned-tab.svg" color="#175DDC"><link rel="manifest" href="manifest.json"><link href="app/main.b98333fa8f5897046237.css" rel="stylesheet"></head><body class="layout_frontend"><app-root><div class="mt-5 d-flex justify-content-center"><div><img src="images/logo-dark@2x.png" class="mb-4 logo" alt="Bitwarden"><p class="text-center"><i class="fa fa-spinner fa-spin fa-2x text-muted" title="Loading" aria-hidden="true"></i></p></div></div></app-root><script src="app/polyfills.b98333fa8f5897046237.js"></script><script src="app/vendor.b98333f* Connection #0 to host kilians-pi4 left intact
a8f5897046237.js"></script><script src="app/main.b98333fa8f5897046237.js"></script></body></html>

5. What I already tried:

I tried setting caddy to listen on another port (i.e. 8080 or 700) and to the hostname itself (‘kilians-pi4’, without any port specified).
Strangely enough, Chrome (and other browsers) have no problems establishing the connection.
I know it can be circumvented by using other ports, but it really bugs me that adding other ports to the Caddyfile brings up the “bind address already in use” - error since I plan to host a couple of other services on the pi later on.
Can anyone help me out?

6. Links to relevant resources:

francislavoie · September 8, 2021, 8:29pm

Please don’t use caddy start if this is meant to be a long-running server. See this article on how to run Caddy as a systemd service:

The best place to put your logs is in /var/log/caddy instead.

Caddy is proxying requests just fine, as evidenced by your curl -v command.

It’s requests from your browser that aren’t reaching Caddy.

How are you making the request from your browser? Are you sure that hostname resolves to the correct IP address? Does you pi have some ipfilter or ufw rules blocking incoming requests from other machines?

Firnin · September 8, 2021, 8:48pm

I will look into this, I am currently trying to figure everything out before running caddy as systemd. But thanks for the link

Will do

I enter “//kilians-pi4” in the address bar. I am sure the hostname resolves to the correct IP, since pings via CMD and connections via the browser (when caddy is listening on different ports) were successful.
I do not have an IP filter running on my pi (at least I haven’t installed or configured any), nor any UFW rules I am aware of.

francislavoie · September 8, 2021, 9:06pm

You should definitely start with using Caddy as a service, because it’s set up for you already right out of the gate with the apt package, and it’s how you’ll end up using it anyway.

Using caddy start is meant for quick startup during local development on your own machine, not so much for when setting up Caddy as a long-running server.

Well, there’s no evidence that it’s a problem with Caddy, so it must be a problem with your networking setup.

Firnin · September 8, 2021, 9:10pm

Hm okay, so I will have to take a deeper look into this…
But can you give me any advice how to handle the “bind address already in use” message that appears when adding another address for caddy to listen to?

francislavoie · September 8, 2021, 9:11pm

Caddy will already be running as a service, so if you’re running two instances of Caddy, they won’t be able to both bind to the same port. Only one process can bind to a port at any given time.

Basically, just don’t use the same port with 2 different processes.

Caddy should be able to bind to ports 80 and 443, i.e. the HTTP and HTTPS ports, especially if you want to make use of Caddy’s Automatic HTTPS.

system · October 8, 2021, 8:22pm

This topic was automatically closed after 30 days. New replies are no longer allowed.