Reverse proxy dial ipv6:port connect: connection refused but ping6 ipv6 receives response

1. The problem I’m having:

  1. The reverse proxy handler is giving a connection refused response.
  2. Pinging receives a response which makes it seem like the connection to the other server is alright.
  3. The other server (being reverse proxied to) does not seem to receive any request so I think the request stops at the Caddy proxy server.

I do not have a lot of experience with network configuration and I’m wondering how I can debug this further? I expected that pinging would not work and that I could use that to test further but since it works… I’m feeling at a loss for how to check what’s happening?

I’m sure this is not an issue with Caddy but hoping someone has an idea of what I can look into next.

2. Error messages and/or full log output:

"level":"error","ts":1684834534.0603356,"logger":"http.log.error.platform","msg":"dial tcp [ipv6]:8088: connect: connection refused".

user@host:/# ping6 ipv6
PING ipv6(ipv6) 56 data bytes
64 bytes from ipv6: icmp_seq=1 ttl=62 time=0.357 ms
64 bytes from ipv6: icmp_seq=2 ttl=62 time=0.377 ms
64 bytes from ipv6: icmp_seq=3 ttl=62 time=0.377 ms
64 bytes from ipv6: icmp_seq=4 ttl=62 time=0.421 ms

3. Caddy version:

2.6.4

4. How I installed and ran Caddy:

a. System environment:

Docker, Debian, run on Fly.io VM using Tailscale MagicDNS. Caddy issues the Tailscale HTTPS certificate just fine.

b. Command:

Starts container:

sysctl -w net.core.rmem_max=2048000

tailscaled --state=/var/lib/tailscale/tailscaled.state --socket=/var/run/tailscale/tailscaled.sock &
sleep 5
if [ ! -S /var/run/tailscale/tailscaled.sock ]; then
    echo "tailscaled.sock does not exist. exit!"
    exit 1
fi

until tailscale up --authkey=${TS_AUTHKEY} --hostname=${TS_HOSTNAME}
do
  sleep 0.1
done

caddy run --config /config/caddy/caddy.json

c. Service/unit/compose file:

If the Fly.io configuration is relevant I can post it.

d. My complete Caddy config:

{
  "apps": {
    "http": {
      "https_port": 443,
      "servers": {
        "services": {
          "listen": [":443"],
          "trusted_proxies": {
            "source": "static",
            "ranges": [
              "172.16.0.0/16",
              "172.19.0.0/16"
            ]
          },
          "logs": {
            "default_logger_name": "platform"
          },
          "routes": [
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "match": [
                        {
                          "host": [
                            "fly-host.tailnet-name.ts.net"
                          ]
                        }
                      ],
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "upstreams": [
                            {
                              "dial": "fly-host.internal:8088"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    }
  }
}

5. Links to relevant resources:

ping uses ICMP so it’s not a representative test for how Caddy tries to connect.

When using HTTP (which is a layer on top of TCP), I recommend you use curl -v to try to make an HTTP request instead. It’s a more representative test of what Caddy is trying to do.

Are you sure your upstream has port 8088 allowed in its firewall?

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.