Reverse proxy dial ipv6:port connect: connection refused but ping6 ipv6 receives response

1. The problem I’m having:

  1. The reverse proxy handler is giving a connection refused response.
  2. Pinging receives a response which makes it seem like the connection to the other server is alright.
  3. The other server (being reverse proxied to) does not seem to receive any request so I think the request stops at the Caddy proxy server.

I do not have a lot of experience with network configuration and I’m wondering how I can debug this further? I expected that pinging would not work and that I could use that to test further but since it works… I’m feeling at a loss for how to check what’s happening?

I’m sure this is not an issue with Caddy but hoping someone has an idea of what I can look into next.

2. Error messages and/or full log output:

"level":"error","ts":1684834534.0603356,"logger":"http.log.error.platform","msg":"dial tcp [ipv6]:8088: connect: connection refused".

user@host:/# ping6 ipv6
PING ipv6(ipv6) 56 data bytes
64 bytes from ipv6: icmp_seq=1 ttl=62 time=0.357 ms
64 bytes from ipv6: icmp_seq=2 ttl=62 time=0.377 ms
64 bytes from ipv6: icmp_seq=3 ttl=62 time=0.377 ms
64 bytes from ipv6: icmp_seq=4 ttl=62 time=0.421 ms

3. Caddy version:

2.6.4

4. How I installed and ran Caddy:

a. System environment:

Docker, Debian, run on Fly.io VM using Tailscale MagicDNS. Caddy issues the Tailscale HTTPS certificate just fine.

b. Command:

Starts container:

sysctl -w net.core.rmem_max=2048000

tailscaled --state=/var/lib/tailscale/tailscaled.state --socket=/var/run/tailscale/tailscaled.sock &
sleep 5
if [ ! -S /var/run/tailscale/tailscaled.sock ]; then
    echo "tailscaled.sock does not exist. exit!"
    exit 1
fi

until tailscale up --authkey=${TS_AUTHKEY} --hostname=${TS_HOSTNAME}
do
  sleep 0.1
done

caddy run --config /config/caddy/caddy.json

c. Service/unit/compose file:

If the Fly.io configuration is relevant I can post it.

d. My complete Caddy config:

{
  "apps": {
    "http": {
      "https_port": 443,
      "servers": {
        "services": {
          "listen": [":443"],
          "trusted_proxies": {
            "source": "static",
            "ranges": [
              "172.16.0.0/16",
              "172.19.0.0/16"
            ]
          },
          "logs": {
            "default_logger_name": "platform"
          },
          "routes": [
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "match": [
                        {
                          "host": [
                            "fly-host.tailnet-name.ts.net"
                          ]
                        }
                      ],
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "upstreams": [
                            {
                              "dial": "fly-host.internal:8088"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    }
  }
}

5. Links to relevant resources:

ping uses ICMP so it’s not a representative test for how Caddy tries to connect.

When using HTTP (which is a layer on top of TCP), I recommend you use curl -v to try to make an HTTP request instead. It’s a more representative test of what Caddy is trying to do.

Are you sure your upstream has port 8088 allowed in its firewall?