Portainer Edge Agent gets heartbeat but no connection behind reverse proxy

1. Output of caddy version:

2.6.0

2. How I run Caddy:

a. System environment:

Raspberry Pi 4 4gb, RaspberryPi OS lite
docker

b. Command:

sudo docker-compose up -d

c. Service/unit/compose file:

services:
  caddy:
    image: caddy:latest
    container_name: caddy
    restart: always
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - ./caddy-config:/config
      - ./caddy-data:/data
    environment:
      - DOMAIN=myexampledomain.de
      - EMAIL=mail@myexampledomain.de
      - LOG_FILE=/data/access.log
    network_mode: host

d. My complete Caddy config:

portainer.{$DOMAIN}:443 {
  tls {$EMAIL}
  reverse_proxy 192.168.178.3:9233
}

portainer.{$DOMAIN}:8000 {
  tls {$EMAIL}
  reverse_proxy 192.168.178.3:8111
}

3. The problem I’m having:

Since a while I try now to connect my portainer instance at home to the home server at my mothers house. For that I red, the safest way to use is edge agent.

At home I run portainer behind a caddy reverse proxy on a raspberry pi 4.
Caddy is configured to route portainer.myexampledomain.de:443 to 192.168.178.3:9233 and portainer.myexampledomain.de:8000 to port 192.168.178.3:8111. In Docker I configured, 8111:8000 and 9233:9000 in the portainer compose file on my server.
I opened up port 8000 and 443 in my router (tcp and udp). Also in ufw I allowed port 443 and 8000.

Portainer and caddy for other services is working well so far.

On my mothers server I opened up port 9001 in the router. Ufw is also configured so far.

Now I tried to set up edge agent. I used:

sudo docker run -d \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /var/lib/docker/volumes:/var/lib/docker/volumes \
  -v /:/host \
  -v portainer_agent_data:/data \
  --restart always \
  -e EDGE=1 \
  -e EDGE_ID=----------------------------------- \
  -e EDGE_KEY=-------------------------------------------------------------- \
  -e EDGE_INSECURE_POLL=1 \
  --name portainer_edge_agent \
  portainer/agent:2.15.0

4. Error messages and/or full log output:

I now can see a heartbeat under Environments but if I try to connect it says Failed loading environment Environment is unreachable.

The portainer logs say:

time="2022-09-15T22:18:25+02:00" level=info msg="2022/09/15 22:18:25 [DEBUG] [chisel,monitoring] [endpoint_id: 8] [status: ACTIVE] [status_time_seconds: 7.157875] [message: environment tunnel monitoring]"
time="2022-09-15T22:18:28+02:00" level=info msg="2022/09/15 22:18:28 http: proxy error: dial tcp 127.0.0.1:51018: connect: connection refused"
time="2022-09-15T22:23:30+02:00" level=info msg="2022/09/15 22:23:30 http error: Unable to find the container (err=Error: No such container: 3bfdd889277c8539ed7f13f4df61339c6821c53ad3a5a404730793545eab88c6) (code=404)"
time="2022-09-15T22:23:30+02:00" level=info msg="2022/09/15 22:23:30 http error: Unable to find the container (err=Error: No such container: dae984b1b0af5e2ab7d8a7d4a8f4d04f8d278091412641c87250d3700a5d10dd) (code=404)"
time="2022-09-15T22:34:45+02:00" level=info msg="2022/09/15 22:34:45 [DEBUG] [chisel,monitoring] [endpoint_id: 8] [status: ACTIVE] [status_time_seconds: 3.817940] [message: environment tunnel monitoring]"
time="2022-09-15T22:34:50+02:00" level=info msg="2022/09/15 22:34:50 http: proxy error: dial tcp 127.0.0.1:64692: connect: connection refused"
time="2022-09-15T22:37:04+02:00" level=info msg="2022/09/15 22:37:04 [DEBUG] [chisel,monitoring] [endpoint_id: 8] [status: ACTIVE] [status_time_seconds: 8.822090] [message: environment tunnel monitoring]"
time="2022-09-15T22:37:05+02:00" level=info msg="2022/09/15 22:37:05 http: proxy error: dial tcp 127.0.0.1:55147: connect: connection refused"
time="2022-09-15T22:41:24+02:00" level=info msg="2022/09/15 22:41:24 [DEBUG] [chisel,monitoring] [endpoint_id: 8] [status: REQUIRED] [status_time_seconds: 0.182232] [message: environment tunnel monitoring]"
time="2022-09-15T22:41:34+02:00" level=info msg="2022/09/15 22:41:34 http: proxy error: dial tcp 127.0.0.1:65013: connect: connection refused"
time="2022-09-15T23:12:44+02:00" level=info msg="2022/09/15 23:12:44 [DEBUG] [chisel,monitoring] [endpoint_id: 8] [status: REQUIRED] [status_time_seconds: 1.361693] [message: environment tunnel monitoring]"
time="2022-09-15T23:12:53+02:00" level=info msg="2022/09/15 23:12:53 http: proxy error: dial tcp 127.0.0.1:60140: connect: connection refused"
time="2022-09-15T23:13:34+02:00" level=info msg="2022/09/15 23:13:34 [DEBUG] [chisel,monitoring] [endpoint_id: 8] [status: ACTIVE] [status_time_seconds: 2.269864] [message: environment tunnel monitoring]"
time="2022-09-15T23:13:41+02:00" level=info msg="2022/09/15 23:13:41 http: proxy error: dial tcp 127.0.0.1:61949: connect: connection refused"
time="2022-09-15T23:14:34+02:00" level=info msg="2022/09/15 23:14:34 [DEBUG] [chisel,monitoring] [endpoint_id: 8] [status: ACTIVE] [status_time_seconds: 2.271884] [message: environment tunnel monitoring]"
time="2022-09-15T23:14:41+02:00" level=info msg="2022/09/15 23:14:41 http: proxy error: dial tcp 127.0.0.1:60159: connect: connection refused"

The Agent logs:

2022/09/16 08:59:53 [INFO] [main] [message: Agent running on Docker platform]
2022/09/16 08:59:53 [INFO] [edge] [message: Edge key loaded from options]
2022/09/16 08:59:53 [INFO] [edge,registry] [message: Starting registry credential server]
2022/09/16 08:59:53 [INFO] [http] [server_addr: 172.01.02.03] [server_port: 9001] [use_tls: false] [api_version: 2.15.0] [message: Starting Agent API server]
2022/09/16 09:00:38 client: Connecting to ws://portainer.myexampledomain.de:8000
2022/09/16 09:00:38 client: Connection error: websocket: bad handshake
2022/09/16 09:00:38 client: Give up
2022/09/16 09:01:38 client: Connecting to ws://portainer.myexampledomain.de:8000
2022/09/16 09:01:38 client: Connection error: websocket: bad handshake
2022/09/16 09:01:38 client: Give up
2022/09/16 09:02:38 client: Connecting to ws://portainer.myexampledomain.de:8000
2022/09/16 09:02:38 client: Connection error: websocket: bad handshake
2022/09/16 09:02:38 client: Give up
2022/09/16 09:03:38 client: Connecting to ws://portainer.myexampledomain.de:8000
2022/09/16 09:03:38 client: Connection error: websocket: bad handshake

And for the sake of completeness here also my portainer docker-compose.yml:

version: '3'

networks:
  caddy:
    external: true

services:
  portainer:
    image: portainer/portainer-ce:latest
    command: -H unix:///var/run/docker.sock
    container_name: portainer
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./portainer-data:/data
    ports:
      - 9233:9000
      - 8111:8000
    networks:
      caddy:
        ipv4_address: 192.168.112.8
        ipv6_address: 2001:ab12::8

5. What I already tried:

I tried a few different caddy configurations, like:

tcp://portainer.{$DOMAIN}:8000 {
  tls {$EMAIL}
  reverse_proxy 192.168.178.3:8111
}

or

ws://portainer.{$DOMAIN}:8000 {
  tls {$EMAIL}
  reverse_proxy ws://192.168.178.3:8111
}

I oppened up ports, disabled ufw, cleared up iptables and a lot of little stuff more…

Im not sure if it is a problem with the agent, portainer or caddy but I hope some of you has an idea!

6. Links to relevant resources:

https://portal.portainer.io/knowledge/how-does-portainer-secure-connectivity-to-and-from-agents-and-edge-agents