Reverse-proxied services cannot cURL anything hosted on the same Caddy instance

hampy · February 3, 2023, 12:12am

1. Caddy version:

2.6.2

2. How I installed, and run Caddy:

a. System environment:

Ubuntu Server 22.04
Docker version 20.10.12, build 20.10.12-0ubuntu4

b. Command:

docker compose up -d

c. Service/unit/compose file:

version: "2"

services:
  caddy:
    image: caddy:2.6.2
    restart: always
    ports:
      - 80:80
      - 443:443
      - 443:443/udp
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - ./caddy-data:/data
      - ./public_html:/var/www
    links:
      - nextcloud
      - collabora
    volumes_from:
      - nextcloud
    labels:
      UFW_MANAGED: "TRUE"

  nextcloud:
    image: nextcloud:25.0.3-fpm
    restart: always
    volumes:
      - ./nextcloud:/var/www/html
    links:
      - collabora

  collabora:
    image: collabora/code:22.05.10.1.1
    restart: always
    environment:
      server_name: collabora.teaguemillette.com
      aliasgroup1: https://nc.teaguemillette.com|https://nc.teaguemillette.com
      extra_params: --o:ssl.enable=false --o:ssl.termination=true
      DONT_GEN_SSL_CERT: true
      username: <redacted>
      password: <redacted>
      dictionaries: en_US

d. My complete Caddy config:

nc.teaguemillette.com {
    log {
        level INFO
        output file "/data/access.log" {
            roll_size 10MB
            roll_keep 10
        }
    }

    redir /.well-known/carddav /remote.php/dav 301
    redir /.well-known/caldav /remote.php/dav 301

    encode gzip

    header {
        Strict-Transport-Security "max-age=31536000;" # force HTTPS
        X-Frame-Options "SAMEORIGIN" # disallow rendering in iframe
        X-Robots-Tag "none" # prevents search engine indexing
    }

    @forbidden {
        path /.htaccess
        path /data/*
        path /config/*
        path /db_structure
        path /.xml
        path /README
        path /3rdparty/*
        path /lib/*
        path /templates/*
        path /occ
        path /console.php
    }
    respond @forbidden 404

    root * /var/www/html
    php_fastcgi nextcloud:9000
    file_server
}

collabora.teaguemillette.com {
    log {
        level INFO
        output file "/data/access.log" {
            roll_size 10MB
            roll_keep 10
        }
    }

    encode gzip

    header {
        Strict-Transport-Security "max-age=31536000;" # force HTTPS
        X-Robots-Tag "none" # prevents search engine indexing
    }

    reverse_proxy collabora:9980
}

3. The problem I’m having:

I was trying to connect my Collabora server to my NextCloud. Both Collabora and NextCloud work fine on their own, but NextCloud cannot detect Collabora. In the NextCloud logs, it says that it was timing out on a curl (see 4). After some testing (see 5), I have determined that the problem is the following: My services behind a reverse proxy cannot curl back onto the same domain. Though, all parts of my domain are managed by Caddy so it might be that they cannot curl back onto the same Caddy instance.

4. Error messages and/or full log output:

NextCloud logs this error every 1-2 minutes.

GuzzleHttp\Exception\ConnectException: cURL error 28: Connection timed out after 45001 milliseconds (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://collabora.teaguemillette.com/hosting/capabilities

No unusual logs from Caddy.

5. What I already tried:

This occurs for the primary domain and all subdomains.

Connecting via browser has no issues.
curl teaguemillette.com from my own computer: success
curl teaguemillette.com from the server: success
docker compose exec nextcloud curl teaguemillette.com: does not connect
docker compose exec collabora curl teaguemillette.com: does not connect
docker compose exec caddy ping teaguemillette.com: success (Caddy doesn’t have curl, and the others don’t have ping so I can only test this much)

6. Links to relevant resources:

The UFW_MANAGED label is from GitHub - shinebayar-g/ufw-docker-automated: Manage docker containers firewall with UFW!

francislavoie · February 3, 2023, 1:20am

The problem is that TCP packets coming inside of a Docker container would need to leave, make it to your router, your router would need to realize the destination is back inside the same network, and route it back in. I’m not exactly sure where it’s going wrong, because it entirely depends on your network circumstances, but it’s not unusual that it doesn’t work.

What I suggest you do is add another address like http://caddy (which matches the service name of your Caddy container) to your teaguemillette.com site, and then configure Collabora to make its requests to http://caddy instead.

hampy · February 3, 2023, 1:30am

I could be misunderstanding you here, but you are saying that traffic from the server cannot be routed back onto itself? I was able to curl from the exact server that is running docker, so how is that different?

francislavoie · February 3, 2023, 1:36am

It might have to do with the “source” IP address on the request TCP packets not allowing the response to make its way back to the correct place.

Honestly, I don’t know for sure, as I already said. I can only make bad guesses

hairyhenderson · February 3, 2023, 2:22am

It’s quite different - your containers and your host OS are running in different network namespaces. The containers in the Docker Compose config are sharing a network namespace, and your links entries make these containers visible by those names, but your domain points to an external IP.

So like @francislavoie guessed, what’s probably going on is a situation known as “hairpin routing”, which takes a bit of finesse to get working correctly.

I think the right fix is going to be to configure nextcloud and/or collabora to connect to each other by their internal names, and not by the domain that points to the external IP.

hampy · February 3, 2023, 3:27am

This does indeed solve the problem of curling between containers. Unfortunately, it is not enough to make NextCloud and Collabora play nice with each other. I will have to continue my quest on the NextCloud forums I guess

hampy · February 8, 2023, 5:50am

After much trial and pain I have come up with a probably-cursed solution but it does work so that is something.

Move Caddy from the internal network to the host network
Expose the necessary ports from services on docker network on the local machine network but not publically (I used ufw-docker-automated to make sure the firewall worked properly).
Configure Caddyfile to use ports from localhost instead of Docker
Everything works now

My understanding of the reason behind this issue (which might be a bit flawed) is that Docker does not allow hairpin routing, but once Caddy is on the host network, hairpin routing works fine.

Docker Compose:

version: "2"

services:
  caddy:
    image: caddy:2.6.2
    restart: always
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - ./caddy-data:/data
      - ./public_html:/var/www
    network_mode: host
    depends_on:
      - nextcloud
    volumes_from:
      - nextcloud

  nextcloud:
    image: nextcloud:25.0.3-fpm
    restart: always
    ports:
      - 9000:9000
    volumes:
      - ./nextcloud:/var/www/html
    depends_on:
      - collabora

  collabora:
    image: collabora/code:22.05.10.1.1
    restart: always
    ports:
      - 9980:9980
    networks:
      - my_network
    environment:
      server_name: collabora.teaguemillette.com
      aliasgroup1: https://nc.teaguemillette.com|https://nc.teaguemillette.com
      extra_params: --o:ssl.enable=false --o:ssl.termination=true
      DONT_GEN_SSL_CERT: true
      username: <redacted>
      password: <redacted>
      dictionaries: en_US
    cap_add:
      - MKNOD

Caddyfile:

nc.teaguemillette.com {
    log {
        level INFO
        output file "/data/access.log" {
            roll_size 10MB
            roll_keep 10
        }
    }

    redir /.well-known/carddav /remote.php/dav 301
    redir /.well-known/caldav /remote.php/dav 301

    encode gzip

    header {
        Strict-Transport-Security "max-age=31536000;" # force HTTPS
        X-Frame-Options "SAMEORIGIN" # disallow rendering in iframe
        X-Robots-Tag "none" # prevents search engine indexing
    }

    @forbidden {
        path /.htaccess
        path /data/*
        path /config/*
        path /db_structure
        path /.xml
        path /README
        path /3rdparty/*
        path /lib/*
        path /templates/*
        path /occ
        path /console.php
    }
    respond @forbidden 404

    root * /var/www/html
    php_fastcgi localhost:9000
    file_server
}

collabora.teaguemillette.com {
    log {
        level INFO
        output file "/data/access.log" {
            roll_size 10MB
            roll_keep 10
        }
    }

    encode gzip

    header {
        Strict-Transport-Security "max-age=31536000;" # force HTTPS
        X-Robots-Tag "none" # prevents search engine indexing
    }

    reverse_proxy localhost:9980
}

system · March 10, 2023, 5:51am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.