Connection Refused With On Demand TLS via Docker

1. The problem I’m having:

For a little bit more context, I am in the process of setting up a SaaS project using docker, laravel, php-fpm and caddy. New customers (tenants) will receive a subdomain automatically (e.g tenant1.ciro.com). This appears to be working well for me locally as I can hit admin.ciro.localhost, ciro.localhost and tenant1.ciro.localhost.

Tenants also have the option of specifying a custom domain as well. For this test, I am trying to hit test.localhost. For this I have tried to setup on_demand_tls and have setup an endpoint to validate the domain that caddy is passing me. Below is how I validate this (it does appear to work fine, and doesn’t seem to be the cause of the probilem I am having).

<?php

namespace App\Http\Controllers;

use App\Store;
use Illuminate\Http\Request;

class CaddyController extends Controller
{
    public function check(Request $request)
    {
        $authorizedDomains = [
            'test.localhost'
        ];

        if (in_array($request->query('domain'), $authorizedDomains)) {
            return response('Domain Authorized');
        }

        // Abort if there's no 200 response returned above
        abort(503);
    }
}

The next part is where my issues start to arise.
When the caddy container runs the ask endpoint, I get a connection refused error. My guess is that I am not calling my dockerized laravel service, ciro, correctly from the caddy container.
I have setup a bridge network that the caddy and ciro containers are both configured to use, but for some reason I am unable to communicate from the caddy container to ciro.

I have also tried going directly into the caddy container and running curl against the endpoints with similar results:

ciro on  main [!] via  v16.17.1 via 🐘 v8.2.3 took 18m5s
❯ docker exec -ti caddy /bin/sh
/var/www/html # curl -i https://ciro:5555
curl: (7) Failed to connect to ciro port 5555 after 2 ms: Connection refused
/var/www/html # curl -i https://ciro:9000
curl: (7) Failed to connect to ciro port 9000 after 1 ms: Connection refused
/var/www/html # curl -i ciro:9000
curl: (7) Failed to connect to ciro port 9000 after 1 ms: Connection refused
/var/www/html # curl -i ciro:9000
curl: (7) Failed to connect to ciro port 9000 after 1 ms: Connection refused
/var/www/html # curl -i ciro
curl: (7) Failed to connect to ciro port 80 after 1 ms: Connection refused
/var/www/html # curl -i ciro:5173
curl: (7) Failed to connect to ciro port 5173 after 1 ms: Connection refused

Any help would be much appreciated! :slight_smile:

2. Error messages and/or full log output:

{
    "level": "error",
    "ts": 1678556751.8410308,
    "logger": "tls",
    "msg": "request to 'ask' endpoint failed",
    "error": "error checking https://ciro:5173/caddy-check to determine if certificate for hostname 'test.localhost' should be allowed: Get \"https://ciro:5173/caddy-check?domain=test.localhost\": dial tcp 192.168.192.6:5173: connect: connection refused",
    "endpoint": "https://ciro:5173/caddy-check",
    "domain": "test.localhost"
}

3. Caddy version:

2.6.4

4. How I installed and ran Caddy:

a. System environment:

Here is my dockerfile I use to build Caddy:

FROM caddy:2.6.4-builder AS builder
RUN xcaddy build \
  --with github.com/caddy-dns/cloudflare

FROM caddy:2.6.4
COPY --from=builder /usr/bin/caddy /usr/bin/caddy

b. Command:

Via offical caddy Docker Container via docker compose.

c. Service/unit/compose file:

Here is my docker-compose.yml that sets up my ciro laravel application and caddy… I have removed the other unecessary services like memcached and mysql for easier reading:

version: '3'
services:
    ciro:
        container_name: ciro
        build:
            context: ./docker/8.2
            dockerfile: Dockerfile
            args:
                WWWGROUP: '${WWWGROUP}'
        image: ciro-8.2/app
        extra_hosts:
            - 'host.docker.internal:host-gateway'
        ports:
            - '5173:5173'
        environment:
            WWWUSER: '${WWWUSER}'
            LARAVEL_SAIL: 1
            XDEBUG_MODE: '${SAIL_XDEBUG_MODE:-off}'
            XDEBUG_CONFIG: '${SAIL_XDEBUG_CONFIG:-client_host=host.docker.internal}'
        volumes:
            - '.:/var/www/html'
            - 'sail-unix-sockets:/var/run'
        networks:
            - sail
        depends_on:
            - mysql
            - memcached
            - mailpit
    caddy:
        container_name: caddy
        image: caddy-cloudflare:latest
        working_dir: /var/www/html
        command: 'caddy run'
        environment:
            APP_URL: '${APP_URL}'
            HTTP_SCHEME: '${HTTP_SCHEME}'
        ports:
            - '${APP_PORT:-80}:80'
            - "443:443"
        networks:
            - sail
        volumes:
            - './:/var/www/html'
            - 'sail-caddy-data:/data'
            - 'sail-caddy-config:/config'
            - 'sail-unix-sockets:/var/run'
networks:
    sail:
        driver: bridge
volumes:
    sail-mysql:
        driver: local
    sail-caddy-data:
    sail-caddy-config:
    sail-unix-sockets:

d. My complete Caddy config:

{
    on_demand_tls {
        # TODO
        # No such host .. something docker related I would imagine.
        ask https://ciro:5173/caddy-check
    }
    local_certs
}

*.{$APP_URL}, {$APP_URL} {
    # For local development use internal
    # https://caddyserver.com/docs/caddyfile/directives/tls#examples
    tls internal
    root * {$SITE_ROOT_PATH:./public}
    header {
        X-Frame-Options SAMEORIGIN
        x-Content-Type-Options nosniff
    }
    @cacheItems path_regexp \.(css|ico|js|woff2)*
    header @cacheItems Cache-Control "max-age=31536000, immutable"
    encode zstd gzip
    php_fastcgi {$PHP_FASTCGI_LISTEN:unix//var/run/php/php8.2-fpm.sock}
    file_server
}

:443 {
    tls {
        on_demand
    }
}

5. Links to relevant resources:

NA

I think I may have figured out a solution, but I am not sure if it is optimal. Instead of hitting the ciro service, I hit caddy because it uses PHP FPM. In order to make sure that these requests come through, I added it to the URL block .

{
    on_demand_tls {
        # Since we use PHP FPM, send this back to the caddy 
        ask https://caddy/caddy-check ## MODIFIED
    }
    local_certs
}
# ADDED Make sure we accept requests from on_demand_tls, ie. "caddy" 
*.{$APP_URL}, {$APP_URL}, caddy {
    # For local development use internal
    # https://caddyserver.com/docs/caddyfile/directives/tls#examples
    tls internal
    root * {$SITE_ROOT_PATH:./public}
    header {
        X-Frame-Options SAMEORIGIN
        x-Content-Type-Options nosniff
    }
    @cacheItems path_regexp \.(css|ico|js|woff2)*
    header @cacheItems Cache-Control "max-age=31536000, immutable"
    encode zstd gzip
    php_fastcgi {$PHP_FASTCGI_LISTEN:unix//var/run/php/php8.2-fpm.sock}
    file_server
}

:443 {
    tls {
        on_demand
    }
}

Sail is a tool for development, you shouldn’t use it for production. Sail uses PHP’s built-in HTTP server which is not production-ready.

Are you sure there’s actually an HTTP server listening on that port? What’s in your Dockerfile?

FYI, in_array is inefficient, because it makes O(n) comparisons to find a match. Make the domains the key in the array (value can be anything, true is fine), then use isset() instead for O(1) lookups.

Keep in mind you used https:// here. That means the server you’re connecting to needs to have TLS set up. It probably doesn’t, since Caddy is terminating TLS.

Yep, that approach is fine. Keep in mind though that caddy isn’t a valid public domain, so you’ll need to split out that site block on its own to apply tls internal to it in production. Or use http://caddy instead as your site address to avoid that problem. Even better if you use a different port like http://caddy:8080 so that it’s only accessible from within the container (because http://caddy would use port 80 which is publicly accessible).

@francislavoie thanks for all the insight, much appreciated!

Getting rid of Sail was not on my radar yet, but I decided to do it anyways and make use of laradock instead as it is production ready (and also works great for local development).

As for ports 5173, those were not in use at all, I think that was a default that the sail project came with for Vite or something. I removed this.

Great call on the performance improvement, I just kind of whipped that up quickly as a proof of concept of my idea, but I will definitely move to your suggestion of making the domains the key!

Here is my updated Caddyfile (note that I really only want caddy-check to be internal, so I made sure to return a 404 if it is every accessed directly.

{
    on_demand_tls {
        # We talk directly with the caddy service over port 8081 to validate that
        # the domain being requested is a domain we actually trust.
        ask http://caddy:8081/caddy-check
    }
    local_certs
}

# Internally exposed for running on demand TLS "ask" via /caddy-check
# In other words, this is used to make sure that not any random actor 
# that points to our server can get a certificate.
http://caddy:8081 {
    root * /var/www/public
    header {
        X-Frame-Options SAMEORIGIN
        x-Content-Type-Options nosniff
    }
    log {
        output stderr
    }
    encode gzip
    php_fastcgi php-fpm:9000
    file_server
}

# https://*.{$APP_URL} -- Subdomain wildcard
# https://{$APP_URL} -- domain
https://*.{$APP_URL}, https://{$APP_URL} {
    # This endpoint is for internal use only, 404 so the internet can't run it.
    respond /caddy-check 404

    # Internal TLS for local development
    tls internal

    root * /var/www/public
    header {
        X-Frame-Options SAMEORIGIN
        x-Content-Type-Options nosniff
    }
    log {
        output stderr
    }
    @cacheItems path_regexp \.(css|ico|js|woff2)*
    header @cacheItems Cache-Control "max-age=31536000, immutable"
    encode gzip
    php_fastcgi php-fpm:9000
    file_server
}

:443 {
    tls {
        on_demand
    }
}


1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.