Exclude some location from automatic HTTP->HTTPS redirect

[r00tsh3ll (Sébastien Riccio)]

1. Caddy version (caddy version):

v2.5.1 h1:bAWwslD1jNeCzDa+jDCNwb8M3UJ2tPa8UZFFzPVmGKs=

2. How I run Caddy:

Caddy is run using systemd with /etc/caddy/Caddyfile as main config

a. System environment:

CloudLinux 8.x (CentOS/AlmaLinux based)

b. Command:

systemctl start caddy

c. Service/unit/compose file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE
ReadWritePaths=/etc/pki

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

Caddyfile

{
        admin 127.0.0.1:8888
        grace_period 3s
        log {
                output file /var/log/caddy/caddy.log {
                        roll_size 250MiB
                        roll_keep_for 15d
                }
                level INFO
        }
        email some@mail.com
        on_demand_tls {
                #ask https://api.swisscenter.com/webservices.php/caddy/dnslookup
                interval 2m
                burst 5
        }
        order realip first
}

(common) {
        bind 127.0.0.1 [::1] 94.103.96.188 [2a00:a500:0:96::188]
        realip {
                header "X-Forwarded-For"
                from cloudflare
                maxhops 5
        }
        @sc_server_fqdn {
                path /_sc_get_server_fqdn
        }
        respond @sc_server_fqdn "web23.swisscenter.com" 200 {
                close
        }
        reverse_proxy http://127.0.0.80:80
}

(manager) {
        bind 94.103.96.188 [2a00:a500:0:96::188]
        reverse_proxy http://127.0.0.1:9000
}

import /etc/caddy/host.conf
import /etc/caddy/customers/*.conf

host.conf

web23.swisscenter.com {
  @only_obs {
    path /imav*
    not remote_ip 192.168.50.0/24 2a00:a500:0:10::/64
  }
  respond @only_obs "We're sorry, but this resource is not available for you. If you feed this is an error, please contact your amazing server administrator." 403 {
    close
  }
  import common
}

manager.web23.swisscenter.com {
  @only_obs {
    not remote_ip 192.168.50.0/24
  }
  route @only_obs {
    respond "We're sorry, but this resource is not available for you. If you feed this is an error, please contact your amazing server administrator." 403 {
      close
    }
  }
  import manager
}

127.0.0.1, [::1], 94.103.96.188, [2a00:a500:0:96::188] {
  import common
  tls internal
}

Example customers/*.conf file

cybermind.ch, www.cybermind.ch, 276668.web23.swisscenter.com {
        import common
        tls {
                on_demand
        }
}

3. The problem I’m having:

We use caddy as a SSL terminator for our hosting. We use it with on-demand and the ASK hook to verify that the domain we want to request a certificate for is really pointing to the server to avoid any requests that would lead to an unsuccessfull result.

For thiis our ASK script call a special URL on the domain and need this url to return a 200 with the servername in the body that matches the servername of the host calling the ASK url.

However, when calling cybermind.ch/_sc_get_server_fqdn URL it is redirected to HTTPS, which makes sense thanks to the auto HTTP->HTTPS, but it is a problem, beacause the certificate is not yet available, so it ends up with an internal SSL error.

$ curl -v cybermind.ch/_sc_get_server_fqdn
*   Trying 2a00:a500:0:96::188:80...
* Connected to cybermind.ch (2a00:a500:0:96::188) port 80 (#0)
> GET /_sc_get_server_fqdn HTTP/1.1
> Host: cybermind.ch
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://cybermind.ch/_sc_get_server_fqdn
< Server: Caddy
< Date: Thu, 26 May 2022 06:18:31 GMT
< Content-Length: 0
< 
* Closing connection 0

We need this particular URL to be exempt from auto HTTP->HTTPS to avoid this issue so the ASK script can confirm the domain is pointing on the correct server before allowing a certificate to be requested.

4. Error messages and/or full log output:

The ASK script we wrote returns that cURL can’t connect to the redirected resource as the SSL engine is doing an internal error (certificate missing…)

caddy_dnslookup.log-20220524.gz:2022-05-23T07:42:58.042905+00:00 > 628b3af80d6f9 > ERROR > Error calling http://cybermind.ch/_sc_get_server_fqdn: cURL error 35: Peer reports it experienced an internal error. (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://cybermind.ch/_sc_get_server_fqdn

5. What I already tried:

To be honest I don’t know where to start to at this point, to exclude a single URL from the auto HTTP->HTTPS redirect.

6. Links to relevant resources:

[francislavoie]

I’m not sure how you’ve implemented this, but don’t do any HTTP requests during your ask lookup. You need it to be as fast as possible. It should typically just be a lookup in your database, and that’s it. If you want to do verification ahead of time, do it out-of-band (like in a cron or job queue) and record the result in your database so the ask lookup can be fast.

But either way, I think it’s okay that you get the HTTP->HTTPS redirect here. You can just check “was the status 308, and is the Server: Caddy header present, and does the Location redirect look as we expect”? If so then you’re probably fine, cause it means that the domain was pointed to Caddy. You could also just do a DNS lookup and check that the A or CNAME records are correct (whatever you tell your customers to do).

[r00tsh3ll (Sébastien Riccio)]

Hello Francis,

Thank you for your feedback.

Well, what we wants to make sure before allowing caddy to start a request for a certificate is that the domain is pointing on the expected server.
I had first coded the script to do this using DNS lookups and it worked well, until we had to deal with domains that use frontends like CloudFlare. This defeated the DNS lookup as it’s pointing on CloudFlare IP’s, so no real way to know if then the requests are forwarded to the correct server.

Using the HTTP call resolved this as when CloudFlare receive the request it forwards it to destination host and if the answer is correct, then it means it’s pointing at the correct place.

Of course you will tell me that in case of CloudFlare or other frontends, the SSL certificate should be handled by the CDN itself, but in the background CloudFlare is talking HTTPS with the target server, so it needs a certificate on the target server.

It also looks that CloudFlare automagically forward letsencrypt requests to the target server when they don’t match a request they’ve issued themselves. So they forward acme requests (/.well-known/acme-challenge stuff) which they did not initiated.

Having our target servers answer that special URL in http seemed to be a good way to be sure the cert request would be successfull and avoid triggering failures that could lead to a rate limiting.

Using a database is an option but the last check still would still be “is the domain currently pointing at the right place”.

Now it’s a great idea you have here to consider the HTTPS redirect (308 and checking the headers) as being successful while doing the check.
In a perfect world I would still need some header with the server ID to be exposed when the 308 redirect is done, so we know we are internally targeting the correct server in our server "farm.
The current headers don’t “leak” this info.
Could we add a customer header to every responses (including automatic redirect for SSL) such something as X-SC-Caddy-Server-FQDN ?

Then I guess I have all the infos to decide to accept or decline the ASK request

Kind regards,
Sébastien

PS: For the ASK script, it’s just some crappy PHP I wrote, if you want to take a look:

<?php
/*
 * caddy-dnslookup v2.0
 *
 * This webservices is a helper for Caddy "ASK" feature.
 * It will do some magic stuff in order to determine if caddy is allowed to initiate
 * an automated SSL certificate request for a given domain name.
 *
 * Author: sriccio@swisscenter.com / SwissCenter
 *
 */

use GuzzleHttp\Exception\GuzzleException;
use Monolog\Logger;
use Monolog\Handler\StreamHandler;
use Monolog\Formatter\LineFormatter;
use GuzzleHttp\Client;

$log_level = Logger::INFO;

$domain = null;
$caller_ip = null;
$caller_ip_type = null;

/*
 *  Functions
 */

/* Check if we are called from cli */
function is_cli(): bool
{
    if (empty($_SERVER['REMOTE_ADDR']) and !isset($_SERVER['HTTP_USER_AGENT']) and count($_SERVER['argv']) > 0) {
        return true;
    }
    return false;
}

function init_logger($log_level = Logger::INFO): Logger
{
    $uid = uniqid();
    $log_output = "%datetime% > %extra.uid% > %level_name% > %message%\n";
    $log_formatter = new LineFormatter($log_output);
    $log_stream = new StreamHandler(LOG_DIR . '/caddy_dnslookup.log', $log_level);
    $log_stream->setFormatter($log_formatter);
    $log = new Logger('caddy-dnslookup');
    $log->pushProcessor(function ($record) use($uid) {
        $record['extra']['uid'] = $uid;
        return $record;
    });
    $log->pushHandler($log_stream);
    return $log;
}

/* Return 403 header with message and exit */
function exit_403($msg = 'Access denied')
{
    header(sprintf('HTTP/1.1 403 %s', $msg));
    exit;
}

/* Return 500 header with message and exit */
function exit_500($msg = 'Error')
{
    header(sprintf('HTTP/1.1 500 %s', $msg));
    exit;
}

/* Return 200 header with message and exit */
function exit_200()
{
    header('HTTP/1.1 200');
    exit;
}

/*
 *  This is where the magic happens
 *  If everything matches we allow caddy to request the certificate
 */
function check($subject, $caller_hostname): bool
{
    global $log;

    // Call server fqdn lookup url
    // TODO: Check both using ipv4 and ipv6, but apanel-cp need support for ipv6 first.
    $client = new Client();
    $url = sprintf('http://%s/_sc_get_server_fqdn', $subject);
    try {
        $result = $client->request('GET', $url);
    } catch (GuzzleException | Exception $e) {
        $log->error(sprintf('Error calling %s: %s', $url, $e->getMessage()));
        return false;
    }
    // Request unsuccessful. Domain probably pointing on wrong server.
    if ($result->getStatusCode() <> '200') {
        $log->error(sprintf('Cannot identify caller IP address %s type', $_SERVER['REMOTE_ADDR']));
        return false;
    }

    if (trim($result->getBody()) != $caller_hostname) {
        return false;
    }

    /* all checks passed */
    return true;
}

/*
 * Main
 */
if (is_cli()) {
    echo "STOP! This tool must be run as a webservice.\n";
    exit(1);
}

global $log;
$log = init_logger($log_level);

/* Check if caller IP address is valid */
if (empty($_SERVER['REMOTE_ADDR'])) {
    $log->error('Caller IP address is unknown');
    exit_403('Could not lookup your IP address');
}
$caller_ip = $_SERVER['REMOTE_ADDR'];
if (filter_var($caller_ip, FILTER_VALIDATE_IP, FILTER_FLAG_IPV4)) {
    $caller_ip_type = 'ipv4';
} else if (filter_var($caller_ip, FILTER_VALIDATE_IP, FILTER_FLAG_IPV6)) {
    $caller_ip_type = 'ipv6';
} else {
    $log->error(sprintf('Cannot identify caller IP address %s type', $_SERVER['REMOTE_ADDR']));
    exit_403('Caller IP address is neither IPv4 or IPv6, huh ?');
}

/* Check if domain name has been specified */
if (!isset($_GET['domain'])) {
    $log->error('No domain specified in the query');
    exit_403('No domain to check specified');
}
$domain = trim($_GET['domain']);

/* Check if domain syntax is valid */
if (!filter_var($domain, FILTER_VALIDATE_DOMAIN, FILTER_FLAG_HOSTNAME)) {
    $log->error(sprintf('Invalid domain name syntax: %s', $domain));
    exit_403('Invalid hostname or domain name syntax');
}

/* Get caller hostname from caller IP */
$caller_hostname = gethostbyaddr($caller_ip);
if (!$caller_ip || $caller_hostname == $caller_ip) {
    $log->error(sprintf('Cannot resolve caller IP address %s to hostname', $caller_ip));
    exit_403('Cannot resolve caller IP address to hostname');
}


/* All fine until now. Let's do the main check */
$log->info(sprintf('Host %s wants to check domain %s', $caller_hostname, $domain));
if (check($domain, $caller_hostname)) {
    $log->info(sprintf('All checks passed for domain %s and caller %s', $domain, $caller_hostname));
    exit_200();
} else {
    $log->warning(sprintf('Domain %s is not pointing to %s. Denying request.', $domain, $caller_hostname));
    exit_403(sprintf('Denied. Domain %s is not pointing at yourself.', $domain));
}

/* If we reach this point, something is missing in the checking logic */
exit_500('Unknown error. This should never happen!');

[r00tsh3ll (Sébastien Riccio)]

So I’ve modded a bit the ASK script so it now treat a HTTP 308 + Server: Caddy as a hint that we’re targetting the correct server.

By handling this additional case it works quite well, but…

For domains passing through CloudFlare the solution is defeated, because CloudFlare rewrites the Server header with cloudflare value.

So I went ahead and tried to add our own custom header to all pages responses.
It seemed a good solution at first, but it looks like it is added in every reponses except in the 308 redirect replies, and this is where I need it of course :).

I guess the custom headers are not taken into account when caddy throws the 308 redirect for auto SSL redirect…

There is the “Server” header in the 308 response so I went ahead and tried to override it with a custom value.
It also works for everything except the 308 redirect where Server header stays the original “Caddy” value.

BTW, that would mean that someone willing to hide or change the Server name can’t do it for 308 auto-SSL redirects responses ?

[francislavoie]

@r00tsh3ll

Well, what we wants to make sure before allowing caddy to start a request for a certificate is that the domain is pointing on the expected server.

Well, if that’s the case, you don’t need to do anything. Because ask is only called if Caddy receives a request with SNI for a domain it doesn’t yet have a cert for. So if you know about the domain in your backend, you already know DNS was properly configured, cause Caddy would never have called your ask endpoint otherwise.