Caddy as TLS terminator in front of apache

Help
1

1. Caddy version (caddy version):

v2.4.3 h1:Y1FaV2N4WO3rBqxSYA8UZsZTQdN+PwcoOcAiZTM8C0I=

2. How I run Caddy:

We use caddy as the TLS terminator for shared hosting that forwards requests to apache via http backend. No TLS configuration exists in apache.

a. System environment:

CloudLinux 8.x, caddy as a systemd service.

b. Command:

Paste command here.

c. Service/unit/compose file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

{
        http_port 80
        https_port 443
        admin 127.0.0.1:8888
        log {
                output file /var/log/caddy/caddy.log {
                        roll_size 250MiB
                        roll_keep_for 15d
                }
                level INFO
        }
        on_demand_tls {
                #TODO: Re-enable when script is done
                #ask https://web23.swisscenter.com/tools/caddy_dns_check.php
                interval 2m
                burst 5
        }
}

(common) {
        bind 127.0.0.1 ::1 94.103.96.188 2a00:a500:0:96::188
        reverse_proxy 127.0.0.80:80
}

(manager) {
        bind 94.103.96.188 2a00:a500:0:96::188
        reverse_proxy 127.0.0.1:9000
}

import /etc/caddy/host.conf
import /etc/caddy/customers/*.conf

host.conf

web23.swisscenter.com {
  import common
}

manager.web23.swisscenter.com {
  @only_obs {
    not remote_ip 192.168.50.0/24
  }
  route @only_obs {
    respond "Access denied" 403 {
      close
    }
  }
  import manager
}

127.0.0.1, [::1], 94.103.96.188, [2a00:a500:0:96::188] {
  import common
  tls internal
}

example customers/*.conf

http://cybermind.ch, https://cybermind.ch, http://www.cybermind.ch, https://www.cybermind.ch, http://276668.web23.swisscenter.com, https://276668.web23.swisscenter.com {
        import common
        tls {
                on_demand
        }
}

3. The problem I’m having:

Everything works fine except when dealing with http->https redirects when they are defined by customers or CMS’es in for example .htaccess files.
It goes into infinite loops.

4. Error messages and/or full log output:

Nothing

5. What I already tried:

The reason is obvious. As caddy is talking to apache through http port the some of different variables to check if the client is connecting with https are reporting http.

I tried many ways to try to let know the scripts we are on https:

For example added this global config:

# Let apache know we're behind a SSL reverse proxy (caddy)
SetEnvIf X-Forwarded-Proto "https" HTTPS=on
<IfModule !mod_remoteip.c>
        LoadModule remoteip_module lib64/httpd/modules/mod_remoteip.so
</IfModule>
<IfModule mod_remoteip.c>
        RemoteIPHeader X-Forwarded-For
        RemoteIPTrustedProxy 127.0.0.1/24
</IfModule>

This helps a bit for some situations but not for every one.

The main problem I would say is the redirections in .htaccess

For example the common:

<IfModule mod_rewrite.c>
       RewriteEngine on
       RewriteCond %{HTTPS} !=on [NC]
       RewriteRule ^(.*)$ https://%{HTTP_HOST}/$1 [R=301,L]
</IfModule>

The rewriting of HTTPS doesn’t work in this case as it seems this variable can’t be overwritten at this point.
It is replaced when forwarded to PHP scripts, but not when checked in .htaccess.

I am looking for some advices about what is the best approach to have apache set all it’s variables like if we were truely connecting with https.

A workaround would be to setup some TLS configs in apache too and have caddy really talk https with apache, but that’s not the goal.

I know it’s more an apache-side issue, but maybe someone has good advice about this ?
I even looked if there was some kind of apache module that could rewrite these variables when knowing that the reverse proxy serves clients with TLS
 But found nothing (yet).

Thanks a lot for your good advice.

6. Links to relevant resources:

1 Like
2

Can’t you just remove those rules in Apache?

I don’t really have anything else to recommend, you’re essentially doing it the right way by checking for X-Forwarded-Proto: https.

Why are you defining all the sites with both http:// and https:// here? If you just use the domain names like cybermind.ch, Caddy will automatically set up the HTTP->HTTPS redirects for those domains. Try this instead:

cybermind.ch, www.cybermind.ch, 276668.web23.swisscenter.com {
	...
}

It doesn’t make sense to turn on on_demand here if you actually specified the domains as site addresses in your config. You should only use on_demand if you don’t actually know the domains you need to manage up-front, like for a SaaS that allows customers to point their own domains to your site for branding.

I see that your ask endpoint is WIP, but keep in mind that with on_demand on, you’re open to DDOS, because some attacker could force your Caddy instance to have domains issued for infinite domains, filling up your storage and making you hit rate limits (causing other legitimate domains to not get issued etc)

You can remove these lines, they’re redundant.

1 Like
3

Hello Francis,

Thanks for your reply.

Can’t you just remove those rules in Apache?

That’s mainly the issue. We have no control about what our customers or their CMS automatically puts in .htaccess.
That is why we are looking for away to have apache know at this point that HTTPS = on, but unfortunately SetEnvIf X-Forwarded-Proto "https" HTTPS=on doesn’t overwrite the value used in the RewriteCond 

Maybe there is a way to overwrite it that I’m not aware of.

Why are you defining all the sites with both http:// and https:// here? If you just use the domain names like cybermind.ch , Caddy will automatically set up the HTTP->HTTPS redirects for those domains. Try this instead:

We have two different kind of hosts definitions. This one using http and https prefix are for customers that doesn’t want automatic http->https redirection.

For the other ones that wants automatic redirection, we do it like you suggested :slight_smile:

I see that your ask endpoint is WIP, but keep in mind that with on_demand on, you’re open to DDOS, because some attacker could force your Caddy instance to have domains issued for infinite domains, filling up your storage and making you hit rate limits (causing other legitimate domains to not get issued etc)

Yes, this is WIP, our script is not ready so for early tests it’s just commented out. It will be enabled befoire going production.

4

If the special variable from mod_rewrite can’t be overwritten you’re probably out of luck. Even if it’s possible and your customers don’t need to change lines like in your example some customer code might need more patches.

For example WordPress: Take a look at the is_ssl() implementation. If the HTTPS == "on" check won’t work the fallback is to check if the request came to port :443. You can’t fake that without follow up problems. In their solution they set the variable within PHP manually early on based on the X-Forwarded-Proto.

Is it really a good idea to hide from the customers the fact that SSL is terminated somewhere downstream?

1 Like
5

Hello Bernd,

For example WordPress: Take a look at the is_ssl() implementation. If the HTTPS == "on" check won’t work the fallback is to check if the request came to port :443. You can’t fake that without follow up problems. In their solution they set the variable within PHP manually early on based on the X-Forwarded-Proto.

Yes if I remember correctly, there is some kind of “fight” in the wordpress community about WordPress also checking X-Forwarded-Proto but they don’t seem to want to add this to the core.
At this point you could also fake $_SERVER['SERVER_PORT'] if it was possible.

Is it really a good idea to hide from the customers the fact that SSL is terminated somewhere downstream?

I agree it might not be the best idea to hide this. Actually we do not want to hide it, it’s more a practical solution, as we are going to migrate thousands of domains from old servers to new servers that will
use caddy as TLS terminator and apache as backend (through http).

We would need then to handle case by case all site migration and assist the customers so they modify their site content that make use of these variables.

This would be a nightmare and time consumming.

6

After reading a bit of the source code and changelog, it looks apache has now a hook that triggers when it’s checking if the connection is SSL.

Maybe I could attempt to write a module that somehow returns true when it detects that X-Forwarded-Proto is https and that the forwarder is authorized.

From 26-May-2021 Changes with Apache 2.4.48

The following has been added so far:
- ap_ssl_conn_is_ssl() to query if a connection is using SSL.
- ap_ssl_var_lookup() to query SSL related variables for a
server/connection/request.
- Hooks for ‘ssl_conn_is_ssl’ and ‘ssl_var_lookup’ where modules
providing SSL can install their own value supplying functions.

7

I think I found something very interresting for this topic:

It looks like it is what I’m trying to do.

1 Like
8

You’re right, this sounds exactly like a match for your needs:

RemoteProtoHeader / RemoteHTTPSEnableProto is a widely used setup supplement. It allows to perform all SSL on frontend balancers, while running backend over HTTP only in a trusted network, removing SSL overhead from backends. This allows to create a header to inform backend scripts and other dynamic stuff the original request came via SSL to make it generate HTTPS URLs while the backend will always serve unencrypted. This also affects rewrite conditions so i.e. SSL=on/off dependent rules may be used. This feature removes the need of tricky SSL/non-SSL state handling on backends in such setups, faking SSL state and allowing third party stuff to run unmodified in SSL proxy setups.

1 Like
9

Yes, I was able to recompile mod_remoteip with the patch and it works as intended.

However I have a strange issue left.

With the caddy config:


{
        admin 127.0.0.1:8888
        http_port 80
        https_port 443
        log {
                output file /var/log/caddy/caddy.log {
                        roll_size 250MiB
                        roll_keep_for 15d
                }
                level INFO
        }
        on_demand_tls {
                #TODO: Re-enable when script is done
                #ask https://web24.swisscenter.com/tools/caddy_dns_check.php
                interval 2m
                burst 5
        }
}

(theheaders) {
          header_up Host {host}
          header_up X-Real-IP {remote}
          header_up X-Forwarded-For {remote}
          header_up X-Forwarded-Port {server_port}
          header_up X-Forwarded-Proto {scheme}
}

(common) {
        bind 127.0.0.1 ::1 94.103.96.197 2a00:a500:0:96::197
        reverse_proxy h2c://127.0.0.80:80 {
                import theheaders
        }
}

(manager) {
        bind 94.103.96.197 2a00:a500:0:96::197
        reverse_proxy http://127.0.0.1:9000 {
                import theheaders
        }
}

import /etc/caddy/host.conf
import /etc/caddy/customers/*.conf

Every added headers are populated correctly except
header_up X-Forwarded-Port {server_port}

It returns an empty string.

If I edit the config and set it to

header_up X-Forwarded-Port 443

It returns the value.
It is like if the value of {server_port} is empty at this point.

Leaving it statically at 443 would not be an option as caddy both serves either http or https (when ssl redirection is not forced).

I googled around about it and find a lot of people using {server_port} at this point so I guess it should work ?

EDIT: I already tried to use http:// instead of h2c:// but it makes no difference

10

Really, all of these should be unnecessary. See the docs:

Caddy takes the port from the request URL. If the request was HTTPS on port 443, then browsers don’t need to specify the port because it’s the default port. Caddy doesn’t fill it in either, because it’s rarely useful when the scheme is already known (if the port is empty and scheme is http, then the port is 80, if scheme is https then port is 443).

All those examples are wrong. It’s honestly incredibly frustrating that those header_up snippets keep going around. We always have to tell people to stop using them.

You should only use h2c:// if you specifically have a reason to (i.e. running a GRPC app that requires it). Otherwise, use http:// (or omit the scheme, because http:// is the default anyways)

11

This topic was automatically closed after 30 days. New replies are no longer allowed.