On-demand Subdomains with HTTPS Failing

Help
1

1. The problem I’m having:

I am trying to make my multi-tenant app that requires ability to create subdomains with https on the fly - the main reason I’m using Caddy.

Right now, I have the main domain jatra.app running with Caddy; however, when I try to create a subdomain, I get the following error on browser:

# This page isn’t working

**community.jatra.app** is currently unable to handle this request.

HTTP ERROR 502

Note : I’ve added a CNAME record with my domain registrar as:

CNAME | * | jatra.app | 600
Not sure if this record is correct and required for serving subdomains with HTTPS.

2. Error messages and/or full log output:

{"level":"info","ts":1698040425.470538,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
{"level":"info","ts":1698040425.4708772,"logger":"admin.api","msg":"load complete"}
{"level":"info","ts":1698040425.4728577,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
{"level":"error","ts":1698040457.6875722,"logger":"http.log.error","msg":"dial tcp 127.0.0.1:9000: connect: connection refused","request":{"remote_ip":"49.36.34.137","remote_port":"61999","client_ip":"49.36.34.137","proto":"HTTP/2.0","method":"GET","host":"community.jatra.app","uri":"/","headers":{"Cache-Control":["no-cache"],"Sec-Ch-Ua":["\"Chromium\";v=\"118\", \"Google Chrome\";v=\"118\", \"Not=A?Brand\";v=\"99\""],"Sec-Ch-Ua-Platform":["\"macOS\""],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Dest":["document"],"Accept-Language":["en-US,en;q=0.9,hi;q=0.8,la;q=0.7,mr;q=0.6"],"Sec-Ch-Ua-Mobile":["?0"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Pragma":["no-cache"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"community.jatra.app"}},"duration":0.000406151,"status":502,"err_id":"azr0wki3g","err_trace":"reverseproxy.statusError (reverseproxy.go:1265)"}
{"level":"error","ts":1698040458.9116673,"logger":"http.log.error","msg":"dial tcp 127.0.0.1:9000: connect: connection refused","request":{"remote_ip":"34.254.53.125","remote_port":"36508","client_ip":"34.254.53.125","proto":"HTTP/1.1","method":"GET","host":"help.jatra.app","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0"],"Accept-Encoding":["gzip, deflate"],"Accept":["*/*"],"Connection":["keep-alive"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"help.jatra.app"}},"duration":0.000374922,"status":502,"err_id":"a8jw1ifpm","err_trace":"reverseproxy.statusError (reverseproxy.go:1265)"}
{"level":"error","ts":1698040459.5463603,"logger":"http.log.error","msg":"dial tcp 127.0.0.1:9000: connect: connection refused","request":{"remote_ip":"34.254.53.125","remote_port":"50135","client_ip":"34.254.53.125","proto":"HTTP/1.1","method":"GET","host":"help.jatra.app","uri":"/","headers":{"Connection":["keep-alive"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0"],"Accept-Encoding":["gzip, deflate"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"help.jatra.app"}},"duration":0.000375224,"status":502,"err_id":"19akk333e","err_trace":"reverseproxy.statusError (reverseproxy.go:1265)"}
{"level":"info","ts":1698040582.5707989,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"56458","headers":{"Accept-Encoding":["gzip"],"Content-Length":["1437"],"Content-Type":["application/json"],"Origin":["http://localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
{"level":"info","ts":1698040582.571156,"msg":"config is unchanged"}
{"level":"info","ts":1698040582.5711958,"logger":"admin.api","msg":"load complete"}
{"level":"error","ts":1698040605.8215187,"logger":"http.log.error","msg":"dial tcp 127.0.0.1:9000: connect: connection refused","request":{"remote_ip":"49.36.34.137","remote_port":"53019","client_ip":"49.36.34.137","proto":"HTTP/3.0","method":"GET","host":"community.jatra.app","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Dest":["document"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"macOS\""],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Site":["none"],"Accept-Language":["en-US,en;q=0.9,hi;q=0.8,la;q=0.7,mr;q=0.6"],"Sec-Ch-Ua":["\"Chromium\";v=\"118\", \"Google Chrome\";v=\"118\", \"Not=A?Brand\";v=\"99\""],"Sec-Fetch-User":["?1"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"community.jatra.app"}},"duration":0.000566143,"status":502,"err_id":"h74mkd3qy","err_trace":"reverseproxy.statusError (reverseproxy.go:1265)"}

3. Caddy version:

v2.7.5 h1:HoysvZkLcN2xJExEepaFHK92Qgs7xAiCFydN5x5Hs6Q=

4. How I installed and ran Caddy:

Simply followed the instructions here: Install — Caddy Documentation

a. System environment:

uBuntu 22.04

b. Command:

NA

c. Service/unit/compose file:

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

d. My complete Caddy config:

{
        on_demand_tls {
                ask https://jatra.app/caddy/ask
                interval 2m
                burst 5
        }
        log {
                output file /var/log/caddy/access.log {
                        roll_size 1gb
                        roll_keep 5
                        roll_keep_for 720h
                }
        }
}

https:// {
        tls {
                on_demand
        }
        reverse_proxy localhost:9000 # PHP-FPM listens on this port
}

# Sites block. 
jatra.app {
        root * /home/forge/jatra.app/public
        php_fastcgi unix//run/php/php8.2-fpm.sock
        file_server
}

5. Links to relevant resources:

2

Looks like Caddy isn’t able to connect to your upstream app at port 9000.

You shouldn’t use reverse_proxy with default settings (which uses the http transport) for PHP-FPM, you should use php_fastcgi, or reverse_proxy configured to use the fastcgi transport.

You used a unix socket elsewhere for php_fastcgi though, why are you trying to connect to PHP with a TCP socket? The comment in your config is confusing.

3

Hi @francislavoie - Sorry for the misleading comment. I had my old notes from about an year ago when I had a working configuration.

What I’m trying to do is setup a Laravel (PHP) application that allows anyone to create subdomains on my main domain jatra.app. I’ve no clue what reverse_proxy localhost:9000 does here. Do I need to add the php_fastcgi unix//run/php/php8.2-fpm.sock like I did below?

{
        on_demand_tls {
                ask https://jatra.app/caddy/ask
                interval 2m
                burst 5
        }
        log {
                output file /var/log/caddy/access.log {
                        roll_size 1gb
                        roll_keep 5
                        roll_keep_for 720h
                }
        }
}

https:// {
        tls {
                on_demand
        }
        php_fastcgi unix//run/php/php8.2-fpm.sock #Do I need this here?
}

# Sites block. 
jatra.app {
        root * /home/forge/jatra.app/public
        php_fastcgi unix//run/php/php8.2-fpm.sock
        file_server
}

Could you please let me know if the above configuration looks optimum for my use case?

4

This is how you’re serving your PHP app:

So it follows that your https:// site should also do the same thing, if you also want to serve your PHP app from any domain allowed by ask.

If all you want is subdomains for jatra.app, then you probably don’t need On-Demand TLS, and you can instead just get a wildcard certificate for *.jatra.app. But for that you’ll need a build of Caddy with the ACME DNS plugin for your DNS provider. See How to use DNS provider modules in Caddy 2

A wildcard cert is ideal because it would mean you only have two certificates (i.e. one for your apex domain, and one wildcard for every subdomain), whereas with On-Demand TLS you’d need a certificate for each subdomain you want to support, which could be infinite depending on what the subdomain is (username?)

On-Demand TLS is best used for domains you do not control, i.e. the domain of a customer of yours that they want to point to your server, to provide a “custom domains” feature for your product.

1 Like
5

@francislavoie - thank you for your support.

On-Demand TLS is best used for domains you do not control, i.e. the domain of a customer of yours that they want to point to your server, to provide a “custom domains” feature for your product.

My SaaS application allows customers to be able to setup their own community platform (similar to Discourse) on their subdomain: {subdomain}.jatra.app. I wish to allow my users to map their primary domains or sub-domains to the subdomain on my app; so that I can serve white-label solution under their domain name.

This is the reason I wish to go with On-Demand TLS.

I am wondering if Caddy can be configured to deliver my community application via /path so that when my client visits clientdomain.com/community, it loads my application from say {client}.jatra.app ? [this, preferably via the the same Caddyfile]?

Update: Something like:

# Wildcard matching for any domain
https://* {
    # Handle path-based reverse proxy
    handle /app/{subdir:.*} {
        uri strip_prefix /app
        reverse_proxy {http.request.uri.path.strip_prefix}/jatra.app
    }
}
6

Okay, in that case On-Demand TLS does make sense for you. But I still strongly recommend using a wildcard cert for *.jatra.app instead of On-Demand TLS, it’ll reduce the amount of certs you need to have maintained, and reduce pressure on ACME issuers.

It’s possible, but I recommend against it. Instead, have your customers point a subdomain of theirs like community.clientdomain.com to your server. Much simpler.

See this article:

You’ll need to make your backend aware of the subpath in its routing. You’d probably need to store a mapping of domain + subpath they use, and then ensure that your router and URL builder in your app is aware of the subpath so that URLs to assets in your shipped HTML has the path prefix (otherwise CSS, JS etc will not load correctly).

This won’t work. * only matches a single domain label (a label is the parts between dots in a domain). Remove the * to match the entire domain, like https:// only.

You’d need to use a named matcher. See the Request Matching docs which explain. You’d use the path_regexp matcher.

This doesn’t really make sense. You can’t use a path segment in a proxy upstream address. If you need to change the URL, you need to use the rewrite or uri directives.

Anyway like I said I strongly suggest you avoid going down that path, it’s not worth it. Subdomains are much simpler and cleaner, and require no extra logic to handle.

7

Thank you for a very detailed response @francislavoie.

I am not aware of the problems I may run into. I am assuming Caddy will manage all those certificates for me.

Is there a number we can put on the amount of certs; before we can call it large to manage and can be a problem? Say, if I am running 500 communities on my SaaS each with their own subdomain or a mapped domain; what are the problems I am likely to run into?

I’ve read this article by Matt before I jumped from NGINX to Caddy: Serving tens of thousands of domains over HTTPS with Caddy

8

Yeah Caddy can manage tons of certificates. It usually gets dicey around 50k+ certs because that’s the default in-memory cert cache size so cache eviction can start causing problems.

But still, why not optimize where you can up-front before you grow more? It’s not a lot of extra effort to set up and you’ll get the benefits of less memory usage (each decoded cert in memory adds up quick) and less processing from issuance and renewals (even though that’s pretty cheap).

Another minor downside of individual subdomain certificates is that the certificates will be known in certificates transparency logs, so the names of your customers (or their usernames or whatever) will be public information. A wildcard cert hides that away.

9

50K is a big limit; not sure if I’ll hit that. But I wish to give wildcard subdomains a try. Here’s my working Caddyfile -

{
    on_demand_tls {
        ask https://jatra.app/caddy/ask
        interval 2m
        burst 5
    }
}

https:// {
    tls {
        on_demand
    }

    root * /home/forge/jatra.app/public
    file_server
    php_fastcgi unix//run/php/php8.2-fpm.sock
}

Now considering that I need a control over the subdomains (by checking them against the /ask endpoint, could you please suggest how could I modify my Caddyfile to make use of Wildcard certificate while maintaining the desired functionality?

10 
{
    on_demand_tls {
        ask https://jatra.app/caddy/ask
        interval 2m
        burst 5
    }
}

&(php-routes) {
	root * /home/forge/jatra.app/public
	encode gzip
	php_fastcgi unix//run/php/php8.2-fpm.sock
	file_server
}

*.jatra.app, jatra.app {
	tls {
		dns <whatever-your-provider-is>
	}

	invoke php-routes
}

https:// {
	tls {
		on_demand
	}

	invoke php-routes
}

Uses named routes for a very slight efficiency gain (only one copy of the handlers in memory) and to deduplicate config.

11

Thank you, @francislavoie . I think I’ll keep my existing configuration and experiment the wildcard setup on a test server. I am however running into a different problem with the current setup; and I am unable to determine if it’s a DNS configuration issue or a Caddyfile configuration issue.

I am replicating the entire setup on my test-domain: layoff.wtf

My Caddyfile Configuration:

{
        on_demand_tls {
                ask http://layoff.wtf/caddy/ask
                interval 2m
                burst 5
        }
}

https:// {
        tls {
                on_demand
        }

        root * /home/forge/layoff.wtf/public
        file_server
        php_fastcgi unix//run/php/php8.2-fpm.sock
}

This setup works perfectly to serve:

  1. My main domain: layoff.wtf with HTTPS
  2. Any subdomain: <subdomain>.layoff.wtf with HTTPS

Problem

My customer has created following subdomain on my SaaS: waitlist.layoff.wtf
They want to serve it via their subdomain: support.waitlist.guru

Here’s how the DNS has been configured for support.waitlist.guru:

CNAME | support | waitlist.layoff.wtf. | 600 seconds

That way, I thought when the user types support.waitlist.guru, they will be served my SaaS application from waitlist.layoff.wtf. However, they are being served the homepage on layoff.wtf and not the appropriate subdomain. You can actually type those names in browser and check.

DNS configuration for my SaaS domain: layoff.wtf is as follows:-
A | @ | 13.233.62.52 | 600 seconds
CNAME | * | layoff.wtf. | 600 seconds

How do I ensure that my customers can create their subdomains and map to their domain using a simple CNAME configuration; which I have seen on multiple SaaS offerings?

12

It’s up to your PHP app to check the hostname of the request and do the appropriate thing. All Caddy sees is support.waitlist.guru, it doesn’t know about the CNAME at all (and it can’t know, because that’s resolved by DNS by the client making the request to get an IP address to connect to).

So you need some mapping in your backend to route the request based on hostname.

13

Thank you again. I think I need to modify the headers that are sent via request to Laravel.

14

I don’t think so. Caddy already passes through the Host header which is what you should be looking at. Laravel’s router can route by hostname. If the hostname is not exactly jatra.app then it should hit your tenanted routes.

15

My Laravel application can handle any {subdomain}.layoff.wtf. When I visit support.waitlitlist.guru, the header is as follows:

X-Original-Host: support.waitlist.guru .

If my laravel application can handle waitlist.layoff.wtf correctly, it should be able to handle the mapped domain.

This is bit confusing.

16

I think you misunderstand how a CNAME works.

It’s at the DNS layer, it does not affect HTTP requests or the Host header. All it does is say “the IP address that this domain should resolve to, is the value of this other domain’s A record”. CNAME records delegate the IP address to another domain’s A record. It’s basically a recursive DNS lookup.

Your application’s router must be configured to handle every possible domain, including your own domains and your customer’s domains. Not just your domain & its subdomains.

The reason you ask your customers to point a CNAME to your domain is that if you need to change your main domain’s IP address (A record) then you can do so without your customers needing to worry about it because their CNAME will stay the same.

2 Likes
17

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.