On_demand_tls ask with UNIX socket?

Help
1

1. The problem I’m having:

Caddy documentation says on_demand_tls should be as fast and low latency as possible. The server software I’m running called Garage (self hosted S3) supports an admin API through UNIX sockets for on-demand TLS checks. And of course I want my S3 server to be as fast as possible.

I’d like to use on_demand_tls with the ask endpoint being pointed at a UNIX socket path, but the on-demand endpoint is at /check and I’m not sure how to make this work with Caddy. See the Caddy config section for what I tried.

2. Error messages and/or full log output:

2024/07/20 22:06:22.226	ERROR	tls	failed to get permission for on-demand certificate	{"domain": "sharkey.s3.garage.kennel.girlcock.ceo", "error": "checking unix//run/garage/admin.sock/check to determine if certificate for hostname 'sharkey.s3.garage.kennel.girlcock.ceo' should be allowed: Get \"unix//run/garage/admin.sock/check?domain=sharkey.s3.garage.kennel.girlcock.ceo\": unsupported protocol scheme \"\""}
2024/07/20 22:06:23.079	ERROR	tls	failed to get permission for on-demand certificate	{"domain": "sharkey.s3.garage.kennel.girlcock.ceo", "error": "checking unix//run/garage/admin.sock/check to determine if certificate for hostname 'sharkey.s3.garage.kennel.girlcock.ceo' should be allowed: Get \"unix//run/garage/admin.sock/check?domain=sharkey.s3.garage.kennel.girlcock.ceo\": unsupported protocol scheme \"\""}
2024/07/20 22:06:23.405	ERROR	tls	failed to get permission for on-demand certificate	{"domain": "sharkey.s3.garage.kennel.girlcock.ceo", "error": "checking unix//run/garage/admin.sock/check to determine if certificate for hostname 'sharkey.s3.garage.kennel.girlcock.ceo' should be allowed: Get \"unix//run/garage/admin.sock/check?domain=sharkey.s3.garage.kennel.girlcock.ceo\": unsupported protocol scheme \"\""}
2024/07/20 22:06:23.644	ERROR	tls	failed to get permission for on-demand certificate	{"domain": "sharkey.s3.garage.kennel.girlcock.ceo", "error": "checking unix//run/garage/admin.sock/check to determine if certificate for hostname 'sharkey.s3.garage.kennel.girlcock.ceo' should be allowed: Get \"unix//run/garage/admin.sock/check?domain=sharkey.s3.garage.kennel.girlcock.ceo\": unsupported protocol scheme \"\""}
2024/07/20 22:06:24.205	ERROR	tls	failed to get permission for on-demand certificate	{"domain": "attic.s3.garage.kennel.girlcock.ceo", "error": "checking unix//run/garage/admin.sock/check to determine if certificate for hostname 'attic.s3.garage.kennel.girlcock.ceo' should be allowed: Get \"unix//run/garage/admin.sock/check?domain=attic.s3.garage.kennel.girlcock.ceo\": unsupported protocol scheme \"\""}
2024/07/20 22:06:24.524	ERROR	tls	failed to get permission for on-demand certificate	{"domain": "attic.s3.garage.kennel.girlcock.ceo", "error": "checking unix//run/garage/admin.sock/check to determine if certificate for hostname 'attic.s3.garage.kennel.girlcock.ceo' should be allowed: Get \"unix//run/garage/admin.sock/check?domain=attic.s3.garage.kennel.girlcock.ceo\": unsupported protocol scheme \"\""}
2024/07/20 22:06:43.829	INFO	admin	admin endpoint started	{"address": "unix//run/caddy/admin.sock", "enforce_origin": false, "origins": ["", "//127.0.0.1", "//::1"]}
2024/07/20 22:06:43.962	INFO	admin	stopped previous server	{"address": "unix//run/caddy/admin.sock"}
2024/07/20 22:06:46.191	ERROR	tls	failed to get permission for on-demand certificate	{"domain": "attic.s3.garage.kennel.girlcock.ceo", "error": "checking unix//run/garage/admin.sock to determine if certificate for hostname 'attic.s3.garage.kennel.girlcock.ceo' should be allowed: Get \"unix//run/garage/admin.sock?domain=attic.s3.garage.kennel.girlcock.ceo\": unsupported protocol scheme \"\""}
2024/07/20 22:06:47.019	ERROR	tls	failed to get permission for on-demand certificate	{"domain": "attic.s3.garage.kennel.girlcock.ceo", "error": "checking unix//run/garage/admin.sock to determine if certificate for hostname 'attic.s3.garage.kennel.girlcock.ceo' should be allowed: Get \"unix//run/garage/admin.sock?domain=attic.s3.garage.kennel.girlcock.ceo\": unsupported protocol scheme \"\""}
2024/07/20 22:06:47.367	ERROR	tls	failed to get permission for on-demand certificate	{"domain": "attic.s3.garage.kennel.girlcock.ceo", "error": "checking unix//run/garage/admin.sock to determine if certificate for hostname 'attic.s3.garage.kennel.girlcock.ceo' should be allowed: Get \"unix//run/garage/admin.sock?domain=attic.s3.garage.kennel.girlcock.ceo\": unsupported protocol scheme \"\""}

3. Caddy version:

f8861ca16bd475e8519e7dbf5a2b55e81b329874+modified (28 Jun 24 18:15 UTC)

Self-compiled Caddy from main branch with 1 plugin (ListenCaddy)

4. How I installed and ran Caddy:

Compiled Caddy from main branch, transferred binary to my server, use my own systemd unit

a. System environment:

Arch Linux x86_64, systemd, self-compiled main branch of Caddy with 1 plugin

b. Command:

Don’t think this is necessary.

d. My complete Caddy config:

Don’t think this is necessary, will just share the relevant parts:

{
# ..
        on_demand_tls {
                ask unix//run/garage/admin.sock/check
                #ask unix//run/garage/admin.sock
                interval 2m
                burst 5
        }
# ..
}

# ..

s3.garage.kennel.girlcock.ceo, *.s3.garage.kennel.girlcock.ceo {
        import common

        tls {
                on_demand
        }

        reverse_proxy unix//run/garage/s3.sock
}
# ..

5. Links to relevant resources:

2

You could do something like this:

{
	on_demand_tls {
		ask http://localhost:5000/check
	}
}

:5000 {
	reverse_proxy unix//run/garage/admin.sock
}

Remove both burst and limit, they’re deprecated and will be removed in the future. They’re broken, it doesn’t work at all as expected.

1 Like
3

Thanks, I think if that’s the only workaround then I’ll just accept that I need to use TCP only for my on_demand_tls ask check, and consider this a feature request instead.

I’ll remove the deprecated options though, didn’t know they were!

4

TCP vs unix sockets aren’t going to be enough of a performance difference to matter here. The comment about speed of the endpoint is more about “don’t do slow IO or expensive validation checks on your endpoint, just do a fast map or DB lookup for the domain”, because this happens during the TLS handshake of the first request for that domain, and the handshake should stay fast enough.

1 Like
5

It’s more so about host security for me, rather than real world performance. I do understand the differences between TCP vs UNIX sockets. I just would prefer to confine things to their permissions as strict as possible, without diving into SELinux territory.

6

Fair enough. You could write a module which does this if you need, ask is now pluggable if you make a tls.permission.* module implementing the caddytls.OnDemandPermission interface. Lets you do anything you want.

It’s tricky to change ask to handle it because it takes an HTTP URL, but how do we encode that we want to connect to a unix socket with that kind of URL? It’s ambiguous.

We could add another permission module which does it, wouldn’t be difficult. If you want to make a PR to do that, it would be acceptable I think. Would be configured like this probably:

{
	on_demand_tls {
		permission unix_http /run/garage/admin.sock /check
	}
}

i.e. permission unix_http <socket-file> [<http-url>]

The default HTTP implementation is here caddy/modules/caddytls/ondemand.go at 806f5b111781a3b2734223c8e14780d86ac4edd5 · caddyserver/caddy · GitHub we’d just need to copy PermissionByHTTP and make a new one PermissionByUnixHTTP etc

2 Likes
7

Thanks for understanding my usecase! I don’t know Go, and I’m not desperate enough for this (I’ve just decided to listen the whole admin API of Garage on TCP in the mean time), but I may give it a shot if I’m bored today or tomorrow. Thanks for the details! I do agree that the encoding URL part might be tricky, but I like your example of just splitting it out into an argument like that.

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.