Reverse-proxy, upgrade to https, custom/non-root ports

[jonward (Jonathan Ward)]

1. Caddy version (caddy version):

v2.3.0

2. How I run Caddy:

As a non-root reverse-proxy

a. System environment:

Archlinux

b. Command:

cd /path/to/Caddyfile
caddy start

c. Service/unit/compose file:

-

d. My complete Caddyfile or JSON config:

{
    http_port 8081
    https_port 8082
}

:8081, :8082 {
    reverse_proxy localhost:8080
}

3. The problem I’m having:

I have a code-server program listening to http://localhost:8080 requests.
I’m trying to run a non-root reverse proxy on custom http(s) ports as shown in the Caddyfile to relay requests to this server through https. The problem is http/8081 works, but https/8082 doesn’t, even though it is not a root port.

Non-root caddy’s certificate is already trusted. Https works as desired when I run the following command instead. I have to insert a https://localhost though for it to work.

sudo caddy reverse-proxy --from https://localhost:8082 --to http://localhost:8080

4. Error messages and/or full log output:

No errors.
Log -

$ caddy start
2021/05/26 14:29:19.470	INFO	using adjacent Caddyfile
2021/05/26 14:29:19.472	INFO	admin	admin endpoint started	{"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2021/05/26 14:29:19.472	INFO	http	server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server	{"server_name": "srv0", "http_port": 8081}
2021/05/26 14:29:19.472	INFO	http	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv1", "https_port": 8082}
2021/05/26 14:29:19.472	INFO	http	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv1"}
2021/05/26 14:29:19.472	WARN	http	user server is listening on same interface as automatic HTTP->HTTPS redirects; user-configured routes might override these redirects	{"server_name": "srv0", "interface": "tcp/:8081"}
2021/05/26 14:29:19.472	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0xc00035f2d0"}
2021/05/26 14:29:19.472	INFO	tls	cleaned up storage units
2021/05/26 14:29:19.472	INFO	autosaved config	{"file": "/home/user/.local/share/caddy/autosave.json"}
2021/05/26 14:29:19.472	INFO	serving initial configuration
Successfully started Caddy (pid=76436) - Caddy is running in the background

5. What I already tried:

Running it as superuser with command line options works (as shown above), but that is far from what I need.
I’ve followed the following resources, no fruit.

6. Links to relevant resources:

Is there a way to make all https connections happen on a high numbered port?

I’m trying to get caddy to work in a userspace docker container (I have other containers which I’d prefer to keep well isolated), so I can’t set_cap, as docker build is run as a userspace user, and doesn’t have permissions for that. So I start caddy like this in Docker: /usr/local/bin/caddy -agree -email email@example.com -ca https://acme-staging-v02.api.letsencrypt.org/directory -conf /etc/Caddyfile -http-port 8080 -https-port 4433 And Docker maps port 8080 to 80, and 4433 to 443. The thing…

No :443 in my config

Port and certificate errors with https file-server on localhost

1. Caddy version (caddy version): v2.0.0 installed via sudo apt install caddy 2. How I run Caddy: a. System environment: Ubuntu 20.04 desktop amd64 b. Command: caddy file-server --domain localhost and caddy file-server --domain localhost --listen :8082 c. Service/unit/compose file: N/A d. My complete Caddyfile or JSON config: N/A using CLI 3. The problem I’m having: I am failing to get caddy to run because of port binding permissions (initially for 80/443) so I tried using a h…

Not a reverse proxy

[francislavoie]

You didn’t tell Caddy to manage any certificates, so the TLS handshake cannot be completed.

When you ran it with localhost, Caddy issued a certificate with the name localhost, using its internal CA, which is automatically enabled if Caddy sees the domain to look like something that can’t be publicly trusted. Read the docs here:

Automatic HTTPS - Caddy Documentation

Caddy can issue certificates on demand for connections without a domain (i.e. an IP address), but you need to enable the on_demand option.

You’d be better off running Caddy as a systemd service, which will run as the caddy user typically (non-root) but with the CAP_NET_BIND_SERVICE capability enabled (which allows binding to low ports for non-root users). If you installed Caddy using pacman then you should have the systemd service already wired up:

Install - Caddy Documentation

If not then you can set up the service yourself:

Install - Caddy Documentation

[jonward (Jonathan Ward)]

Thank you.
Just read up on tls and on_demand_tls directives.
So is this config good?

{
    http_port 8081
    https_port 8082
    on_demand_tls {
        interval 5s
        burst 1
    }
}

:8081, :8082 {
    tls {
        on_demand
    }
    reverse_proxy localhost:8080
}

Also, does this require a new caddy trust for every new website?

Edit -
On entering https://localhost:8082 in the browser, this is what I’m getting.

I added the interval and the burst directives because of this output (too many "obtaining new certificate" messages). I’m not sure why this is happening.

[matt (Matt Holt)]

@jonward

So is this config good?

That allows 1 new certificate every 5 seconds. If that’s suitable for you, then sure. Only you will really be able to decide what limits are safe.

@jonward

Also, does this require a new caddy trust for every new website?

Nope, just a single root is trusted and that root lasts for 10 years.

@jonward

On entering https://localhost:8082 in the browser, this is what I’m getting.

It’s possible the browser is retrying every second. Try using curl -v instead. Are those the only log messages? That’s a bit unusual.

[jonward (Jonathan Ward)]

@matt

It’s possible the browser is retrying every second. Try using curl -v instead. Are those the only log messages? That’s a bit unusual.

You’re right. It’s four attempts per request, twice a second on average.
It’s not the only output though.

Here’s the complete log after one curl request (I put the on_demand_tls directive to avoid duplicate certificate requests).

2021/05/27 05:31:10.477	INFO	using adjacent Caddyfile
2021/05/27 05:31:10.479	INFO	admin	admin endpoint started	{"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2021/05/27 05:31:10.479	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0xc0003bca10"}
2021/05/27 05:31:10.486	INFO	http	server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server	{"server_name": "srv0", "http_port": 8081}
2021/05/27 05:31:10.486	INFO	http	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv1", "https_port": 8082}
2021/05/27 05:31:10.486	INFO	http	enabling automatic HTTP->HTTPS redirects	"server_name": "srv1"}
2021/05/27 05:31:10.486	WARN	http	user server is listening on same interface as automatic HTTP->HTTPS redirects; user-configured routes might override these redirects	"server_name": "srv0", "interface": "tcp/:8081"}
2021/05/27 05:31:10.486	INFO	tls	cleaned up storage units
2021/05/27 05:31:10.486	INFO	autosaved config	{"file": "/home/user/.local/share/caddy/autosave.json"}
2021/05/27 05:31:10.486	INFO	serving initial configuration
2021/05/27 06:18:46.591	INFO	tls.on_demand	obtaining new certificate	{"server_name": "192.168.1.42"}

And here’s the curl output.

$ curl -v "http://192.168.1.42:8081"
*   Trying 192.168.1.42:8081...
* Connected to 192.168.1.42 (192.168.1.42) port 8081 (#0)
> GET / HTTP/1.1
> Host: 192.168.1.42:8081
> User-Agent: curl/7.76.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Content-Length: 29
< Content-Type: text/plain; charset=utf-8
< Date: Wed, 26 May 2021 20:37:36 GMT
< Location: ./login
< Server: Caddy
< Vary: Accept, Accept-Encoding
<
* Connection #0 to host 192.168.1.42 left intact
Found. Redirecting to ./login
                                                      ~
$ curl -v "http://192.168.1.42:8082"
*   Trying 192.168.1.42:8082...
* Connected to 192.168.1.42 (192.168.1.42) port 8082 (#0)
> GET / HTTP/1.1
> Host: 192.168.1.42:8082
> User-Agent: curl/7.76.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 400 Bad Request
<
Client sent an HTTP request to an HTTPS server.
* Closing connection 0

$ curl -v "https://192.168.1.42:8082"
*   Trying 192.168.1.42:8082...
* Connected to 192.168.1.42 (192.168.1.42) port 8082 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, internal error (592):
* error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
* Closing connection 0
curl: (35) error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error

Also, I need 8081 to stay on http and 8082 to upgrade to https (if the request is http). Does this config do that?

[francislavoie]

@jonward

Also, I need 8081 to stay on http and 8082 to upgrade to https (if the request is http). Does this config do that?

That’s not possible. You need to request HTTP over the HTTP port, and HTTPS over the HTTPS port.

[jonward (Jonathan Ward)]

Update -
Apparently for the user instance, the data folder ~/.local/share/caddy was owned by root. Must have been when I ran caddy with superuser privileges without changing XDG_DATA_HOME. So instead of taking /var/lib/caddy, it took ~/.local/share/caddy and so the certificates could not be accessed.

I’ve since changed the owner back to the non-root user. Here is the Caddyfile I’m using.

:8081 {
    tls {
        on_demand
    }
    reverse_proxy localhost:8080
}

curl output -

$ curl -v "http://localhost:8081"
*   Trying 127.0.0.1:8081...
* Connected to localhost (127.0.0.1) port 8081 (#0)
> GET / HTTP/1.1
> Host: localhost:8081
> User-Agent: curl/7.76.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Content-Length: 29
< Content-Type: text/plain; charset=utf-8
< Date: Thu, 27 May 2021 16:39:31 GMT
< Location: ./login
< Server: Caddy
< Vary: Accept, Accept-Encoding
<
* Connection #0 to host localhost left intact
Found. Redirecting to ./login
$ curl -v "https://localhost:8081"
*   Trying 127.0.0.1:8081...
* Connected to localhost (127.0.0.1) port 8081 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

And caddy’s log -

$ caddy run
2021/05/27 16:39:16.828	INFO	using adjacent Caddyfile
2021/05/27 16:39:16.830	INFO	admin	admin endpoint started	{"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["127.0.0.1:2019", "localhost:2019", "[::1]:2019"]}
2021/05/27 16:39:16.830	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0xc000352e70"}
2021/05/27 16:39:16.831	INFO	autosaved config	{"file": "/home/user/.local/share/caddy/autosave.json"}
2021/05/27 16:39:16.831	INFO	serving initial configuration
2021/05/27 16:39:16.831	INFO	tls	cleaned up storage units

Isn’t on-demand tls supposed to redirect http to https here?

[francislavoie]

@jonward

Isn’t on-demand tls supposed to redirect http to https here?

Yes, if a request comes on the HTTP port, it will respond with a redirect to the HTTPS port. An HTTP request to the HTTPS port is a bad request (wrong type of data on the wire).

A redirect is an HTTP response which contains the Location: header telling the client to try again with a different URL.

An HTTPS server cannot respond to HTTP requests.

This is all simplified if you use ports 80 and 443 because the browser assumes those port numbers automatically when you type http:// and https:// in the browser.

Like I said, if you run Caddy as a service, you can run Caddy as a non-root user, with permission to bind to ports 80 and 443.

[jonward (Jonathan Ward)]

I do run it as a service for the actual server, I’m just seeing what all can be done as a non-root user.

So one last thing.
With on-demand tls, when a https request is sent, I’m getting an error in the log.

2021/05/27 18:26:57.599	INFO	using adjacent Caddyfile
2021/05/27 18:26:57.601	INFO	admin	admin endpoint started	{"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["[::1]:2019", "127.0.0.1:2019", "localhost:2019"]}
2021/05/27 18:26:57.601	INFO	http	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 8082}
2021/05/27 18:26:57.601	INFO	http	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2021/05/27 18:26:57.601	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0xc00034f3b0"}
2021/05/27 18:26:57.601	INFO	http	enabling automatic TLS certificate management	{"domains": ["localhost"]}
2021/05/27 18:26:57.601	INFO	autosaved config	{"file": "/home/user/.local/share/caddy/autosave.json"}
2021/05/27 18:26:57.601	INFO	serving initial configuration
2021/05/27 18:26:57.602	INFO	tls	cleaned up storage units
2021/05/27 18:26:57.608	INFO	tls.on_demand	obtaining new certificate	{"server_name": "localhost"}
2021/05/27 18:26:57.608	INFO	tls.obtain	acquiring lock	{"identifier": "localhost"}
2021/05/27 18:26:57.609	INFO	tls.obtain	lock acquired	{"identifier": "localhost"}
2021/05/27 18:26:57.616	ERROR	tls.obtain	will retry	{"error": "[localhost] Obtain: subject does not qualify for a public certificate: localhost", "attempt": 1, "retrying_in": 60, "elapsed": 0.007231348, "max_duration": 2592000}

And it doesn’t proceed from there.

[francislavoie]

The caddy user is non-root though.

[jonward (Jonathan Ward)]

Yeah, but to start/enable the service you have to be root eh?

Can you look at the above question too (edited reply)?
Error says localhost doesn’t qualify for a public certificate. So isn’t it supposed to get a local CA signed one instead? I’ve already made the system trust the root certificate, so I thought it was supposed to work.

[francislavoie]

@jonward

Yeah, but to start/enable the service you have to be root eh?

If you’re using systemd, then yes, systemd runs as root. But I don’t see the concern here.

@jonward

Error says localhost doesn’t qualify for a public certificate. So isn’t it supposed to get a local CA signed one instead? I’ve already made the system trust the certificate, so I thought it was supposed to work.

You need to also enable the internal issuer by specifying tls internal, otherwise it will always try to use the public issuers (Let’s Encrypt/ZeroSSL).

@jonward

:8081 {
    tls internal {
        on_demand
    }
    reverse_proxy localhost:8080
}

[jonward (Jonathan Ward)]

Yes. this is what I needed, thanks.

[jonward (Jonathan Ward)]

@francislavoie

If you’re using systemd, then yes, systemd runs as root. But I don’t see the concern here.

systemctl start or systemctl enable requires root access, doesn’t it?