Reverse proxy without domain name

Okay I’m getting closer!!

I want to go host this so it’s accessible only from 192.168.1.200

so i can turn on a vpn and access it via: https://192.168.1.200:9999

currently i have:

>>>Caddyfile
https://192.168.1.200:9999

reverse_proxy bitwarden:8889
>>>docker-compose.yml
version: '3'
services:
  caddy:
    image: caddy:latest
    volumes:
      - /home/me/Documents/bitwarden/Caddyfile:/etc/caddy/Caddyfile
      - /home/me/Documents/bitwarden/caddy-data:/data
      - /home/me/Documents/bitwarden/caddy-config:/config
    ports:
      - 443:443
      - 9999:9999
      - 80:80

  bitwarden:
    image: bitwardenrs/server:latest
    volumes:
      - /home/me/Documents/bitwarden/bw-data/:/data
    environment:
      - ROCKET_PORT=8889
      - DOMAIN=https://192.168.1.200:9999
    restart: unless-stopped
    ports:
      - 8889:8889

This works fine with localhost but i can’t seem to access it when on another network via the vpn

the problem seems to be this:

{"level":"info","ts":1588964943.3766,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"info","ts":1588964943.4025936,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1588964943.404573,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
2020/05/08 19:09:03 [INFO][cache:0xc000444190] Started certificate maintenance routine
{"level":"info","ts":1588964943.4277575,"logger":"tls","msg":"setting internal issuer for automation policy that has only internal subjects but no issuer configured","subjects":["192.168.1.200"]}
{"level":"info","ts":1588964943.4287522,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["192.168.1.200"]}
2020/05/08 19:09:03 [WARNING] Stapling OCSP: no OCSP stapling for [192.168.1.200]: no OCSP server specified in certificate
{"level":"info","ts":1588964943.4309916,"logger":"tls","msg":"cleaned up storage units"}
{"level":"warn","ts":1588964943.5721502,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2020/05/08 19:09:03 Warning: "certutil" is not available, install "certutil" with "apt install libnss3-tools" or "yum install nss-tools" and try again
2020/05/08 19:09:03 define JAVA_HOME environment variable to use the Java trust
2020/05/08 19:09:03 certificate installed properly in linux trusts
{"level":"info","ts":1588964943.661539,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1588964943.6615646,"msg":"serving initial configuration"}
2020/05/08 19:10:10 http: TLS handshake error from 10.8.0.6:43674: EOF
2020/05/08 19:11:29 http: TLS handshake error from 10.8.0.6:43700: no certificate available for '172.19.0.3'

How can i get it to allow access from a non-localhost address?

I might suggest that if your Caddyfile starts with https://192.168.1.200:9999, but the server still responds to localhost, you may not actually be running the updated configuration.

Strictly speaking, localhost != https://192.168.1.200:9999, so a request to localhost should result in a 404 Site not served on this interface response edit: a status 200 with an empty body. Definitely shouldn’t work as normal.

Hello all,
Ok I sort of figured it out, but I’m not sure if this is a great way to do it…

>>>Caddyfile
https://192.168.1.200:9999 172.19.0.3

reverse_proxy bitwarden:8889
>>>docker-compose.yml
version: '3'
services:
  caddy:
    image: caddy:latest
    volumes:
      - /home/me/Documents/bitwarden/Caddyfile:/etc/caddy/Caddyfile
      - /home/me/Documents/bitwarden/caddy-data:/data
      - /home/me/Documents/bitwarden/caddy-config:/config
    ports:
      - 9999:9999
      # - 443:443
      # - 80:80

  bitwarden:
    image: bitwardenrs/server:latest
    volumes:
      - /home/me/Documents/bitwarden/bw-data/:/data
    environment:
      - ROCKET_PORT=8889
      - DOMAIN=https://192.168.1.200:9999
    restart: unless-stopped

I would like to know 4 things (if anyone could help me out)

  1. Is this a good way to do things?
  2. What am I doing by adding the 172.19.0.3 part to my caddyfile?
  3. Is there a way to wildcard this? I assume that 172.19.3 won’t always be the address that pops up here.
  4. The 172 IP address is from docker which is giving the containers (some research suggests its the userland-proxy which seems to give a lot of people headaches)

There’s no right way to do stuff like this, only a few wrong ways. In the sense that there’s nothing wrong with this config, you could say this is a good way to go about it.

Good question?

Kinda looks like a Docker network internal IP, so I suppose it would allow Caddy to server requests coming from other Docker containers directly requesting the Caddy container’s IP.

You can’t guarantee the container will always have that IP address, Docker Network IPs are dynamically allocated.

Question is, why do other Docker containers need to be accommodated? Surely they don’t need to be served anything from Caddy, Caddy is there so you can access the other services through Docker. Unless it breaks something, I’d probably just not bother.

Yep, you could remove the hostname and Caddy would respond using this site config regardless of what clients request. But that seems like it would clash with your earlier-stated objective:

That’s one possibility. The Caddy image will then be running outside of the Docker network, directly on the host’s network stack.

One side effect is that reverse_proxy bitwarden:8889 will stop working (as the hostname bitwarden won’t resolve any more, since it’s no longer inside the Docker network). In fact, with no ports exposed on your Bitwarden service, it will become inaccessible until you expose a port, and Caddy would have to refer to it via localhost on the exposed port in question.

1 Like

thank you whitestrake for your super fast answer!

what do you mean by:

Question is, why do other Docker containers need to be accommodated? Surely they don’t need to be served anything from Caddy, Caddy is there so you can access the other services through Docker. Unless it breaks something, I’d probably just not bother.

If your question is: why do I need this? then my answer would be: because without it I get the error:

2020/05/08 19:11:29 http: TLS handshake error from 10.8.0.6:43700: no certificate available for '172.19.0.3'

I don’t have enough networking knowledge to really know what that means unfortunately, but i’m guessing caddy somehow needs to be told what ip addresses should have access to that proxy endpoint.

I guess the salient question here is: is this an error you need to deal with?

For example, I get this error in spades from my Caddy instances that face the internet at large.

Bots crawl the internet all the time, the entire IPv4 address space is pretty much indexed, and I’m constantly hit with these errors in my logs. They mean nothing to me - they only serve to let me know that a bot tried to connect to my HTTP server via the IP address and wasn’t served anything because they didn’t actually request one of the sites I actually serve.

By way of explanation, the error is Caddy saying:

http: TLS handshake error “I couldn’t establish a secure connection” from 10.8.0.6:43700 “with the client connecting from 10.8.0.6:43700” no certificate available “because I didn’t have any HTTPS certificate to transmit to them” for '172.19.0.3' “because they requested the hostname 172.19.0.3 (implying I’m not configured to serve that over HTTPS)”.

So - question is, why do you need to resolve this error? It clearly comes from one of the Docker containers trying to connect directly to Caddy’s container via its IP address. Is that something you need them to be able to do? If you don’t know, and nothing’s broken, the answer is almost certainly no.

1 Like

Again – thank you for the detailed explanation!

why do you need to resolve this error?

When I get this error, I cannot access my service. (https bitwarden in this case, but even a https hello-world works similarly)

If I do not have 172.19.0.3 added to the Caddyfile line, I cannot access that service that caddy is routing. chrome yields: ERR_SSL_PROTOCOL_ERROR

So if I am required to know the docker network address name, so that i can add it to my caddyfile and make my services visible, then it would appear I cannot use caddy since this (as you mentioned) is dynamically set – is that the case?

Also: the logfile lines i’m mentioned are not from some bot crawling the internet. This is just on a local server that isn’t published to the greater web. They happen once every time I try to access the page and get the SSL_PROTOCOL_ERROR that i’m describing, so they appear to be directly related to what I’m doing.

When the 172.19.03 is added to the caddyfile I get: ERR_CERT_AUTHORITY_INVALID from chrome – but that’s expected and can access the page – but instead get:

2020/05/12 03:14:27 http: TLS handshake error from 192.168.1.16:54478: remote error: tls: unknown certificate

but the page seems to work fine, so i’m ok ignoring that.

Okay, whoa, something is weird and I must have missed something big. But I only see two services in your Docker Compose project. Are there other services in the way here? Any other proxy layers etc?

Are you browsing directly to 172.19.0.3? Like, in your browser URL bar are you going straight to https://172.19.0.3/ ?

Specifically - that error happens when some client connects to a Caddy HTTPS listener and indicates, via SNI, that it specifically wants the web page at 172.19.0.3.

Like, if you put the literal URL https://foobarhostname in your browser and that somehow resolved to 172.19.0.3, you wouldn’t get this error, you’d get an error saying that it didn’t have a certificate for foobarhostname.

The thing that’s even weirder to me - 172.19.0.3 shouldn’t even be routable for any host I can imagine you’d be making requests from. It’s a Docker Network IP address. Your LAN has no conception of a route to 172.19 and would probably forward the request out to WAN if you made it from, say, a laptop. Or even from the Docker host itself, as far as I know.

This might make sense if you had some other proxy layer, and that proxy layer was intercepting requests for whatever-address-you’re-actually-typing and then translating to connect directly to 172.19.0.3 over standard HTTPS. But that, in and of itself, doesn’t make much sense to me.

With only caddy and bitwarden services running in your Docker Compose project, the only two hosts in play here that could feasibly be making requests to Caddy and asking for the website 172.19.0.3 are in fact Caddy itself, and Bitwarden. Why would Caddy need to serve a webpage to itself? Why would Bitwarden need Caddy to serve it a webpage?

That’s it – the described docker-compose.yml file + caddyfile is all i have running. no other docker containers are running. I’m not excluding anything from my files as I’ve posted them and will post another example below.

Are you browsing directly to 172.19.0.3?

I’m not going to 172.19.0.3, In my address bar I’m typing:

https://192.168.1.200:9999

With the following caddyfile + d-c.yml

>>>docker-compose.yml
version: '3'
services:
  caddy:
    image: caddy:latest
    volumes:
      - /home/me/Documents/bitwarden/Caddyfile:/etc/caddy/Caddyfile
      - /home/me/Documents/bitwarden/caddy-data:/data
      - /home/me/Documents/bitwarden/caddy-config:/config
    ports:
      - 9999:9999

  crcc:
    image: crccheck/hello-world
    restart: unless-stopped

<<<
>>> Caddyfile
https://192.168.1.200:9999 {
        reverse_proxy crcc:8000
}
<<<

If I access the website 3 times from chrome at https://192.168.1.200:9999/ I see the error: ERR_SSL_PROTOCOL_ERROR and cannot access the service each time and I see the logs:

{"level":"info","ts":1589255900.1151738,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"info","ts":1589255900.1204681,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1589255900.1206877,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
2020/05/12 03:58:20 [INFO][cache:0xc0007a0500] Started certificate maintenance routine
{"level":"info","ts":1589255900.1297123,"logger":"tls","msg":"setting internal issuer for automation policy that has only internal subjects but no issuer configured","subjects":["192.168.1.200"]}
{"level":"warn","ts":1589255900.1674604,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2020/05/12 03:58:20 Warning: "certutil" is not available, install "certutil" with "apt install libnss3-tools" or "yum install nss-tools" and try again
2020/05/12 03:58:20 define JAVA_HOME environment variable to use the Java trust
2020/05/12 03:58:20 certificate installed properly in linux trusts
{"level":"info","ts":1589255900.2134988,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["192.168.1.200"]}
{"level":"info","ts":1589255900.213643,"logger":"tls","msg":"cleaned up storage units"}
{"level":"info","ts":1589255900.2137372,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1589255900.2137475,"msg":"serving initial configuration"}
2020/05/12 03:58:20 [INFO][192.168.1.200] Obtain certificate; acquiring lock...
2020/05/12 03:58:20 [INFO][192.168.1.200] Obtain: Lock acquired; proceeding...
2020/05/12 03:58:20 [INFO][192.168.1.200] Certificate obtained successfully
2020/05/12 03:58:20 [INFO][192.168.1.200] Obtain: Releasing lock
2020/05/12 03:58:20 [WARNING] Stapling OCSP: no OCSP stapling for [192.168.1.200]: no OCSP server specified in certificate
2020/05/12 03:58:26 http: TLS handshake error from 192.168.1.200:39010: no certificate available for '172.19.0.4'
2020/05/12 03:58:27 http: TLS handshake error from 192.168.1.200:39014: no certificate available for '172.19.0.4'
2020/05/12 03:59:15 http: TLS handshake error from 192.168.1.200:39018: no certificate available for '172.19.0.4'

If I THEN alter the Caddyfile ONLY and make it look like:

>>> Caddyfile
https://192.168.1.200:9999 172.19.0.4 {
        reverse_proxy crcc:8000
}

rebuild the containers with:

sudo rm -rf caddy-config/caddy caddy-data/caddy && docker-compose up -d --force-recreate

then I ERR_CERT_AUTHORITY_INVALID but can access the webpage anyway via chrome (again i’m visiting https://192.168.1.200:9999 ) and CAN access the webpage/see the whale saying hello-world

my logs after loading that page 3 times are:

{"level":"info","ts":1589256146.1785617,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"info","ts":1589256146.2017343,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1589256146.2019217,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1589256146.2019448,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv1","https_port":443}
{"level":"info","ts":1589256146.2019558,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv1"}
2020/05/12 04:02:26 [INFO][cache:0xc000820190] Started certificate maintenance routine
{"level":"info","ts":1589256146.2127438,"logger":"tls","msg":"setting internal issuer for automation policy that has only internal subjects but no issuer configured","subjects":["172.19.0.4","192.168.1.200"]}
{"level":"info","ts":1589256146.2130857,"logger":"tls","msg":"cleaned up storage units"}
{"level":"warn","ts":1589256146.2562604,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2020/05/12 04:02:26 Warning: "certutil" is not available, install "certutil" with "apt install libnss3-tools" or "yum install nss-tools" and try again
2020/05/12 04:02:26 define JAVA_HOME environment variable to use the Java trust
2020/05/12 04:02:26 certificate installed properly in linux trusts
{"level":"info","ts":1589256146.2857757,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["172.19.0.4","192.168.1.200"]}
{"level":"info","ts":1589256146.2859979,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1589256146.2860198,"msg":"serving initial configuration"}
2020/05/12 04:02:26 [INFO][192.168.1.200] Obtain certificate; acquiring lock...
2020/05/12 04:02:26 [INFO][172.19.0.4] Obtain certificate; acquiring lock...
2020/05/12 04:02:26 [INFO][192.168.1.200] Obtain: Lock acquired; proceeding...
2020/05/12 04:02:26 [INFO][172.19.0.4] Obtain: Lock acquired; proceeding...
2020/05/12 04:02:26 [INFO][192.168.1.200] Certificate obtained successfully
2020/05/12 04:02:26 [INFO][192.168.1.200] Obtain: Releasing lock
2020/05/12 04:02:26 [INFO][172.19.0.4] Certificate obtained successfully
2020/05/12 04:02:26 [INFO][172.19.0.4] Obtain: Releasing lock
2020/05/12 04:02:26 [WARNING] Stapling OCSP: no OCSP stapling for [192.168.1.200]: no OCSP server specified in certificate
2020/05/12 04:02:26 [WARNING] Stapling OCSP: no OCSP stapling for [172.19.0.4]: no OCSP server specified in certificate
2020/05/12 04:03:26 http: TLS handshake error from 192.168.1.200:39088: remote error: tls: unknown certificate
2020/05/12 04:03:26 http: TLS handshake error from 192.168.1.200:39092: remote error: tls: unknown certificate
2020/05/12 04:03:34 http: TLS handshake error from 192.168.1.200:39094: remote error: tls: unknown certificate

edit: in this case I’m accessing it from chrome directly on the server computer (192.168.1.200)

If you navigate to https://192.168.1.200:9999 in your browser, as far as I know, this should be impossible.

The browser should be sending 192.168.1.200 for SNI, not 172.19.0.4. It should work fine with the first Caddyfile, because that’s what your browser should be asking for.

Docker exposed ports aren’t terminating proxies, they’re just forwarding.

Something weird is going on, and I can’t quite put my finger on it.

The only other (possibly useless) information i can give is that 172.19.0.4 is added as a SubjectAlternativeName.

Yeah, look. I’m stumped.

I’ve been reduced to posting bad memes about this on Slack.

image

Hopefully this will tide you over until someone else (or myself) realizes this is probably some really simple thing and gives you like a two-sentence fix, because that’s how these things usually go. Hah.

2 Likes

Fun fact, browsers (at least Firefox) don’t send IP addresses in SNI. I think that’s against spec, actually. Meh, well, sort of:

Currently, the only server names supported are DNS hostnames

Of course, TLS clients can do whatever they want. :stuck_out_tongue:

1 Like

So is Caddy inferring the server name as the IP address the client is connecting to?

That implies that having an actual DNS hostname for the Docker host would work around this issue.

P.S. That was four sentences. Pretty close to the two I predicted. :thumbsup:

1 Like

As far as choosing a certificate goes, yes. If there’s no ServerName in the TLS ClientHello, then Caddy looks for a certificate with the IP address the client is connecting to. (Technically, CertMagic does this dirty work, but you get the picture.)

Edit: Not sure if still relevant, but this error:

TLS handshake error from 192.168.1.200:39088: remote error: tls: unknown certificate

just means that the browser (Chrome) rejected the ServerHello because of “unknown certificate” which is just an arbitrary message they chose to probably mean “I don’t trust this certificate” – in any case, Caddy apparently presented a cert in this case but the browser didn’t “know” it (whatever that means).

1 Like

…which, because Docker is port forwarding from the host to the Docker container’s IP address, is some 172.19.x.x IP…

Congratulations, everybody. The issue technically wasn’t DNS, this time! :tada:

So, basically, @dclman - I expect this would work just fine if you did this over HTTP instead OR if your server had a hostname you could browse to instead of an IP address (some kind of example.local or something).

Alternately, putting the Caddy container on the host network stack. But then, same issue - you’d have to expose Bitwarden on one of the host’s ports and probably refer to it as localhost:9999 from Caddy.

I think that’s fine - that’s just from when he added the 172.19.x.x to his Caddyfile, Caddy served a cert for it, and his browser auto rejected it the first few attempts until he manually accepted (see: “then I ERR_CERT_AUTHORITY_INVALID but can access the webpage anyway via chrome”).

1 Like

Cool. Mystery solved then?

You can also configure Caddy with a default_sni which it will use if the client does not specify a ServerName. In that case, it will use the name in default_sni when looking for a certificate. (Of course, the client may still reject it.)

1 Like

Yes! That’d be quite neat, actually. There’s a global option for it in the Caddyfile, too.

  • default_sni sets a default TLS ServerName for when clients do not use SNI in their ClientHello.

Global options (Caddyfile) — Caddy Documentation

@dclman, instead of adding 172.19.0.4 to your list of site labels, add default_sni 192.168.1.200 to a global options block (open a braced block at the very top of your Caddyfile) and see if that solves this issue.

1 Like

Adding that works!! Thanks so much for all of ya’lls help! 100% couldn’t have figured that out so I’m glad i asked heh

(for reference the final Caddyfile:)

{
        default_sni 192.168.1.200
}

https://192.168.1.200:9999  {
        reverse_proxy crcc:8000
}
2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.