Reverse proxy without domain name

I’m wondering if this is possible, and it’s ok for ‘not possible’ to be the right answer.

Briefly: I want to reverse proxy a service (bitwarden) that’s at a port over https.
I’d like to be able to access my service from
https://192.168.1.100:9999 and am not interested in getting a domain name.

I’m trying to set it up so that it works with docker-compose up -d and being a noob with docker networking + caddy seems to be my problem.

as something that’s super minimal:

docker-compose.yml

version: '3'
services:
  caddy:
    image: caddy:latest
    volumes:
      - /home/me/tmp/Caddyfile:/etc/Caddyfile
    ports:
        - 443:443
        - 80:80

  bitwarden:
    image: bitwardenrs/server:latest
    volumes:
      - /home/me/tmp/bw-data/:/data
    environment:
      - DOMAIN=http://192.168.1.100:8889
      - ROCKET_PORT=8889
    ports:
      - 8889:8889
    restart: unless-stopped

This serves the bitwarden service at http://192.168.1.100:8889 as intended

Caddyfile

:9999

reverse_proxy 127.0.0.1:8889

(via Reverse proxy quick-start — Caddy Documentation )
This doesn’t appear to work somewhat obviously as the caddy image can’t access port 8889 but I’m not sure how to give both images access to the same ports.

any help in making this happen would be appreciated!

This would need to be:

reverse_proxy bitwarden:8889

When running Caddy in a docker container, 127.0.0.1 refers to the container that Caddy is running in, not your host machine. When running with docker-compose, it has an internal DNS server that will resolve each of your services to the IP address of the container. In this case, your bitwarden container has the service name bitwarden, so you can use bitwarden as the domain to proxy to.

Also, big warning, make sure to set up volumes for the /data and /config directories when running Caddy in Docker or you will lose your certificates and keys every time you reboot Caddy! It’s very important to persist these directories to avoid hitting rate limits and other issues.

    volumes:
      - caddy_data:/data
      - caddy_config:/config
3 Likes

Hi francislavoie,
Thank you for the information!
I think i’m missing something though?
how does the Caddyfile know that bitwarden is a network service?

also, this still unfortunately doesn’t work – caddy has posted the slanted page to localhost, but bitwarden isn’t accessible at port 9999.

>>>caddyfile
:9999

reverse_proxy bitwarden:8889
>>>docker-compose.yml
version: '3'
services:
  caddy:
    image: caddy:latest
    volumes:
      - /home/me/tmp/Caddyfile:/etc/Caddyfile
      - /home/me/tmp/caddy-data:/data
      - /home/me/tmp/caddy-config:/config
    ports:
        - 443:443
        - 80:80

  bitwarden:
    image: bitwardenrs/server:latest
    volumes:
      - /home/me/tmp/bw-data/:/data
    environment:
      - DOMAIN=http://192.168.1.100:8889
      - ROCKET_PORT=8889
    ports:
      - 8889:8889
    restart: unless-stopped
``

You only publish ports 80 and 443, not port 9999. Either use :80 in your Caddyfile, or add - 9999:9999 to your docker-compose.yml

1 Like

Thank you for your help so far! i think i’m starting to understand it.

Now i’m stuck on getting HTTPS going though.
as is, I’m getting the error:

server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server

I tried execing into the caddy image and running the following because I wasn’t sure exactly what the Caddyfile was doing and I saw this in the docs.

caddy reverse-proxy --from bitwarden:8889 --to localhost:9999

This resulted in:

2020/05/08 09:27:39 [ERROR] acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rejectedIdentifier :: Error creating new order :
: Cannot issue for "bitwarden": Domain name needs at least one dot, url:  (challenge=tls-alpn-01 remaining=[http-01])
2020/05/08 09:27:41 [INFO] [bitwarden] acme: Obtaining bundled SAN certificate given a CSR
2020/05/08 09:27:41 [ERROR] acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rejectedIdentifier :: Error creating new order :
: Cannot issue for "bitwarden": Domain name needs at least one dot, url:  (challenge=http-01 remaining=[])
2020/05/08 09:27:43 [ERROR] attempt 1: [bitwarden] Obtain: [bitwarden] acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rejec
tedIdentifier :: Error creating new order :: Cannot issue for "bitwarden": Domain name needs at least one dot, url:  - retrying in 1m0s (5.024333769s/720h0m0s elapsed)...

But trying something really silly like changing my service to bitwarden.com yeilds errors as well (obviously because it’s trying the real bitwarden.com)

so what is the solution to something like this? What should i name my bitwarden service so that caddy can generate certs for it

That means “listen to requests from bitwarden:8889 and proxy them to localhost:9999”. That’s backwards from what you want. You would need to flip the from and to parameters.

But for now, let’s just stick to the Caddyfile. Since you’re already running it in Docker, it’ll be easier that way.

That’s not an error, just a notice to tell you that HTTPS won’t be enabled because you’re just listening on port :80.

To recap, this is what I’m assuming your Caddyfile looks like:

:80

reverse_proxy bitwarden:8889

From the host machine, you should be able to load http://localhost in your browser and it should give you bitwarden.

If that’s all working, then the next step is to set up HTTPS. It’s unclear to me why you’re trying to use port 9999 for this.

What you probably want to do is point a public domain to your machine, and replace :80 in your Caddyfile with that domain.

1 Like

Okay I’m getting closer!!

I want to go host this so it’s accessible only from 192.168.1.200

so i can turn on a vpn and access it via: https://192.168.1.200:9999

currently i have:

>>>Caddyfile
https://192.168.1.200:9999

reverse_proxy bitwarden:8889
>>>docker-compose.yml
version: '3'
services:
  caddy:
    image: caddy:latest
    volumes:
      - /home/me/Documents/bitwarden/Caddyfile:/etc/caddy/Caddyfile
      - /home/me/Documents/bitwarden/caddy-data:/data
      - /home/me/Documents/bitwarden/caddy-config:/config
    ports:
      - 443:443
      - 9999:9999
      - 80:80

  bitwarden:
    image: bitwardenrs/server:latest
    volumes:
      - /home/me/Documents/bitwarden/bw-data/:/data
    environment:
      - ROCKET_PORT=8889
      - DOMAIN=https://192.168.1.200:9999
    restart: unless-stopped
    ports:
      - 8889:8889

This works fine with localhost but i can’t seem to access it when on another network via the vpn

the problem seems to be this:

{"level":"info","ts":1588964943.3766,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"info","ts":1588964943.4025936,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1588964943.404573,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
2020/05/08 19:09:03 [INFO][cache:0xc000444190] Started certificate maintenance routine
{"level":"info","ts":1588964943.4277575,"logger":"tls","msg":"setting internal issuer for automation policy that has only internal subjects but no issuer configured","subjects":["192.168.1.200"]}
{"level":"info","ts":1588964943.4287522,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["192.168.1.200"]}
2020/05/08 19:09:03 [WARNING] Stapling OCSP: no OCSP stapling for [192.168.1.200]: no OCSP server specified in certificate
{"level":"info","ts":1588964943.4309916,"logger":"tls","msg":"cleaned up storage units"}
{"level":"warn","ts":1588964943.5721502,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2020/05/08 19:09:03 Warning: "certutil" is not available, install "certutil" with "apt install libnss3-tools" or "yum install nss-tools" and try again
2020/05/08 19:09:03 define JAVA_HOME environment variable to use the Java trust
2020/05/08 19:09:03 certificate installed properly in linux trusts
{"level":"info","ts":1588964943.661539,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1588964943.6615646,"msg":"serving initial configuration"}
2020/05/08 19:10:10 http: TLS handshake error from 10.8.0.6:43674: EOF
2020/05/08 19:11:29 http: TLS handshake error from 10.8.0.6:43700: no certificate available for '172.19.0.3'

How can i get it to allow access from a non-localhost address?

I might suggest that if your Caddyfile starts with https://192.168.1.200:9999, but the server still responds to localhost, you may not actually be running the updated configuration.

Strictly speaking, localhost != https://192.168.1.200:9999, so a request to localhost should result in a 404 Site not served on this interface response edit: a status 200 with an empty body. Definitely shouldn’t work as normal.

Hello all,
Ok I sort of figured it out, but I’m not sure if this is a great way to do it…

>>>Caddyfile
https://192.168.1.200:9999 172.19.0.3

reverse_proxy bitwarden:8889
>>>docker-compose.yml
version: '3'
services:
  caddy:
    image: caddy:latest
    volumes:
      - /home/me/Documents/bitwarden/Caddyfile:/etc/caddy/Caddyfile
      - /home/me/Documents/bitwarden/caddy-data:/data
      - /home/me/Documents/bitwarden/caddy-config:/config
    ports:
      - 9999:9999
      # - 443:443
      # - 80:80

  bitwarden:
    image: bitwardenrs/server:latest
    volumes:
      - /home/me/Documents/bitwarden/bw-data/:/data
    environment:
      - ROCKET_PORT=8889
      - DOMAIN=https://192.168.1.200:9999
    restart: unless-stopped

I would like to know 4 things (if anyone could help me out)

  1. Is this a good way to do things?
  2. What am I doing by adding the 172.19.0.3 part to my caddyfile?
  3. Is there a way to wildcard this? I assume that 172.19.3 won’t always be the address that pops up here.
  4. The 172 IP address is from docker which is giving the containers (some research suggests its the userland-proxy which seems to give a lot of people headaches)

There’s no right way to do stuff like this, only a few wrong ways. In the sense that there’s nothing wrong with this config, you could say this is a good way to go about it.

Good question?

Kinda looks like a Docker network internal IP, so I suppose it would allow Caddy to server requests coming from other Docker containers directly requesting the Caddy container’s IP.

You can’t guarantee the container will always have that IP address, Docker Network IPs are dynamically allocated.

Question is, why do other Docker containers need to be accommodated? Surely they don’t need to be served anything from Caddy, Caddy is there so you can access the other services through Docker. Unless it breaks something, I’d probably just not bother.

Yep, you could remove the hostname and Caddy would respond using this site config regardless of what clients request. But that seems like it would clash with your earlier-stated objective:

That’s one possibility. The Caddy image will then be running outside of the Docker network, directly on the host’s network stack.

One side effect is that reverse_proxy bitwarden:8889 will stop working (as the hostname bitwarden won’t resolve any more, since it’s no longer inside the Docker network). In fact, with no ports exposed on your Bitwarden service, it will become inaccessible until you expose a port, and Caddy would have to refer to it via localhost on the exposed port in question.

1 Like

thank you whitestrake for your super fast answer!

what do you mean by:

Question is, why do other Docker containers need to be accommodated? Surely they don’t need to be served anything from Caddy, Caddy is there so you can access the other services through Docker. Unless it breaks something, I’d probably just not bother.

If your question is: why do I need this? then my answer would be: because without it I get the error:

2020/05/08 19:11:29 http: TLS handshake error from 10.8.0.6:43700: no certificate available for '172.19.0.3'

I don’t have enough networking knowledge to really know what that means unfortunately, but i’m guessing caddy somehow needs to be told what ip addresses should have access to that proxy endpoint.

I guess the salient question here is: is this an error you need to deal with?

For example, I get this error in spades from my Caddy instances that face the internet at large.

Bots crawl the internet all the time, the entire IPv4 address space is pretty much indexed, and I’m constantly hit with these errors in my logs. They mean nothing to me - they only serve to let me know that a bot tried to connect to my HTTP server via the IP address and wasn’t served anything because they didn’t actually request one of the sites I actually serve.

By way of explanation, the error is Caddy saying:

http: TLS handshake error “I couldn’t establish a secure connection” from 10.8.0.6:43700 “with the client connecting from 10.8.0.6:43700” no certificate available “because I didn’t have any HTTPS certificate to transmit to them” for '172.19.0.3' “because they requested the hostname 172.19.0.3 (implying I’m not configured to serve that over HTTPS)”.

So - question is, why do you need to resolve this error? It clearly comes from one of the Docker containers trying to connect directly to Caddy’s container via its IP address. Is that something you need them to be able to do? If you don’t know, and nothing’s broken, the answer is almost certainly no.

1 Like

Again – thank you for the detailed explanation!

why do you need to resolve this error?

When I get this error, I cannot access my service. (https bitwarden in this case, but even a https hello-world works similarly)

If I do not have 172.19.0.3 added to the Caddyfile line, I cannot access that service that caddy is routing. chrome yields: ERR_SSL_PROTOCOL_ERROR

So if I am required to know the docker network address name, so that i can add it to my caddyfile and make my services visible, then it would appear I cannot use caddy since this (as you mentioned) is dynamically set – is that the case?

Also: the logfile lines i’m mentioned are not from some bot crawling the internet. This is just on a local server that isn’t published to the greater web. They happen once every time I try to access the page and get the SSL_PROTOCOL_ERROR that i’m describing, so they appear to be directly related to what I’m doing.

When the 172.19.03 is added to the caddyfile I get: ERR_CERT_AUTHORITY_INVALID from chrome – but that’s expected and can access the page – but instead get:

2020/05/12 03:14:27 http: TLS handshake error from 192.168.1.16:54478: remote error: tls: unknown certificate

but the page seems to work fine, so i’m ok ignoring that.

Okay, whoa, something is weird and I must have missed something big. But I only see two services in your Docker Compose project. Are there other services in the way here? Any other proxy layers etc?

Are you browsing directly to 172.19.0.3? Like, in your browser URL bar are you going straight to https://172.19.0.3/ ?

Specifically - that error happens when some client connects to a Caddy HTTPS listener and indicates, via SNI, that it specifically wants the web page at 172.19.0.3.

Like, if you put the literal URL https://foobarhostname in your browser and that somehow resolved to 172.19.0.3, you wouldn’t get this error, you’d get an error saying that it didn’t have a certificate for foobarhostname.

The thing that’s even weirder to me - 172.19.0.3 shouldn’t even be routable for any host I can imagine you’d be making requests from. It’s a Docker Network IP address. Your LAN has no conception of a route to 172.19 and would probably forward the request out to WAN if you made it from, say, a laptop. Or even from the Docker host itself, as far as I know.

This might make sense if you had some other proxy layer, and that proxy layer was intercepting requests for whatever-address-you’re-actually-typing and then translating to connect directly to 172.19.0.3 over standard HTTPS. But that, in and of itself, doesn’t make much sense to me.

With only caddy and bitwarden services running in your Docker Compose project, the only two hosts in play here that could feasibly be making requests to Caddy and asking for the website 172.19.0.3 are in fact Caddy itself, and Bitwarden. Why would Caddy need to serve a webpage to itself? Why would Bitwarden need Caddy to serve it a webpage?

That’s it – the described docker-compose.yml file + caddyfile is all i have running. no other docker containers are running. I’m not excluding anything from my files as I’ve posted them and will post another example below.

Are you browsing directly to 172.19.0.3?

I’m not going to 172.19.0.3, In my address bar I’m typing:

https://192.168.1.200:9999

With the following caddyfile + d-c.yml

>>>docker-compose.yml
version: '3'
services:
  caddy:
    image: caddy:latest
    volumes:
      - /home/me/Documents/bitwarden/Caddyfile:/etc/caddy/Caddyfile
      - /home/me/Documents/bitwarden/caddy-data:/data
      - /home/me/Documents/bitwarden/caddy-config:/config
    ports:
      - 9999:9999

  crcc:
    image: crccheck/hello-world
    restart: unless-stopped

<<<
>>> Caddyfile
https://192.168.1.200:9999 {
        reverse_proxy crcc:8000
}
<<<

If I access the website 3 times from chrome at https://192.168.1.200:9999/ I see the error: ERR_SSL_PROTOCOL_ERROR and cannot access the service each time and I see the logs:

{"level":"info","ts":1589255900.1151738,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"info","ts":1589255900.1204681,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1589255900.1206877,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
2020/05/12 03:58:20 [INFO][cache:0xc0007a0500] Started certificate maintenance routine
{"level":"info","ts":1589255900.1297123,"logger":"tls","msg":"setting internal issuer for automation policy that has only internal subjects but no issuer configured","subjects":["192.168.1.200"]}
{"level":"warn","ts":1589255900.1674604,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2020/05/12 03:58:20 Warning: "certutil" is not available, install "certutil" with "apt install libnss3-tools" or "yum install nss-tools" and try again
2020/05/12 03:58:20 define JAVA_HOME environment variable to use the Java trust
2020/05/12 03:58:20 certificate installed properly in linux trusts
{"level":"info","ts":1589255900.2134988,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["192.168.1.200"]}
{"level":"info","ts":1589255900.213643,"logger":"tls","msg":"cleaned up storage units"}
{"level":"info","ts":1589255900.2137372,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1589255900.2137475,"msg":"serving initial configuration"}
2020/05/12 03:58:20 [INFO][192.168.1.200] Obtain certificate; acquiring lock...
2020/05/12 03:58:20 [INFO][192.168.1.200] Obtain: Lock acquired; proceeding...
2020/05/12 03:58:20 [INFO][192.168.1.200] Certificate obtained successfully
2020/05/12 03:58:20 [INFO][192.168.1.200] Obtain: Releasing lock
2020/05/12 03:58:20 [WARNING] Stapling OCSP: no OCSP stapling for [192.168.1.200]: no OCSP server specified in certificate
2020/05/12 03:58:26 http: TLS handshake error from 192.168.1.200:39010: no certificate available for '172.19.0.4'
2020/05/12 03:58:27 http: TLS handshake error from 192.168.1.200:39014: no certificate available for '172.19.0.4'
2020/05/12 03:59:15 http: TLS handshake error from 192.168.1.200:39018: no certificate available for '172.19.0.4'

If I THEN alter the Caddyfile ONLY and make it look like:

>>> Caddyfile
https://192.168.1.200:9999 172.19.0.4 {
        reverse_proxy crcc:8000
}

rebuild the containers with:

sudo rm -rf caddy-config/caddy caddy-data/caddy && docker-compose up -d --force-recreate

then I ERR_CERT_AUTHORITY_INVALID but can access the webpage anyway via chrome (again i’m visiting https://192.168.1.200:9999 ) and CAN access the webpage/see the whale saying hello-world

my logs after loading that page 3 times are:

{"level":"info","ts":1589256146.1785617,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"info","ts":1589256146.2017343,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1589256146.2019217,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1589256146.2019448,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv1","https_port":443}
{"level":"info","ts":1589256146.2019558,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv1"}
2020/05/12 04:02:26 [INFO][cache:0xc000820190] Started certificate maintenance routine
{"level":"info","ts":1589256146.2127438,"logger":"tls","msg":"setting internal issuer for automation policy that has only internal subjects but no issuer configured","subjects":["172.19.0.4","192.168.1.200"]}
{"level":"info","ts":1589256146.2130857,"logger":"tls","msg":"cleaned up storage units"}
{"level":"warn","ts":1589256146.2562604,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2020/05/12 04:02:26 Warning: "certutil" is not available, install "certutil" with "apt install libnss3-tools" or "yum install nss-tools" and try again
2020/05/12 04:02:26 define JAVA_HOME environment variable to use the Java trust
2020/05/12 04:02:26 certificate installed properly in linux trusts
{"level":"info","ts":1589256146.2857757,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["172.19.0.4","192.168.1.200"]}
{"level":"info","ts":1589256146.2859979,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1589256146.2860198,"msg":"serving initial configuration"}
2020/05/12 04:02:26 [INFO][192.168.1.200] Obtain certificate; acquiring lock...
2020/05/12 04:02:26 [INFO][172.19.0.4] Obtain certificate; acquiring lock...
2020/05/12 04:02:26 [INFO][192.168.1.200] Obtain: Lock acquired; proceeding...
2020/05/12 04:02:26 [INFO][172.19.0.4] Obtain: Lock acquired; proceeding...
2020/05/12 04:02:26 [INFO][192.168.1.200] Certificate obtained successfully
2020/05/12 04:02:26 [INFO][192.168.1.200] Obtain: Releasing lock
2020/05/12 04:02:26 [INFO][172.19.0.4] Certificate obtained successfully
2020/05/12 04:02:26 [INFO][172.19.0.4] Obtain: Releasing lock
2020/05/12 04:02:26 [WARNING] Stapling OCSP: no OCSP stapling for [192.168.1.200]: no OCSP server specified in certificate
2020/05/12 04:02:26 [WARNING] Stapling OCSP: no OCSP stapling for [172.19.0.4]: no OCSP server specified in certificate
2020/05/12 04:03:26 http: TLS handshake error from 192.168.1.200:39088: remote error: tls: unknown certificate
2020/05/12 04:03:26 http: TLS handshake error from 192.168.1.200:39092: remote error: tls: unknown certificate
2020/05/12 04:03:34 http: TLS handshake error from 192.168.1.200:39094: remote error: tls: unknown certificate

edit: in this case I’m accessing it from chrome directly on the server computer (192.168.1.200)

If you navigate to https://192.168.1.200:9999 in your browser, as far as I know, this should be impossible.

The browser should be sending 192.168.1.200 for SNI, not 172.19.0.4. It should work fine with the first Caddyfile, because that’s what your browser should be asking for.

Docker exposed ports aren’t terminating proxies, they’re just forwarding.

Something weird is going on, and I can’t quite put my finger on it.

The only other (possibly useless) information i can give is that 172.19.0.4 is added as a SubjectAlternativeName.

Yeah, look. I’m stumped.

I’ve been reduced to posting bad memes about this on Slack.

image

Hopefully this will tide you over until someone else (or myself) realizes this is probably some really simple thing and gives you like a two-sentence fix, because that’s how these things usually go. Hah.

2 Likes

Fun fact, browsers (at least Firefox) don’t send IP addresses in SNI. I think that’s against spec, actually. Meh, well, sort of:

Currently, the only server names supported are DNS hostnames

Of course, TLS clients can do whatever they want. :stuck_out_tongue:

1 Like

So is Caddy inferring the server name as the IP address the client is connecting to?

That implies that having an actual DNS hostname for the Docker host would work around this issue.

P.S. That was four sentences. Pretty close to the two I predicted. :thumbsup:

1 Like