Https on a local ip address with docker

1. Caddy version (caddy version):

2.3

2. How I run Caddy:

a. System environment:

Running caddy via docker-compose

b. Command:

docker-compose up

c. Service/unit/compose file:

version: '3.8'
services:
  caddy:
    image: caddy:latest
    container_name: caddy
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./caddy_data:/data
      - ./caddy_config:/config
      - ./site:/usr/share/caddy/docs
    ports:
      - 80:80
      - 443:443
    restart: unless-stopped
~

d. My complete Caddyfile or JSON config:

192.168.100.100 {
 root * /usr/share/caddy/docs
 file_server
}

3. The problem I’m having:

What I am trying to achieve is to have https working on a local/private ip using docker.
So I would like to connect to https://192.168.100.100
Unfortunately I cannot connect to https.

4. Error messages and/or full log output:

{"level":"info","ts":1610465476.8707533,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"info","ts":1610465476.888572,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1610465476.891617,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x2388eb0"}
{"level":"info","ts":1610465476.891653,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1610465476.892956,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1610465476.9287758,"logger":"tls","msg":"cleaned up storage units"}
{"level":"warn","ts":1610465477.1552107,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2021/01/12 15:31:17 Warning: "certutil" is not available, install "certutil" with "apt install libnss3-tools" or "yum install nss-tools" and try again
2021/01/12 15:31:17 define JAVA_HOME environment variable to use the Java trust
2021/01/12 15:31:17 certificate installed properly in linux trusts
{"level":"info","ts":1610465477.334109,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["192.168.100.100"]}
{"level":"warn","ts":1610465477.3459907,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [192.168.100.100]: no OCSP server specified in certificate"}
{"level":"info","ts":1610465477.3467617,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1610465477.3468492,"msg":"serving initial configuration"}

5. What I already tried:

I know that http is working, when I use

192.168.100.100:80 {
 root * /usr/share/caddy/docs
 file_server
}

I am able to access it.

It does give me a warning about certutil not being available.
But after that it says that the certificate has been properly installed.

If I run the binary on the raspberry pi without docker I am able to access it via https.
So it seems to be possible.

6. Links to relevant resources:

Those warnings are harmless, they’re basically saying Caddy couldn’t install the root CA certificate in the system’s trust store. Which is to be expected, since you’re running it in a container, which doesn’t have firefox or java environments for it to install it onto.

You’ll need to extract the root CA cert from the container (which is in the /data directory, so you should be able to find it in your ./caddy_data dir on your host machine) and install it to your system trust stores for your various devices you care to connect with, manually.

Thanks for your reply Francis!

I tried ‘installing’ the certificates that are under caddy/certificates/local/192.168.100.100
which are

192.168.100.100.crt
192.168.100.100.key

into

/etc/ssl/certs
/etc/ssl/private

respectively.

I suppose this is not the correct way as I am still unable to access the page through https.

Also I don’t quite understand why Caddy isn’t pointing to the certificates it created inside the docker container.

What I am expecting to see is a standard page that says that the key isn’t trusted but that I can accept the risk and continue (when I don’t trust the root CA).
I am mainly looking for the host to serve everything over https.

I meant the root CA cert in pki/authorities/local/root.crt, as mentioned in your logs:

Unfortunately I can’t seem to have caddy serve https contect locally.

This site can’t provide a secure connection192.168.100.100 sent an invalid response.
ERR_SSL_PROTOCOL_ERROR

is what Chrome keeps telling me.

when I try curl I get

* Expire in 0 ms for 6 (transfer 0x2d48a0)
*   Trying 192.168.100.100...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x2d48a0)
* Connected to 192.168.100.100 (192.168.100.100) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, internal error (592):
* error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
* Closing connection 0
curl: (35) error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error

However if I run the caddy binary directly:

* Expire in 0 ms for 6 (transfer 0xc0b8a0)
*   Trying 192.168.100.100...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0xc0b8a0)
* Connected to 192.168.100.100 (192.168.100.100) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_CHACHA20_POLY1305_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: [NONE]
*  start date: Jan 15 12:58:04 2021 GMT
*  expire date: Jan 16 00:58:04 2021 GMT
*  subjectAltName: host "192.168.100.100" matched cert's IP address!
*  issuer: CN=Caddy Local Authority - ECC Intermediate
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0xc0b8a0)
> GET / HTTP/2
> Host: 192.168.100.100
> User-Agent: curl/7.64.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200
< accept-ranges: bytes
< content-type: text/html; charset=utf-8
< etag: "qmtrq67x"
< last-modified: Tue, 12 Jan 2021 14:18:06 GMT
< server: Caddy
< content-length: 285
< date: Fri, 15 Jan 2021 12:59:52 GMT
<
<!doctype html>
<html>
  <head>
    <title>This is the title of the webpage!</title>
  </head>
  <body>
    <p>This is an example paragraph. Anything in the <strong>body</strong> tag will appear on the page, just like this <strong>p</strong> tag and its contents.</p>
  </body>
* Connection #0 to host 192.168.100.100 left intact
</html>

When I go there with Chrome I see

NET::ERR_CERT_AUTHORITY_INVALID

But I can click on advanced and proceed to the website anyway.
Which is what I want.

I found the following thread:

That is the same thing that I am experiencing.

So it seems I am connecting to docker’s own ip-space for which no certificate is available.
But of course over http it does work.

The fix you gave here:

is what you suggested me to try, which I did.
I copied the the root and intermediate crt files to /etc/ssl/certs/
and I copied the key fiels to /etc/ssl/private on the host machine.
Then I ran update-ca-certificates.
This doesn’t seem to work.

The only think I can think of is to not have docker run in it’s own network.
But let it connect directly to the host with the option network_mode: host (This works :partying_face: )
As discussed here:

But I wish to understand how to make it work with your suggestion.

You shouldn’t move the keys, only the certificate. Caddy should be the only thing that knows about the keys.

Chrome may cache the list of root certs. It’s known to do funky things.

When you run Caddy outside of the container, it will basically run caddy trust the first time it runs to install the root cert to the trust stores it can find. It might be possible to overwrite the certificate from your Docker storage to your host machine, then run caddy trust on the host to try to get it to install it for you.

1 Like

Thanks for your patience Francis!

The only way I can get it working without setting the networkmode to host is by
also adding the local ip of the docker container to the Caddyfile.

What doesn’t work:

{
 debug
}

192.168.100.100 {
 root * /usr/share/caddy/docs
 file_server
}
caddy    | {"level":"info","ts":1610828361.9812384,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy    | {"level":"info","ts":1610828361.9949377,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
caddy    | {"level":"info","ts":1610828361.9970167,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy    | {"level":"info","ts":1610828361.996882,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x37f72c0"}
caddy    | {"level":"info","ts":1610828361.9980552,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy    | {"level":"info","ts":1610828362.02465,"logger":"tls","msg":"cleaned up storage units"}
caddy    | {"level":"info","ts":1610828362.2759356,"logger":"pki.ca.local","msg":"root certificate is already trusted by system","path":"storage:pki/authorities/local/root.crt"}
caddy    | {"level":"debug","ts":1610828362.2771897,"logger":"http","msg":"starting server loop","address":"[::]:443","http3":false,"tls":true}
caddy    | {"level":"debug","ts":1610828362.2781663,"logger":"http","msg":"starting server loop","address":"[::]:80","http3":false,"tls":false}
caddy    | {"level":"info","ts":1610828362.278662,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["192.168.100.100"]}
caddy    | {"level":"warn","ts":1610828362.2851422,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [192.168.100.100]: no OCSP server specified in certificate"}
caddy    | {"level":"info","ts":1610828362.2864199,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
caddy    | {"level":"info","ts":1610828362.286672,"msg":"serving initial configuration"}
caddy    | {"level":"debug","ts":1610828366.7251027,"logger":"http.stdlib","msg":"http: TLS handshake error from 192.168.100.235:50760: no certificate available for '172.18.0.2'"}

By adding 172.18.0.2 which is the container ip, I managed to get it to work.
This isn’t a working solution as the container’s IP can change.

{
 debug
}

172.18.0.2 192.168.100.100 {
 root * /usr/share/caddy/docs
 file_server
}
caddy    | {"level":"info","ts":1610828614.9821067,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy    | {"level":"info","ts":1610828614.9986396,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
caddy    | {"level":"info","ts":1610828614.999977,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x303d090"}
caddy    | {"level":"info","ts":1610828615.001091,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy    | {"level":"info","ts":1610828615.0013645,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy    | {"level":"info","ts":1610828615.0268261,"logger":"tls","msg":"cleaned up storage units"}
caddy    | {"level":"info","ts":1610828615.282612,"logger":"pki.ca.local","msg":"root certificate is already trusted by system","path":"storage:pki/authorities/local/root.crt"}
caddy    | {"level":"debug","ts":1610828615.2837198,"logger":"http","msg":"starting server loop","address":"[::]:443","http3":false,"tls":true}
caddy    | {"level":"debug","ts":1610828615.2842872,"logger":"http","msg":"starting server loop","address":"[::]:80","http3":false,"tls":false}
caddy    | {"level":"info","ts":1610828615.2844894,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["192.168.100.100","172.18.0.2"]}
caddy    | {"level":"warn","ts":1610828615.291258,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [192.168.100.100]: no OCSP server specified in certificate"}
caddy    | {"level":"warn","ts":1610828615.2970598,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [172.18.0.2]: no OCSP server specified in certificate"}
caddy    | {"level":"info","ts":1610828615.2982106,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
caddy    | {"level":"info","ts":1610828615.2984517,"msg":"serving initial configuration"}
caddy    | {"level":"debug","ts":1610828621.591482,"logger":"http.handlers.file_server","msg":"sanitized path join","site_root":"/usr/share/caddy/docs","request_path":"/","result":"/usr/share/caddy/docs"}
caddy    | {"level":"debug","ts":1610828621.5921113,"logger":"http.handlers.file_server","msg":"located index file","filename":"/usr/share/caddy/docs/index.html"}
caddy    | {"level":"debug","ts":1610828621.5924804,"logger":"http.handlers.file_server","msg":"opening file","filename":"/usr/share/caddy/docs/index.html"}

You could use on-demand TLS with local certs if you don’t know ahead of time which IP the request will come from:

tls internal {
    on_demand
}

Thanks @francislavoie!
Your solution does the trick :smiling_face_with_three_hearts:

:443 {
 root * /usr/share/caddy/docs
 file_server
 tls internal {
   on_demand
 }
}
caddy    | {"level":"info","ts":1610872200.3049197,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy    | {"level":"info","ts":1610872200.3128304,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
caddy    | {"level":"info","ts":1610872200.3139806,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x4148f50"}
caddy    | {"level":"info","ts":1610872200.3219833,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy    | {"level":"info","ts":1610872200.3220723,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy    | {"level":"debug","ts":1610872200.3237476,"logger":"http","msg":"starting server loop","address":"[::]:443","http3":false,"tls":true}
caddy    | {"level":"debug","ts":1610872200.3245497,"logger":"http","msg":"starting server loop","address":"[::]:80","http3":false,"tls":false}
caddy    | {"level":"info","ts":1610872200.602408,"logger":"pki.ca.local","msg":"root certificate is already trusted by system","path":"storage:pki/authorities/local/root.crt"}
caddy    | {"level":"info","ts":1610872200.6034985,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
caddy    | {"level":"info","ts":1610872200.6035972,"msg":"serving initial configuration"}
caddy    | {"level":"info","ts":1610872200.6087456,"logger":"tls","msg":"cleaned up storage units"}
caddy    | {"level":"warn","ts":1610872214.7150831,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [172.20.0.2]: no OCSP server specified in certificate"}
caddy    | {"level":"debug","ts":1610872214.7488616,"logger":"http.stdlib","msg":"http: TLS handshake error from 192.168.100.235:51249: remote error: tls: unknown certificate authority"}
caddy    | {"level":"debug","ts":1610872221.0907576,"logger":"http.handlers.file_server","msg":"sanitized path join","site_root":"/usr/share/caddy/docs","request_path":"/","result":"/usr/share/caddy/docs"}
caddy    | {"level":"debug","ts":1610872221.0910258,"logger":"http.handlers.file_server","msg":"located index file","filename":"/usr/share/caddy/docs/index.html"}
caddy    | {"level":"debug","ts":1610872221.0911207,"logger":"http.handlers.file_server","msg":"opening file","filename":"/usr/share/caddy/docs/index.html"}
caddy    | {"level":"debug","ts":1610872221.208771,"logger":"http.handlers.file_server","msg":"sanitized path join","site_root":"/usr/share/caddy/docs","request_path":"/favicon.ico","result":"/usr/share/caddy/docs/favicon.ico"}

So if I understood the following correctly:

When I spin up a clean container, no certificate is available and none has been made yet.
As soon as I connect to the server, the server obtains a certificate and uses it.
The certificate it obtains is for the ip address that Docker is using.
So this would mitigate the problem that if the container would get a different ip, https would still work.

1 Like

Pretty much, yeah.

IMO, HTTPS with IP addresses is kinda janky, I’d recommend configuring your /etc/hosts with domains to use instead, or run a DNS server like CoreDNS or dnsmasq to make local domains.