HTTPs Caddy docker not working

1. Caddy version (caddy version):

v2.4.4
and
“org.opencontainers.image.version”: “v2.4.3”

2. How I run Caddy:

caddy run --watch [as root]
and
docker run -d -p 10.2.236.9:443:443 -p 10.2.236.9:80:80 -v $PWD/Caddyfile:/etc/caddy/Caddyfile -v caddy_data:/data caddy

a. System environment:

OS: Arch Linux x86_64
Kernel: 5.13.12-arch1-1
Docker version 20.10.8, build 3967b7d28e

d. My complete Caddyfile or JSON config:

{
    http_port 80
    https_port 443
    skip_install_trust
    local_certs
}
10.2.236.9
encode gzip
@insecureadmin {
    not remote_ip 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8 127.0.0.1
    path /admin*
}
redir @insecureadmin /
reverse_proxy /notifications/hub/negotiate 172.17.0.2:80
reverse_proxy /notifications/hub 172.17.0.2:3012
reverse_proxy 172.17.0.2:80 {
    # Send the true remote IP to Rocket, so that vaultwarden can put this in the
    # log, so that fail2ban can ban the correct IP.
    header_up X-Real-IP {remote_host}
}

3. The problem I’m having:

My caddyfile works great on a system without Docker, but in Docker I get an HTTPS error from the browser.

4. Error messages and/or full log output:

curl -v -k https://10.2.236.9
*   Trying 10.2.236.9:443...
* Connected to 10.2.236.9 (10.2.236.9) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, internal error (592):
* error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
* Closing connection 0
curl: (35) error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
docker logs magical_kapitsa
{"level":"info","ts":1630424142.4293022,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1630424142.43465,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1630424142.4363222,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["[::1]:2019","127.0.0.1:2019","localhost:2019"]}
{"level":"info","ts":1630424142.436763,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002be8c0"}
{"level":"info","ts":1630424142.4591525,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1630424142.4591875,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"warn","ts":1630424142.4601185,"logger":"pki.ca.local","msg":"root certificate trust store installation disabled; unconfigured clients may show warnings","path":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":1630424142.4602673,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
{"level":"info","ts":1630424142.4603434,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1630424142.4604115,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["10.2.236.9"]}
{"level":"info","ts":1630424142.4606602,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1630424142.4606822,"msg":"serving initial configuration"}
{"level":"info","ts":1630424142.4611225,"logger":"tls.obtain","msg":"acquiring lock","identifier":"10.2.236.9"}
{"level":"info","ts":1630424142.5088987,"logger":"tls.obtain","msg":"lock acquired","identifier":"10.2.236.9"}
{"level":"info","ts":1630424142.5127416,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"10.2.236.9"}
{"level":"info","ts":1630424142.5127726,"logger":"tls.obtain","msg":"releasing lock","identifier":"10.2.236.9"}
{"level":"warn","ts":1630424142.514271,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [10.2.236.9]: no OCSP server specified in certificate"}
openssl s_client -connect 172.17.0.3:443
CONNECTED(00000003)
139750776890752:error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:ssl/record/rec_layer_s3.c:1543:SSL alert number 80
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 293 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

5. What I already tried:

I built the caddyfile using a locale console with watch enabled as an argument. This worked for me without problems, when I wanted to test it in Docker I encountered this problem and unfortunately have no idea for a solution.

thank you for your help

Try turning on debug global option, it might reveal some more information in the logs. I’m not seeing the cause of the problem from those logs.

okay added debug to the Caddyfile

curl -v -k https://10.2.236.9
*   Trying 10.2.236.9:443...
* Connected to 10.2.236.9 (10.2.236.9) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, internal error (592):
* error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
* Closing connection 0
curl: (35) error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
35 % docker logs kind_visvesvaraya
{"level":"info","ts":1630427230.1861217,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1630427230.1901386,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1630427230.1939044,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["[::1]:2019","127.0.0.1:2019","localhost:2019"]}
{"level":"info","ts":1630427230.1943064,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003c0000"}
{"level":"info","ts":1630427230.2126198,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1630427230.2126515,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"warn","ts":1630427230.213393,"logger":"pki.ca.local","msg":"root certificate trust store installation disabled; unconfigured clients may show warnings","path":"storage:pki/authorities/local/root.crt"}
{"level":"debug","ts":1630427230.213571,"logger":"http","msg":"starting server loop","address":"[::]:443","http3":false,"tls":true}
{"level":"debug","ts":1630427230.2136729,"logger":"http","msg":"starting server loop","address":"[::]:80","http3":false,"tls":false}
{"level":"info","ts":1630427230.2136645,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
{"level":"info","ts":1630427230.2137,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["10.2.236.9"]}
{"level":"info","ts":1630427230.2137415,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1630427230.213882,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1630427230.2138994,"msg":"serving initial configuration"}
{"level":"info","ts":1630427230.2142239,"logger":"tls.obtain","msg":"acquiring lock","identifier":"10.2.236.9"}
{"level":"info","ts":1630427230.2667806,"logger":"tls.obtain","msg":"lock acquired","identifier":"10.2.236.9"}
{"level":"debug","ts":1630427230.2677712,"logger":"tls.obtain","msg":"trying issuer 1/1","issuer":"local"}
{"level":"info","ts":1630427230.269831,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"10.2.236.9"}
{"level":"info","ts":1630427230.2698483,"logger":"tls.obtain","msg":"releasing lock","identifier":"10.2.236.9"}
{"level":"warn","ts":1630427230.2707024,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [10.2.236.9]: no OCSP server specified in certificate"}
{"level":"debug","ts":1630427236.2316306,"logger":"http.stdlib","msg":"http: TLS handshake error from 10.2.236.9:56700: no certificate available for '172.17.0.3'"}

I see the handshake error, but i do not understand it :frowning:

I think what’s going on is you’re running Docker in swarm mode, so that enables its proxy, which changes the remote address on incoming connections. See this issue:

Using IP addresses for HTTPS is funky, I recommend just using a proper internal domain instead, to avoid this issue.

Or you could set your site address to :443 and enable On-Demand TLS for your site, if it’s for internal use only (so that Caddy will issue certificate on the fly for whatever IP address it sees on the request)

I think I do not use docker swarm:

docker service ls
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.

When you use Caddy’s internal CA for HTTPS in Docker, you have to install the root certificate manually into your trust stores (Because obviously a container does not have access to its host unless you do that.)

I use curl with -k… so no problem with trust

I have tested How to allow Docker containers to see the source IP address, but it did not help.

@francislavoie could you help me to set up On-Demand TLS? I think the clients would only see the docker internal IP in the certificate?

Yeah, you’d add this to your config:

tls {
	on_demand
}

Make sure to read these docs to understand how it works. Don’t use this without configuring the ask global options if you plan to make this publicly accessible.

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.