Docker caddy SSL protocol error

Help
#1

1. Output of caddy version:

I am using the latest version of the caddy Docker image:

$ docker pull caddy
Using default tag: latest
latest: Pulling from library/caddy
Digest: sha256:50743fc6130295e9e8feccd8b2f437d8c472f626bf277dc873734ed98219f44f
Status: Image is up to date for caddy:latest
docker.io/library/caddy:latest

2. How I run Caddy:

a. System environment:

I am using Docker on Ubuntu 22.04.1.

b. Command:

docker compose up imgsel-caddy

c. Service/unit/compose file:

services:
  imgsel-api:
    container_name: imgsel-api
    image: imgsel-api
    networks:
      - imgsel-net
    expose:
      - 8080:8080
    volumes:
      - /home/gamer/kwikserv/imgsel-data/data:/data
# this volume definition has been left out for brevity, it works as expected
      - nas_sftp:/media
    environment:
      DATA_FOLDER: "/data"
      MASTER_KEY: "<REDACTED>"
      MEDIA_DIR: "/media"
    tty: true
    stdin_open: true
  imgsel-client:
    container_name: imgsel-client
    image: imgsel-client
    networks:
      - imgsel-net
    expose:
      - 8090:8090
  imgsel-caddy:
    container_name: imgsel-caddy
    image: caddy
    depends_on:
      - imgsel-api
      - imgsel-client
    networks:
      - imgsel-net
      - imgsel-web
    ports:
      - 80:80
      - 443:443
      - 443:443/udp
    volumes:
      - /home/gamer/kwikserv/caddy/Caddyfile:/etc/caddy/Caddyfile
      - /home/gamer/kwikserv/caddy/data:/data

networks:
  imgsel-net:
    driver: bridge
    external: false
  imgsel-web:
    driver: bridge
    external: true

d. My complete Caddy config:

:443 {
	log {
		level DEBUG
	}
	reverse_proxy /api/v1/* imgsel-api:8080
	reverse_proxy imgsel-client:8090
}

3. The problem I’m having:

I am trying to proxy a client and api through caddy. Both are written by me, and I know it’s not a problem with my programs. The problem is that caddy seems to be unable to connect to other containers. If I change the compose file to use network_mode: host and update the reverse_proxy line to use localhost (and change the expose fields to ports), caddy works exactly as expected. I have tried everything I can think of and I am certain the problem is a networking problem.

The problem presents itself as a cryptic error in curl:

$ curl https://localhost -v
*   Trying 127.0.0.1:443...
* Connected to localhost (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Unknown (21):
* TLSv1.3 (IN), TLS alert, internal error (592):
* error:0A000438:SSL routines::tlsv1 alert internal error
* Closing connection 0
curl: (35) error:0A000438:SSL routines::tlsv1 alert internal error

4. Error messages and/or full log output:

Caddy does not print any errors when this happens. The logs are completely silent as I try this:

$ docker compose up imgsel-caddy
[+] Running 3/0
 ⠿ Container imgsel-client  Created                                                                                                     0.0s
 ⠿ Container imgsel-api     Created                                                                                                     0.0s
 ⠿ Container imgsel-caddy   Created                                                                                                     0.0s
Attaching to imgsel-caddy
imgsel-caddy  | {"level":"info","ts":1670138293.420376,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
imgsel-caddy  | {"level":"warn","ts":1670138293.4213204,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":9}
imgsel-caddy  | {"level":"info","ts":1670138293.421894,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
imgsel-caddy  | {"level":"info","ts":1670138293.422104,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
imgsel-caddy  | {"level":"info","ts":1670138293.4221163,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
imgsel-caddy  | {"level":"info","ts":1670138293.4221861,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000412770"}
imgsel-caddy  | {"level":"info","ts":1670138293.422477,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
imgsel-caddy  | {"level":"info","ts":1670138293.4226046,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
imgsel-caddy  | {"level":"info","ts":1670138293.4226315,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
imgsel-caddy  | {"level":"info","ts":1670138293.4227667,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
imgsel-caddy  | {"level":"info","ts":1670138293.4227867,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
imgsel-caddy  | {"level":"info","ts":1670138293.422935,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
imgsel-caddy  | {"level":"info","ts":1670138293.42294,"msg":"serving initial configuration"}
imgsel-caddy  | {"level":"info","ts":1670138293.4231048,"logger":"tls","msg":"finished cleaning storage units"

5. What I already tried:

I have tried using various hostnames to connect to the containers, as well as many different network setups. I even created a test container as such (attached to the same network as described above):

  imgsel-test:
    image: debian
    networks:
      - imgsel-net
    command: /bin/bash

and when I run it:

$ docker compose run imgsel-test
root@cfa9bed4a2d0:/# apt update >/dev/null 2>/dev/null && apt install curl -y >/dev/null 2>/dev/null
root@cfa9bed4a2d0:/# curl imgsel-client:8090 -v
*   Trying 172.22.0.2:8090...
* Connected to imgsel-client (172.22.0.2) port 8090 (#0)
> GET / HTTP/1.1
> Host: imgsel-client:8090
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Vary: RSC, Next-Router-State-Tree, Next-Router-Prefetch, Accept-Encoding
< X-Powered-By: Next.js
< Content-Type: text/html; charset=utf-8
< Cache-Control: private, no-cache, no-store, max-age=0, must-revalidate
< Date: Sun, 04 Dec 2022 07:24:12 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
< Transfer-Encoding: chunked
<
<!DOCTYPE html><html lang="en"> [ truncated for brevity, it sends what it's supposed to ]

This clearly demonstrates my network setup is correct, but Caddy still does not properly serve the HTML content. I am unable to debug the error any further because I cannot see the cause of the curl error, and googling it turned up nothing.

6. Links to relevant resources:

I have thoroughly read through the following documentation and am certain I am using the networks correctly:

#2

I was able to work around this problem by using a domain name instead of an IP address. I am still not satisfied with that, because I still have no idea what was causing the problem in the first place. The app works in network_mode: host, and it works with a domain name, but specifically does not work when routing between docker containers without a domain name. Am I missing something? I don’t really want to use a domain name for this project, but I currently don’t have any other options.

Thanks in advance.

#3

If you use :443 as a site address, Caddy has no idea what host requests will come in on.

This is important so that it can generate TLS certificates to complete the TLS handshake (the encryption part of the HTTPS connection).

You could enable On-Demand TLS to allow Caddy to request a cert to be issued on-the-fly as requests come in, but this has some caveats, and depends on your usecase.

Is this only for local/internal use? If so, you could do this:

https:// {
	tls internal {
		on_demand
	}
	reverse_proxy /api/v1/* imgsel-api:8080
	reverse_proxy imgsel-client:8090
}

This will instruct Caddy to always use its internal issuer to issue certificates for whatever host a request has, so if you make a request by IP, a cert will be issued for that IP. If a request has localhost, is will issue a cert for localhost, etc.

Keep in mind, the internal issuer uses a CA (Certificate Authority) managed by Caddy itself, so it won’t be publicly trusted. You should copy the root CA cert from Caddy’s storage (found at /data/caddy/pki/authorities/local/root.crt) to whatever systems will be making requests to Caddy, and install it to those systems’ trust stores. That’ll get rid of any trust warnings/errors from your clients.

FYI, level DEBUG doesn’t do much for you here, because access logs are never logged at DEBUG level.

This does enable access logs though, but all you need for that is log on its own.

To enable debug level logging, you can use the debug global option, by adding this at the top of your Caddyfile:

{
	debug
}

So, all that said, make sure to read through the On-Demand TLS docs to understand how it works. You’ll want to make sure to protect it from abuse with an ask URL if you go to production with this. (Or you probably won’t even need on-demand in production, unless you actually need more than one domain for the app, or you don’t know the domain ahead of time if it’s owned by a customer).

1 Like
#4

This topic was automatically closed after 30 days. New replies are no longer allowed.