Where are LetsEncrypt certs stored?

gammaray · April 28, 2023, 6:56pm

1. The problem I’m having:

I am using Caddy in a Docker container as a reverse proxy to backend services. I am also using Tailscale with HTTPS enabled so it creates LetsEncrypt certs.

The problem is that every time the container restarts, the LE certs are lost, and need to be regenerated again. I am trying to figure out the location in the container where the LE certs are saved, so I can add that as a volume so that the existing certs are re-used.

2. Error messages and/or full log output:

I have set the environmenet variable XDG_DATA_HOME=/var/lib in the container.

When Caddy starts, I can see that it is using the location.

{"level":"info","ts":1682706459.556254,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy"}

When I connect to the service over Tailscale, I can see Caddy creates the LE cert (via Tailscale), and I can access the site, but there are no files in the directory.

# tailscale
2023/04/28 18:35:48 cert("server.taild3f12.ts.net"): registered ACME account.
2023/04/28 18:35:48 cert("server.taild3f12.ts.net"): starting SetDNS call...
2023/04/28 18:35:50 tshttpproxy: CONNECT response from http://proxy.mycompany.com:3128 for target "log.tailscale.io:443" (auth ""): 200 Connection established
2023/04/28 18:35:57 Accept: TCP{100.71.202.19:50865 > 100.126.47.29:443} 40 tcp non-syn
2023/04/28 18:35:59 cert("server.taild3f12.ts.net"): did SetDNS
2023/04/28 18:36:00 cert("server.taild3f12.ts.net"): requesting cert...
2023/04/28 18:36:01 cert("server.taild3f12.ts.net"): got cert

# caddy
{"level":"debug","ts":1682706961.15812,"logger":"tls.handshake","msg":"using externally-managed certificate","remote_ip":"127.0.0.1","remote_port":"39146","sni":"server.taild3f12.ts.net","names":["server.taild3f12.ts.net"],"expiration":1690479362}

The data directory in the container

d3c58d7db47c:/# ls -al /var/lib/caddy
total 8
drwxr-xr-x    2 root     root          4096 Apr 28 17:24 .
drwxr-xr-x    1 root     root          4096 Apr 28 18:27 ..

3. Caddy version:

v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=

4. How I installed and ran Caddy:

The container is Alpine Linux, with Tailscale and Caddy binaries added.
I have a script the generates the Caddyfile from env vars, and then starts it with

caddy run --config /etc/caddy/Caddyfile --adapter caddyfile --environ

a. System environment:

Docker container running on a Mac

d. My complete Caddy config:

{
    debug
}

server.taild3f12.ts.net

reverse_proxy https://server.mycompany.com {
    header_up Host server.mycompany.com
    header_down Location "^https://server.mycompany.com(.*)$" "https://server.taild3f12.ts.net$1"
}

5. Links to relevant resources:

Env var for data location

jok · April 28, 2023, 7:05pm

if you want to persist certificates between container restarts you have to use volumes.
on the caddy docker hub page is a section " A note about persisted data" where the details are explained.

gammaray · April 28, 2023, 7:25pm

Hi,

I know that I need to use a volume, but I’m trying to figure out what directory that volume should be.

There are no files in the /data or /config directory after the certificate is created.

jok · April 28, 2023, 7:37pm

can you share your Dockerfile? the docker hub page is for the stock caddy docker image, with its own XDG_CONFIG_HOME and XDG_DATA_HOME definitions (/config and /data)
i see you set XDG_DATA_HOME=/var/lib
What is the content of /var/lib ?

gammaray · April 28, 2023, 7:48pm

Here is the Dockerfile.

ARG CADDY_TAG
ARG TAILSCALE_TAG
FROM caddy:${CADDY_TAG} as caddy
FROM tailscale/tailscale:${TAILSCALE_TAG} as tailscale
FROM our-custom-image:alpine-py3.10

ENV XDG_DATA_HOME /var/lib

RUN mkdir -p \
    /config \
    /data \
    /var/log/supervisord \
    /etc/caddy \
    /var/lib/caddy \
    && pip3 install supervisor \
    && apk add iptables

COPY --from=tailscale /usr/local/bin/tailscale* /usr/local/bin
COPY --from=tailscale /usr/local/bin/containerboot /usr/local/bin
COPY --from=caddy /etc/caddy /etc
COPY --from=caddy /usr/share/caddy /usr/share
COPY --from=caddy /usr/bin/caddy /usr/bin
COPY --from=caddy /config/caddy /config
COPY --from=caddy /data/caddy /data

COPY etc/caddy/Caddyfile /etc/caddy
COPY etc/supervisor/supervisord.conf /etc/supervisor/supervisord.conf
COPY scripts /usr/local/bin

RUN chmod +x /usr/local/bin/*.sh
RUN chmod +x /usr/local/bin/*.py


ENTRYPOINT ["/usr/local/bin/supervisord", "-c", "/etc/supervisor/supervisord.conf"]

It’s basically a combination of the Tailscale and Caddy Dockerfiles, using Supervisord to start them both.

89ad079dedb8:/# ls -al /var/lib/
total 36
drwxr-xr-x    1 root     root          4096 Apr 28 18:34 .
drwxr-xr-x    1 root     root          4096 Apr 28 17:24 ..
drwxr-xr-x    2 root     root          4096 Mar 29 14:45 apk
drwxr-xr-x    2 root     root          4096 Apr 28 17:24 caddy
drwxr-xr-x    2 root     root          4096 Apr 28 17:25 iptables
drwxr-xr-x    2 root     root          4096 Mar 29 14:45 misc
drwx------    2 root     root          4096 Apr 28 18:34 tailscale
drwxr-xr-x    2 root     root          4096 Mar 29 14:45 udhcpd

jok · April 28, 2023, 8:25pm

ok, the certificates should be in /var/lib/caddy/certificates
with the stock caddy image i have this:

/srv # ls -al /data/
total 12
drwxr-xr-x    3 root     root          4096 Mar 20  2020 .
drwxr-xr-x    1 root     root          4096 Apr 17 16:15 ..
drwx------    6 root     root          4096 Apr 11 03:32 caddy
/srv # ls -al /data/caddy/
total 24
drwx------    6 root     root          4096 Apr 11 03:32 .
drwxr-xr-x    3 root     root          4096 Mar 20  2020 ..
drwx------    5 root     root          4096 Sep 10  2021 acme
drwx------    3 root     root          4096 Jan 26  2022 certificates
drwx------    2 root     root          4096 Apr 11 03:32 locks
drwx------    2 root     root          4096 Apr 28 10:54 ocsp

hm, maybe it’s the supervisord. what does the supervisord.conf looks like?

gammaray · April 28, 2023, 8:30pm

Here’s the supervisor config

[unix_http_server]
file=/tmp/supervisor.sock

[supervisord]
pidfile = /var/run/supervisord.pid
nodaemon = true

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///tmp/supervisor.sock

[program:tailscale]
command = /usr/local/bin/containerboot tailscaled
numprocs = 1
process_name = tailscaled
user = root
startsecs = 1
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stdout
stderr_logfile_maxbytes=0

[program:caddy]
command = /usr/local/bin/run-caddy.sh
numprocs = 1
process_name = caddy
user = root
startsecs = 1
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stdout
stderr_logfile_maxbytes=0

The run-caddy.sh script:

#!/bin/sh

python3 /usr/local/bin/modify-caddyfile.py
exec caddy run --config /etc/caddy/Caddyfile --adapter caddyfile --environ

The Python script just modifies the Caddyfile based on environmenet variables

gammaray · April 28, 2023, 9:00pm

Ok, I figured it out, and it turns out, it’s not a Caddy problem. Or a problem at all.

I thought that Caddy was doing the LE request, but its actually Tailscale.
I took a look though the logs, and saw this:

2023/04/28 18:34:11 Program starting: v1.38.3-t47ebe6f95, Go 1.20.1-tsdb4dc90: []string{"tailscaled", "--socket=/var/run/tailscale/tailscaled.sock", "--state=mem:", "--statedir=/tmp", "--tun=userspace-networking"}

I looked in /tmp, and there it was, the certs directory, and the LE certs.

89ad079dedb8:/# ls -al /tmp/certs/
total 24
drwx------    2 root     root          4096 Apr 28 18:36 .
drwxrwxrwt    1 root     root          4096 Apr 28 18:35 ..
-rw-------    1 root     root           227 Apr 28 18:35 acme-account.key.pem
-rw-r--r--    1 root     root          5343 Apr 28 18:36 server.taild3f12.ts.net.crt
-rw-------    1 root     root           227 Apr 28 18:36 server.taild3f12.ts.net.key

I also found that there is an env var you can add to tell Tailscale where to store its state:

TS_TAILSCALED_EXTRA_ARGS=--statedir /var/lib/caddy

And now the certs are in /var/lib/caddy/certs.

Thank you for your help @jok!

jok · April 28, 2023, 9:05pm

yes, tailscale and *.ts.net-domains seems to be a special case, where tailscale manages the certs and not caddy.
i run both myself, but not combined as you.

francislavoie · April 28, 2023, 9:38pm

You don’t need this, because data doesn’t exist in those directories when the image is built. They only get filled at runtime.

But yeah, glad you figured it out with tailscale, Caddy asks the tailscale daemon for the certs as it needs them, so it doesn’t need to store anything.

system · May 28, 2023, 9:38pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.