Can't get TLS cert when running in rootful container

1. The problem I’m having:

When running in a rootful container, I get an error obtaining a TLS certificate. The error does not appear when running rootless. I’m not sure what the problem is, because it seems to work on a different machine (when connecting to localhost). But the combination of rootful + hostname can’t get a TLS cert for some reason.

2. Error messages and/or full log output:

{"level":"info","ts":1743890292.8198779,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1743890292.8213668,"msg":"adapted config to JSON","adapter":"caddyfile"}
{"level":"info","ts":1743890292.8225834,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1743890292.8229473,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1743890292.8229668,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"warn","ts":1743890292.8229747,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
{"level":"info","ts":1743890292.82315,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003e8b80"}
{"level":"info","ts":1743890292.8235612,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1743890292.8242018,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"warn","ts":1743890292.8242671,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"warn","ts":1743890292.824274,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"info","ts":1743890292.8242786,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
{"level":"info","ts":1743890292.8242834,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["thebicpen.run.place"]}
{"level":"info","ts":1743890292.824496,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1743890292.8245063,"msg":"serving initial configuration"}
{"level":"info","ts":1743890292.8285968,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"380fbf4d-4f2a-45e5-9c8c-a5c3c5cfe5f7","try_again":1743976692.8285935,"try_again_in":86399.999999298}
{"level":"info","ts":1743890292.8291771,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1743890292.8453996,"logger":"tls.obtain","msg":"acquiring lock","identifier":"thebicpen.run.place"}
{"level":"info","ts":1743890292.87246,"logger":"tls.obtain","msg":"lock acquired","identifier":"thebicpen.run.place"}
{"level":"info","ts":1743890292.8725789,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"thebicpen.run.place"}
{"level":"info","ts":1743890292.8732195,"logger":"http","msg":"creating new account because no account for configured email is known to us","email":"","ca":"https://acme-v02.api.letsencrypt.org/directory","error":"open /data/caddy/acme/acme-v02.api.letsencrypt.org-directory/users/default/default.json: no such file or directory"}
{"level":"info","ts":1743890292.8732753,"logger":"http","msg":"ACME account has empty status; registering account with ACME server","contact":[],"location":""}
{"level":"info","ts":1743890292.8760707,"logger":"http","msg":"creating new account because no account for configured email is known to us","email":"","ca":"https://acme-v02.api.letsencrypt.org/directory","error":"open /data/caddy/acme/acme-v02.api.letsencrypt.org-directory/users/default/default.json: no such file or directory"}
{"level":"warn","ts":1743890297.8783362,"msg":"HTTP request failed; retrying","url":"https://acme-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-v02.api.letsencrypt.org on 10.89.0.1:53: read udp 10.89.0.27:59216->10.89.0.1:53: read: connection refused"}
{"level":"warn","ts":1743890300.2697873,"logger":"http","msg":"looking up info for HTTP challenge","host":"thebicpen.run.place","remote_addr":"66.133.109.36:34227","user_agent":"Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)","error":"no information found to solve challenge for identifier: thebicpen.run.place"}
{"level":"info","ts":1743890300.269844,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"66.133.109.36","remote_port":"34227","client_ip":"66.133.109.36","proto":"HTTP/1.1","method":"GET","host":"thebicpen.run.place","uri":"/.well-known/acme-challenge/lNvoA4VnaxBh8G8jVO9KhuBv9UKuGWUBgrCYcy0DOfM","headers":{"User-Agent":["Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"],"Accept":["*/*"],"Accept-Encoding":["gzip"],"Connection":["close"]}},"bytes_read":0,"user_id":"","duration":0.000119414,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://thebicpen.run.place/.well-known/acme-challenge/lNvoA4VnaxBh8G8jVO9KhuBv9UKuGWUBgrCYcy0DOfM"],"Content-Type":[]}}
{"level":"error","ts":1743890302.6919863,"logger":"tls","msg":"tls-alpn challenge","remote_addr":"66.133.109.36:32267","server_name":"thebicpen.run.place","error":"no information found to solve challenge for identifier: thebicpen.run.place"}
{"level":"warn","ts":1743890303.159352,"msg":"HTTP request failed; retrying","url":"https://acme-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-v02.api.letsencrypt.org on 10.89.0.1:53: read udp 10.89.0.27:56242->10.89.0.1:53: read: connection refused"}
{"level":"warn","ts":1743890308.4117455,"msg":"HTTP request failed; retrying","url":"https://acme-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-v02.api.letsencrypt.org on 10.89.0.1:53: read udp 10.89.0.27:46548->10.89.0.1:53: i/o timeout"}
{"level":"error","ts":1743890308.4119136,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"thebicpen.run.place","issuer":"acme-v02.api.letsencrypt.org-directory","error":"registering account [] with server: provisioning client: performing request: Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-v02.api.letsencrypt.org on 10.89.0.1:53: read udp 10.89.0.27:46548->10.89.0.1:53: i/o timeout"}
{"level":"error","ts":1743890308.4119632,"logger":"tls.obtain","msg":"will retry","error":"[thebicpen.run.place] Obtain: registering account [] with server: provisioning client: performing request: Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-v02.api.letsencrypt.org on 10.89.0.1:53: read udp 10.89.0.27:46548->10.89.0.1:53: i/o timeout","attempt":1,"retrying_in":60,"elapsed":15.539475939,"max_duration":2592000}
{"level":"info","ts":1743890368.4130275,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"thebicpen.run.place"}
{"level":"info","ts":1743890368.4135437,"logger":"http","msg":"creating new account because no account for configured email is known to us","email":"","ca":"https://acme-staging-v02.api.letsencrypt.org/directory","error":"open /data/caddy/acme/acme-staging-v02.api.letsencrypt.org-directory/users/default/default.json: no such file or directory"}
{"level":"info","ts":1743890368.4135907,"logger":"http","msg":"ACME account has empty status; registering account with ACME server","contact":[],"location":""}
{"level":"info","ts":1743890368.4185772,"logger":"http","msg":"creating new account because no account for configured email is known to us","email":"","ca":"https://acme-staging-v02.api.letsencrypt.org/directory","error":"open /data/caddy/acme/acme-staging-v02.api.letsencrypt.org-directory/users/default/default.json: no such file or directory"}
{"level":"warn","ts":1743890373.4198558,"msg":"HTTP request failed; retrying","url":"https://acme-staging-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 10.89.0.1:53: read udp 10.89.0.27:42152->10.89.0.1:53: read: connection refused"}
{"level":"warn","ts":1743890378.6723008,"msg":"HTTP request failed; retrying","url":"https://acme-staging-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 10.89.0.1:53: read udp 10.89.0.27:47676->10.89.0.1:53: read: connection refused"}
{"level":"warn","ts":1743890383.9242473,"msg":"HTTP request failed; retrying","url":"https://acme-staging-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 10.89.0.1:53: read udp 10.89.0.27:35740->10.89.0.1:53: read: connection refused"}
{"level":"error","ts":1743890383.924418,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"thebicpen.run.place","issuer":"acme-v02.api.letsencrypt.org-directory","error":"registering account [] with server: provisioning client: performing request: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 10.89.0.1:53: read udp 10.89.0.27:35740->10.89.0.1:53: read: connection refused"}

3. Caddy version:

4. How I installed and ran Caddy:

sudo iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
sudo iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
# Improve QUIC perf
# https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes
sudo sysctl -w net.core.rmem_max=7500000
sudo sysctl -w net.core.wmem_max=7500000
sudo loginctl enable-linger
sudo sysctl --system

sudo podman-compose up -d

a. System environment:

podman version 4.9.3

b. Command:

N/A

c. Service/unit/compose file:

services:
  caddy:
    image: docker.io/caddy:2.9
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./site:/srv
      - caddy_data:/data
      - caddy_config:/config

volumes:
  caddy_data:
  caddy_config:

d. My complete Caddy config:

(static_site) {
        # Set root directory to the dir mounted from the host
        root * /srv

        # Enable the static file server
        file_server

        # Enable logging
        log

        handle_errors {
                respond "{err.status_code} {err.status_text}"
        }
}

# Needs to match the DNS A record for TLS to work
thebicpen.run.place {
        import static_site
}

# TLS won't work for this config
:80 {
        import static_site
}

5. Links to relevant resources:

You seem to have a couple of problems here.

First, don’t mount the Caddyfile directly. Check out Caddy’s Docker page for details:

Then, the very first error message in your log is:

"error":"open /data/caddy/acme/acme-v02.api.letsencrypt.org-directory/users/default/default.json: no such file or directory"

Double-check that Caddy has the right permissions to create files and folders in /data.

Next error:

"error":"performing request: Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-v02.api.letsencrypt.org on 10.89.0.1:53: read udp 10.89.0.27:59216->10.89.0.1:53: read: connection refused"

Zooming in on this part:

read udp 10.89.0.27:59216->10.89.0.1:53: read: connection refused

It looks like your DNS server at 10.89.0.1:53 is refusing connections.

Start with these two issues first - fixing those might resolve the rest. If not, we can dig into whatever’s left after that.

First, don’t mount the Caddyfile directly.

Thanks for the info. I updated the compose file to mount the directory containing the caddyfile rather than the file itself.

It’s inside the container, so I think caddy should have permissions to access it. Caddy runs as root in the container after all.

$ sudo podman exec --interactive --workdir /etc/caddy --tty thebicpen-site_caddy_1 /bin/sh
/etc/caddy # ls -l /data/caddy/
total 12
-rw-------    1 root     root            36 Feb 23 08:11 instance.uuid
-rw-------    1 root     root           106 Apr  5 21:09 last_clean.json
drwx------    2 root     root          4096 Apr  5 23:32 locks
/etc/caddy # ps -ef
PID   USER     TIME  COMMAND
    1 root      0:00 caddy run --config /etc/caddy/Caddyfile --adapter caddyfile
   11 root      0:00 /bin/sh
   13 root      0:00 ps -ef
/etc/caddy #

Still getting the same result with the 1st issue taken care of.

Not sure if this is related, but 10.89.0.1 is the gateway for the podman network. Is the outgoing DNS request being blocked by podman?

$ podman network inspect thebicpen-site_default 
[
     {
          "name": "thebicpen-site_default",
          "id": "bd3145e47bac6fbe5dcc522286600de01f76d20e32a61996b7bb5b72f4a6ee4c",
          "driver": "bridge",
          "network_interface": "podman1",
          "created": "2025-02-23T08:06:21.984652055Z",
          "subnets": [
               {
                    "subnet": "10.89.0.0/24",
                    "gateway": "10.89.0.1"
               }
          ],
          "ipv6_enabled": false,
          "internal": false,
          "dns_enabled": true,
          "labels": {
               "com.docker.compose.project": "thebicpen-site",
               "io.podman.compose.project": "thebicpen-site"
          },
          "ipam_options": {
               "driver": "host-local"
          }
     }
]