Networking Caddy in a Podman container to other users (Resolved: Podman isolates user accounts)

1. The problem I’m having:

I am rebuilding my homelab server, ButonMash (Rocky 8.8), to use Caddy as a reverse proxy instead of binding each service to a port on the host.

I am trying to reverse proxy three already-existent services: Vaultwarden, Pi-Hole, and Nextcloud. Each is in a rootless Podman container (Nextcloud has a pod with MariaDB) and has its own user account. At present, I am working with a Caddy container running off Pi-Hole’s account. Pi-Hole is serving a DNS record for buttonmash.lan, and I have already worked out the inbound traffic with firewalld and Podman.

Caddy port forwarding
(Host_port->Host_port):container_port
(80->8000):80
(443->44300):443
2019:2019

Curling buttonmash.lan:2019 returns Hello World! when Caddy is running.
Both TCP and UDP are forwarded on ports 80 and 443.

My present goal is to reach vaultwarden.buttonmash.lan with a Bitwarden client and/or Firefox, but I’m having trouble finding the best practice way to reach it in terms of security. I have started up Caddy, Pi-Hole, and Vaultwarden with the flag “–network podman”, but was only able to curl <podman network ip of Caddy’s container>:2019 from Pi-Hole, the container Caddy is sharing a user with for now.

My present problem is getting traffic back out of Caddy’s Podman container and on to containers run by other users.

I’d prefer running Caddy rootless
I’d prefer using a Podman container.
I am aware of using a host-mode network, but security concerns.
I’d prefer to maintain user-account separation for various services, but am open to the idea as a backup plan before going rootfull.
My ideal solution would be to use a rootfull method to either make a Podman network rootless containers can connect to or somehow use slirp4netns (I know little beyond the name) or netavark (with Podman 4.0+ it works with rootless, and I’m on Podman 4.4.1)

2. Error messages and/or full log output:

Problem does not produce error logs.

3. Caddy version:

Caddy v2.7.5
docker.io/library/caddy:latest

4. How I installed and ran Caddy:

Caddy is installed from Docker

I use a script with to reset it (available below).

a. System environment:

Rocky Linux 8.8
CPU architecture: x86_64 (Intel Core i5-3470 CPU @ 3.20GHz)
Podman 4.4.1

systemd

$ systemctl --version
systemd 239 (239-74.el8_8.5)
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy

b. Command:

podman stop Caddy
podman rm Caddy
podman run -d \
        --name Caddy \
        --network podman \
        -p 2019:2019 \
        -p 8000:80 \
        -p 44300:443 \
        -v $PWD/Caddyfile:/etc/caddy/Caddyfile:z \
        -v caddy-config:/config \
        -v caddy-data:/data \
        caddy

c. Service/unit/compose file:

d. My complete Caddy config:

vaultwarden.buttonmash.lan {
        tls internal
        reverse_proxy localhost:8002
}

piholeadmin.buttonmash.lan {
        tls internal
        reverse_proxy 10.88.0.3:8001
}

:2019 {
        respond "Hello World!"
}

5. Links to relevant resources:

Too many to know which ones are relevant and which are trash.

I don’t have any experience with podman so I don’t think you’ll get much useful help here unfortunately.

Are all the relevant containers in the same network? I don’t know how podman networking differs from docker’s but are networks in podman isolated to particular users? Is that the problem?

Probably best if you ask for help on podman forums or something. But I’d be curious to know what the solution is if you find out.

Thanks for getting back. User separation does appear to be Podman’s philosophy, but I’m not entirely sure there is nothing left to try. “Firewall magic” is on my to-try list for a reason.

Update on my progress: Netavark and Aardvark are the tools I think I need. Slirp4nets is the old standard and the vast majority of online documentation still points in that direction.

“This is not possible, podman is designed to fully isolate users from each that includes networking.”

How Do I Network Rootless Containers Between Users? · containers/podman · Discussion #20408 · GitHub

I was heavily suspecting as much. My known options now are to

  1. use use networking mode host. (most popular)
  2. run all containers on a single user.
  3. run Caddy rootful.
  4. run Caddy natively (my original approach when I tried learning Caddy a couple years ago).
  5. route traffic back over localhost (least researched)

For now, I’m going to go with moving everything to a single, underprivileged user. While I’d appreciate some additional discussion with the Podman staff/community on what would be more secure were this a production environment instead of a homelab situation, I need to know when to stop researching and start doing.

1 Like

Good to know that you found your answer :+1:

FWIW, my opinion is that this is unnecessary paranoia and that Docker’s level of security/isolation is enough for 99.9999% of users.

RHEL switched its native containerization solution to Podman for version 8, I believe. And Rocky 8 is the downstream of RHEL 8. I’m also blogging about my projects, so even if I’d be fine totally yoloing it as long as I don’t do anything downright stupid, I at least feel some responsibility in case that .0001% of users shows up searching for a little-talked about tidbit.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.