Caddy reverse proxy with podman networking problem

Hello Everyone,

I an trying to set up my first own home server have problems working with Caddy because I am just a beginner and who is just learing by doing.

I think my problems are mainly networking as I don’t really know how to correctly work with https and how everything works.

My set up is as follows:

  • I am running Alma Linux 10 on my server with podman version 5.4.0.
  • I have talked to our internet provider and now have an IPV4 address instead of an IPV6, as my mobile carrier only has ipv4. I think I need this to be able to connect to the server over the internet.
  • I use the dynamic DNS service from Infomaniak. I wrote a bash script which updates my ip address every half an hour.

My goal is to use podman container for everything which update on their own and everything is always up-to-date. For that I am working with Quadlets files as it makes things easily reproducible.

I want to use a podman Caddy container as a reverse proxy, so that I can have multiple domains pointing to my server and the reverse proxy then guides the traffic to the correct container.
All the traffic should be encrypted, but the reverse proxy runs on the same server as all my services.
I run the reverse proxy in a separate container, but all the nextcloud related containers are together in one single pod with the following Quadlet file.

[Pod]
PodName=nextcloud
PublishPort=8080:80
PublishPort=8443:443

[Service]
Restart=always

[Install]
WantedBy=default.target

As you see, I send the server port 8443 to the port 443 inside of the pod for the nextcloud containers.

This is where my problems start. I tried to set up a simple caddy reverse proxy in the following way with the following Caddyfile.

nextcloud.ppflaum.ch {
    reverse_proxy https://192.168.1.188:8443 {
        transport http { 
            tls_insecure_skip_verify 
        }
    }
}

For context, my Quadlet file looks as follows.

[Container]
ContainerName=caddy-nextcloud
Pod=nextcloud.pod
Image=docker.io/library/caddy:latest
AutoUpdate=registry
Volume=/mnt/storage/nextcloud/config:/var/www/html:z
Volume=/mnt/storage/nextcloud/caddy/Caddyfile:/etc/caddy/Caddyfile:z

[Unit]
After=nextcloud.pod
After=mariadb-nextcloud.container
After=nextcloud-app.container
Requires=nextcloud.pod
Requires=mariadb-nextcloud.container
Requires=nextcloud-app.container

[Service]
Restart=always

[Install]
WantedBy=default.target

I also get all the containers up an running without any issue. If I test the nextcloud-caddy container over the network via the http I also get the sign in page.
I also see incoming traffic in the logs of the reverse proxy container BUT I am not able to reach the nextcloud instance via the reverse proxy. If I look at the log of the reverse proxy I always see error along those lines.

{"level":"error","ts":1748613563.7038028,"logger":"http.log.error","msg":"remote error: tls: internal error","request":{"remote_ip":"84.75.56.130","remote_port":"50693","client_ip":"84.75.56.130","proto":"HTTP/2.0","method":"GET","host":"nextcloud.ppflaum.ch","uri":"/:8443","headers":{"Te":["trailers"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"Priority":["u=0, i"],"Sec-Fetch-Dest":["document"],"Accept-Language":["de,en-US;q=0.7,en;q=0.3"],"Cookie":["REDACTED"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:139.0) Gecko/20100101 Firefox/139.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"Dnt":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"nextcloud.ppflaum.ch"}},"duration":0.001214852,"status":502,"err_id":"dp0n0w4kn","err_trace":"reverseproxy.statusError (reverseproxy.go:1390)"}

But I don’t see anything in logs on the nextcloud caddy server which has the following Caddyfile

https://192.168.1.188:443 {
    tls internal

    root * /var/www/html
    file_server

    php_fastcgi nextcloud-app:9000 {
        root /var/www/html
        env front_controller_active true
    }
    encode gzip
    log {
        output file /data/nextcloud-access.log
    }
    header {
        Strict-Transport-Security "max-age=15768000;includeSubDomains;preload"
    }
    # .htaccess / data / config / ... shouldn't be accessible from outside
    @forbidden {
        path /.htaccess
        path /data/*
        path /config/*
        path /db_structure
        path /.xml
        path /README
        path /3rdparty/*
        path /lib/*
        path /templates/*
        path /occ
        path /console.php
    }
    respond @forbidden 404
    redir /.well-known/carddav /remote.php/dav 301
    redir /.well-known/caldav /remote.php/dav 301
}

I have tried multiple variations of my setup and tried to solve the issue for houres using chatgpt but nothing worked.

I think, that the problem is, that the nextcloud caddy server is not able to talk to the reverse proxy.

I would be very greatful, if someone could explain to me, what is going wrong here and what I should do to solve this problem.

If you generally have some feedback towards my approach of setting up my home server, please leave some comments, as I am just a beginner and feedback could be very helpful.

It would also be helpful, if someone could explain to me, how exactly https work, and if someone has minimal working example, with two container and reverse proxying via https it would be great to share it with me. From there I could try to work it out by myself.

Thank you in advance for your help.

Dear @beginner.42

To really figure this out, we’d need some more information about the exact topology of the host network and your containers.

  1. How exactly are you running nextcloud? Are you using nextcloud AIO container or do you have a custom build container? (In the latter case I don’t think I can help with nextcloud specific issues, just maybe the networking)
  2. Can you specify all the caddy containers, I am a bit confused about which caddy is the reverse proxy, since you seem to have a second “nextcloud caddy server”.
  3. Can you provide the commands and output of the http access which worked, and the request which did not, including from which container if any the second non functioning one is issued.
  4. Whats the main nextclouds quadlet file?
  5. Which caddyfile belongs to which quadlet?
  6. is 192.168.1.188 the pods IP? did you assign it somewhere? In that case the rewrite from url to ip:8443 will not be rewritten to 443 in case you were depending on the pod rewrite for the port mapping nextcloud.domain (implicilty on 443 in https) → ip:8443 → ip:443 (pod doing the rewrite). Pods can internally communicate via localhost. So you could use localhost:443 as the proxy directive, taking to account the docs about https proxying
  7. From what I can tell is that you want to use the reverse proxy in general, not just for nextcloud, in which case it has little value to have them in the same pod, unless you want to runn every other unrelated container service you want to proxy to in the nextcloud pod too.
  8. I think in any case, to avoid naming, ip and port conflicts, it makes sense to not have them in the same pod.
  9. If using nextcloud AIO, best follow their docs here (general) and here (specific). Also this example docker-compose config, but may be adaptable to quadlets.
  10. Have you considered firewall issues?

Preliminary thoughts:

Here’s what I see as potential issues.

Note
My mental model for your settup is that the “nextcloud caddy server” has a similar role to the Nextcloud AIO apache webserver, i.e. the server which directly servers nextcloud directly.

I will go through you following you posts flow.

Nextcloud pod

According to podmans “basic” networking docs:

https://github.com/containers/podman/blob/main/docs/tutorials/basic_networking.md#communicating-between-containers-and-pods

and here (pods are like k8n pods), all containers belonging to the same pod share the same network. This means they all share the same IP, MAC and port mappings.
Now, what exactly is the correct setup depends on your topology/setup, for general info see this redhat blog post: https://www.redhat.com/en/blog/container-networking-podman.
But the fact that they share all these resources, means that no two containers in the pod can use the same port (see https://github.com/containers/podman/blob/main/troubleshooting.md#23-container-with-exposed-ports-wont-run-in-a-pod).

Reverse proxy caddy container

According to

You are putting the caddy-reverse proxy container into the same pod as nextcloud. This means that they cannot share the https port. Additionally, according to your pod definition, 443 is not exposed on the host, but I infer from the log thay you are querying port 8443.

I can see a number of possible issues here:

  1. the caddy file for the reverse proxy may not be listening on 8443, but, since it is in a pod with mapping 8443:443, it gets the request on 443 default and caddy knows to apply the nextcloud.ppflaum.ch directive - (HOPP SCHWIIZ!) - but that is proxied to https:8443, which on which the “nextcloud caddy server” is not listening according to

in that caddyfile.

  1. If they are on same IP then querying https://nextcloud.ppflaum.ch:8443 in a way above, where noone is listening on. (192.168.1.188:8443), You can try changing it to 443 and see what happens. Secondly,
  2. According to caddys [https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#transports], you need to (sometimes) override the Host header when proxying to HTTPS (see those docs for specs).

Sorry, answer is a bit all over the place now. I think to drill down on the issue I need more precise setup information. (In case you are using custom nextcloud images, I can also only help with podman stuff, as I have no experience running nextcloud regularly, only using the AIO setup)

PS: pls excuse the mix n match of linking, I was not allowed to provide more than 4 regular markdown links, so I had to enclose some of them in block comments

1 Like

Dear @sebaschi,

First of all thank you for your answer.

I will try to explain my problem in more detail and also the workaround I have used now.

The basic setup looks the following way:

I run a caddy server as a reverse proxy in a podman container on its own. I run a separate pod for the nextcloud instance, independent of the reverse proxy. It is done this way, as I want to use the reverse proxy also for other services. (I have found a work around) This is the Quadlet file for the reverse proxy,

[Container]
ContainerName=caddy-reverse-proxy
Image=docker.io/library/caddy:latest
AutoUpdate=registry
PublishPort=80:80
PublishPort=443:443
Volume=/mnt/storage/reverse-proxy/Caddyfile:/etc/caddy/Caddyfile:z
Volume=/mnt/storage/reverse-proxy/data:/data:z
Network=host 

[Service]
Restart=always

[Install]
WantedBy=default.target

and this is Caddyfile (my current version) for the reverse proxy

{
    email my@mail.com
}

nextcloud.ppflaum.ch {
    redir /.well-known/carddav /remote.php/dav/ 301
    redir /.well-known/caldav /remote.php/dav/ 301

    reverse_proxy http://127.0.0.1:8080
}

I don’t use the nextcloud AIO image, rather I use the nextcloud-fpm image without webserver. It runs in a pod with another caddy server as its webserver and with a mariadb image for the database. These are my quadlet files for nextcloud:

[Pod]
PodName=nextcloud
PublishPort=8080:80
PublishPort=8443:443

[Service]
Restart=always

[Install]
WantedBy=default.target
[Container]
ContainerName=mariadb-nextcloud
Pod=nextcloud.pod
Image=docker.io/library/mariadb:latest
AutoUpdate=registry
Volume=/mnt/storage/nextcloud/mariadb:/var/lib/mysql:z
Environment=MYSQL_DATABASE=mariadb-nextcloud
Environment=MYSQL_USER=username
Environment=MYSQL_PASSWORD=myPassword
Environment=MYSQL_ROOT_PASSWORD=myPassword
Exec=--transaction-isolation=READ-COMMITTED --log-bin=binlog --binlog-format=ROW --max_allowed_packet=256000000

[Unit]
After=nextcloud.pod
Requires=nextcloud.pod

[Service]
Restart=always

[Install]
WantedBy=default.target
[Container]
ContainerName=nextcloud-app
Pod=nextcloud.pod
Image=docker.io/library/nextcloud:fpm
AutoUpdate=registry
Environment=MYSQL_DATABASE=mariadb-nextcloud
Environment=MYSQL_USER=username
Environment=MYSQL_PASSWORD=myPassword
Environment=MYSQL_HOST=mariadb-nextcloud
Volume=/mnt/storage/nextcloud/nextcloud:/var/www/html:z
Volume=/mnt/storage/nextcloud/data:/var/www/html/data:z

[Unit]
After=nextcloud.pod
After=mariadb-nextcloud.container
Requires=nextcloud.pod
Requires=mariadb-nextcloud.container

[Service]
Restart=always

[Install]
WantedBy=default.target
[Container]
ContainerName=caddy-nextcloud
Pod=nextcloud.pod
Image=docker.io/library/caddy:latest
AutoUpdate=registry
Volume=/mnt/storage/nextcloud/nextcloud:/var/www/html:z
Volume=/mnt/storage/nextcloud/caddy/Caddyfile:/etc/caddy/Caddyfile:z

[Unit]
After=nextcloud.pod
After=mariadb-nextcloud.container
After=nextcloud-app.container
Requires=nextcloud.pod
Requires=mariadb-nextcloud.container
Requires=nextcloud-app.container

[Service]
Restart=always

[Install]
WantedBy=default.target

and this is the Caddyfile for the nextcloud caddy container

https://192.168.1.188:443 {
    tls internal

    root * /var/www/html
    file_server

    php_fastcgi nextcloud-app:9000 {
        root /var/www/html
        env front_controller_active true
    }
    encode gzip
    log {
        output file /data/nextcloud-access.log
    }
    header {
        Strict-Transport-Security "max-age=15768000;includeSubDomains;preload"
    }
    # .htaccess / data / config / ... shouldn't be accessible from outside
    @forbidden {
        path /.htaccess
        path /data/*
        path /config/*
        path /db_structure
        path /.xml
        path /README
        path /3rdparty/*
        path /lib/*
        path /templates/*
        path /occ
        path /console.php
    }
    respond @forbidden 404
    redir /.well-known/carddav /remote.php/dav 301
    redir /.well-known/caldav /remote.php/dav 301
}

I have found a workaround which is probably fine. I just forward all the traffic via http to the respective port, which works and should be fine I guess, since everything is running on the same server.
But what I want is to have the traffic encrypted for security reasons and maybe at some point I want to have bigger home lab and then should all the traffic between different servers by encrypted.

I think my problem is, that I don’t really understand how the https works. It works fine, if I forward everything over http but how could I do the same via https, such that the reverse proxy and the webserver (in this case for nextcloud) understand each other? I have tried playing around with these cert files but I don’t really understand what I am doing and it did not seam to work.
Also I don’t want to do to many things, which I don’t understand.
The reason why I have used Caddy in the first place, was because the syntax looks so simple that I hoped it makes it harder for me to do stupid shit. This might be the problem now, as caddy just did all of this for me, when it receives the traffic for the reverse proxy.

Some minimal example where one sends traffic over https to a different server might be enough to understand what is going on, but I have not managed to make it work.

Ps: Hopp Schwiiz!