Accessing Nextcloud & Issues with Caddy on an Arch-based LXC on Proxmox

1. Caddy version (caddy version):

community/caddy 2.3.0-1 [installed]
community/nextcloud 21.0.0-9 [installed]

2. How I run Caddy:

I am currently running Caddy from an Arch Linux-based LXC container that is hosted on Proxmox.

My current setup is modeled like the following:

(Node) Banshee

  • (LXC - sql-server) Centos7 based and hosts mariadb and phpmyadmin
    192.168.1.7:3306 - SQL
    192.168.1.7/phpmyadmin
  • (LXC - media-server) Arch Linux based and hosts Jellyfin, Openhab, and Nextcloud
    192.168.1.8:8080 - Openhab interface
    192.168.1.8:8096 - Jellyfin interface
    192.168.1.8/nextcloud Nextcloud interface
  • (LXC - web-server) Arch Linux based and this runs Caddy

a. System environment:

Proxmox System:
Linux web-server 5.4.103-1-pve #1 SMP PVE 5.4.103-1 (Sun, 07 Mar 2021 15:55:09 +0100) x86_64 GNU/Linux
The LXC Container used where Caddy is installed:
archlinux-base_20201116-1_amd64.tar.gz

b. Service file:

[Unit]
Description=Caddy web server
Documentation=https://caddyserver.com/docs/
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service
StartLimitIntervalSec=14400
StartLimitBurst=10

[Service]
User=caddy
Group=caddy
Environment=XDG_DATA_HOME=/var/lib
Environment=XDG_CONFIG_HOME=/etc
ExecStartPre=/usr/bin/caddy validate --config /etc/caddy/Caddyfile
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
ExecStopPost=/usr/bin/rm -f /run/caddy/admin.socket

# Do not allow the process to be restarted in a tight loop. If the
# process fails to start, something critical needs to be fixed.
Restart=on-abnormal

# Use graceful shutdown with a reasonable timeout
TimeoutStopSec=5s

LimitNOFILE=1048576
LimitNPROC=512

# Hardening options
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
DevicePolicy=closed
LockPersonality=true
MemoryAccounting=true
MemoryDenyWriteExecute=true
NoNewPrivileges=true
PrivateDevices=true
PrivateTmp=true
ProcSubset=pid
ProtectClock=true
ProtectControlGroups=true
ProtectHome=true
ProtectHostname=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectProc=invisible
ProtectSystem=strict
RemoveIPC=true
ReadWritePaths=/var/lib/caddy /var/log/caddy /run/caddy
RestrictNamespaces=true
RestrictRealtime=true
RestrictSUIDSGID=true

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile config:

(webconf) {
        encode gzip
        #    bind 192.168.1.9
}

www.site.net, site.net {
        root * /srv/http/site.net
        php_fastcgi * unix//var/run/php-fpm/php-fpm.sock
        file_server
        log {
                output file /var/log/site.access.log
        }

        import webconf
}

www.site2.com, site2.com {
        root * /srv/http/site2.com
        file_server
        import webconf
}

www.site3.com, site3.com {
        root * /srv/http/site3.com
        file_server
        import webconf
}

www.site4.com, site4.com {
        root * /srv/http/site4.com
        php_fastcgi unix//var/run/php-fpm/php-fpm.sock
        file_server
        import webconf
}

media.site.net {
        reverse_proxy 192.168.1.8:8096
        import webconf
}

cloud.site.net {
        reverse_proxy * 192.168.1.8/nextcloud 
        import webconf
}
[root@warthog ~]# 

3. The problem I’m having:

I can access Nextcloud from within my network (from 192.168.1.8/nextcloud) but when I try to access it from the reverse proxy, the page is blank.

cloud.site.net {
        reverse_proxy * 192.168.1.8/nextcloud 
        import webconf
}

I have made the applicable changes to the nextcloud configuration so that the trusted names are available. Regardless, it will not display any information.

/usr/share/webapps/nextcloud/config/config.php

...
  'trusted_domains' =>
  array (
    0 => '192.168.1.8',
    1 => 'cloud.site.net',
    2 => 'site.net',
  ),
  'dbtype' => 'mysql',
  'version' => '21.0.0.18',
  'overwrite.cli.url' => 'cloud.site.net',
  'dbname' => '[DBNAME]',
  'dbhost' => '192.168.1.7:3306',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => '[DBUSER]',
  'dbpassword' => '[DBPASSWORD]',
  'installed' => true,
  'overwritehost' => 'cloud.site.net',
...

All other sites are working fine! Jellyfin, the HTML-based websites and the Wordpress Sites are functional, are properly accessing the SQL server hosted on 192.168.1.7 and are accessible from outside of my local network. The only site that isn’t working properly is the Nextcloud setup.

When I go to https://cloud.site.net, redirection to HTTPS works properly and the certificate is displayed with the lock. It also shows that it’s verified by Let’s Encrypt.

4. Error messages and/or full log output:

Postman displays a 502 error. In the header information, it provides the time accessed and that the server is Caddy. The body is blank.

On the server hosting Nextcloud (192.168.1.8), the only complaints in the error logs are related to a failure in loading the PHP sodium library. I doubt this is the cause of the issue, but regardless, below is the snippet from the logs:

...
PHP Warning:  PHP Startup: Unable to load dynamic library 'sodium' (tried: /usr/lib/php/modules/sodium (/usr/lib/php/modules/sodium: cannot open shared object file:  No such file or directory), /usr/lib/php/modules/sodium.so (/usr/lib/php/modules/sodium.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
PHP Warning:  Version warning: Imagick was compiled against ImageMagick version 1802 but version 1803 is loaded. Imagick will run but may behave surprisingly in Unknown on line 0
[Sat Mar 13 02:00:02.415730 2021] [mpm_prefork:notice] [pid 1738] AH00163: Apache/2.4.46 (Unix) mpm-itk/2.4.7-04 PHP/8.0.3 configured -- resuming normal operations
[Sat Mar 13 02:00:02.415748 2021] [core:notice] [pid 1738] AH00094: Command line: '/usr/bin/httpd -D FOREGROUND'

The access logs on 192.168.1.8 are empty.

The logs on 192.168.1.9 (the LXC running Caddy) are also blank. However, when I look at the status of the service running, it does provide some information (confirms the 502 error for me):

[root@web-server ~]# systemctl status caddy > file
[root@web-server ~]# cat file
* caddy.service - Caddy web server
     Loaded: loaded (/usr/lib/systemd/system/caddy.service; enabled; vendor preset: disabled)
     Active: active (running) since Sat 2021-03-13 20:15:17 UTC; 1min 32s ago
       Docs: https://caddyserver.com/docs/
    Process: 251 ExecStartPre=/usr/bin/caddy validate --config /etc/caddy/Caddyfile (code=exited, status=0/SUCCESS)
   Main PID: 256 (caddy)
      Tasks: 6 (limit: 4915)
     Memory: 24.4M
     CGroup: /system.slice/caddy.service
             `-256 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

Mar 13 20:15:17 web-server caddy[256]: {"level":"info","ts":1615666517.3903573,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Mar 13 20:15:17 web-server caddy[256]: {"level":"info","ts":1615666517.390603,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Mar 13 20:15:17 web-server caddy[256]: {"level":"info","ts":1615666517.3906627,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Mar 13 20:15:17 web-server caddy[256]: {"level":"info","ts":1615666517.3914702,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["site2.com","www.site.net","media.site.net","cloud.site.net","site3.com","site.net","www.site3.com","git.site.net","www.site2.com","site4.com","www.site4.com"]}
Mar 13 20:15:17 web-server caddy[256]: {"level":"info","ts":1615666517.4055128,"msg":"autosaved config","file":"/var/lib/caddy/autosave.json"}
Mar 13 20:15:17 web-server caddy[256]: {"level":"info","ts":1615666517.4055657,"msg":"serving initial configuration"}
Mar 13 20:15:17 web-server caddy[256]: {"level":"info","ts":1615666517.4064057,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000243a40"}
Mar 13 20:15:17 web-server caddy[256]: {"level":"info","ts":1615666517.408479,"logger":"tls","msg":"cleaned up storage units"}
Mar 13 20:15:22 web-server caddy[256]: {"level":"error","ts":1615666522.0970001,"logger":"http.log.error","msg":"dial 192.168.1.8: unknown network 192.168.1.8","request":{"remote_addr":"192.168.1.1:58822","proto":"HTTP/2.0","method":"GET","host":"cloud.site.net","uri":"/","headers":{"Te":["trailers"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:86.0) Gecko/20100101 Firefox/86.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"],"Accept-Language":["en-US,en;q=0.5"],"Accept-Encoding":["gzip, deflate, br"],"Upgrade-Insecure-Requests":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"cloud.site.net"}},"duration":0.000089733,"status":502,"err_id":"ymdtmwf9k","err_trace":"reverseproxy.statusError (reverseproxy.go:783)"}
Mar 13 20:15:22 web-server caddy[256]: {"level":"error","ts":1615666522.0973647,"logger":"http.log.access","msg":"handled request","request":{"remote_addr":"192.168.1.1:58822","proto":"HTTP/2.0","method":"GET","host":"cloud.site.net","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:86.0) Gecko/20100101 Firefox/86.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"],"Accept-Language":["en-US,en;q=0.5"],"Accept-Encoding":["gzip, deflate, br"],"Upgrade-Insecure-Requests":["1"],"Te":["trailers"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"cloud.site.net"}},"common_log":"192.168.1.1 - - [13/Mar/2021:20:15:22 +0000] \"GET / HTTP/2.0\" 502 0","duration":0.000089733,"size":0,"status":502,"resp_headers":{"Server":["Caddy"]}}

5. What I already tried:

I have tried looking for other configurations where people have a similar setup, but none of their solutions have worked for me as most people are running their setup through pods or dockers.

There seems to be a lot of references to Caddy v1 solutions, but not Caddy v2.

6. Links to relevant resources:

NA (I can’t add more than 4 links as I am a new user, but I used the Google machine; “caddy nextcloud proxmox” provided good results)

The issue is that in Caddy v2, you cannot directly proxy with a path, because that would imply a path rewrite, and the reverse_proxy handler does not do this to separate concerns.

If you need a path rewrite, then you should do it explicitly before proxying. Like this:

rewrite * /nextcloud{uri}
reverse_proxy 192.168.1.8:80

This is mentioned in the docs:

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.