Nextcloud High Performance Backend (notify_push)

1. Caddy version (2.5.1):

2. How I run Caddy:

a. System environment:

Almalinux 9
Native Binary
Systemd Service
Nothing in front

b. Command:

c. Service/unit/compose file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

[Unit]
Description = Push daemon for Nextcloud clients
After=redis.service
Wants=redis.service

[Service]
Environment=PORT=7867
Environment=NEXTCLOUD_URL=https://domain
ExecStart=/var/www/html/nextcloud/apps/notify_push/bin/x86_64/notify_push /var/www/html/nextcloud/config/config.php

User=nextcloud
Group=nextcloud

Restart=always

[Install]
WantedBy = multi-user.target

d. My complete Caddyfile or JSON config:

domain {
        file_server
        root * /var/www/html/nextcloud
        php_fastcgi unix//run/php-fpm/nextcloud.sock {
                env modHeadersAvailable true
                env front_controller_active true
                env HTTPS on
        }
        redir /.well-known/carddav /remote.php/dav 301
        redir /.well-known/caldav /remote.php/dav 301
        # .htaccess / data / config / ... shouldn't be accessible from outside
        @forbidden {
                path    /.htaccess
                path    /.xml
                path    /3rdparty/*
                path    /config/*
                path    /data/*
                path    /db_structure
                path    /README
                path    /lib/*
                path    /templates/*
                path    /occ
                path    /console.php
                path    /test.php
        }
        respond @forbidden "Access denied" 403 {
                close
        }
        handle_path /push/* {
               reverse_proxy http://127.0.0.1:7867
        }
}

3. The problem I’m having:

Nextcloud High Performance Backend (HBP/notify_push) does not work after Caddy version 2.4.6

4. Error messages and/or full log output:

# occ notify_push:self-test
✓ redis is configured
✓ push server is receiving redis messages
✓ push server can load mount info from database
✓ push server can connect to the Nextcloud server
🗴 push server is not a trusted proxy, please add 'external_ip' to the list of trusted proxies or configure any existing reverse proxy to forward the 'x-forwarded-for' send by the push server.
  See https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/reverse_proxy_configuration.html#defining-trusted-proxies for how to set trusted proxies.
  The following trusted proxies are currently configured: "127.0.0.1", "::1", "external_ipv6", "external_ipv4"
  The following x-forwarded-for header was received by Nextcloud: external_ipv6
    from the following remote: external_ipv6

  If you're having issues getting the trusted proxy setup working, you can try bypassing any existing reverse proxy
  in your setup by setting the `NEXTCLOUD_URL` environment variable to point directly to the internal Nextcloud webserver url
  (You will still need the ip address of the push server added as trusted proxy)

5. What I already tried:

6. Links to relevant resources:

That doesn’t seem like an issue with Caddy. Nextcloud is giving you a pretty clear error message, you need to configure trusted proxies.

1 Like

It is configured, like I said, it worked with Caddy 2.4.6 and before.

I have no reason to believe this is a bug. If it used to work, then it shouldn’t have.

You didn’t completely fill out the help topic template, so we don’t have enough information to go on here. And I think you redacted part of the logs, which means we can’t really do much with that information.

With Caddy 2.4.6:

# occ notify_push:self-test
✓ redis is configured
✓ push server is receiving redis messages
✓ push server can load mount info from database
✓ push server can connect to the Nextcloud server
✓ push server is a trusted proxy
✓ push server is running the same version as the app

So it has to be Caddy;

I understand, but this is not a bug.

Please fully fill out the help topic template. You haven’t told us enough information about your setup for us to help diagnose the issue.

Do you have some other server/proxy in front of Caddy?

1 Like

No, there is nothing in front of it, just Caddy.

Is this section of my config correct or is there something missing?

handle_path /push/* {
               reverse_proxy http://127.0.0.1:7867
}

I have tried the following without success:

handle_path /push/* {
               reverse_proxy http://127.0.0.1:7867 {
                               trusted_proxies private_range external_ipv4 external_ipv6
        }
}

This config option is only necessary if you have something in front of Caddy, like another proxy (or Cloudflare, which is a proxy), which is why I asked.

1 Like

So what should I do now?

I also read the error message as a wrong configuration of nextcloud, but its hard to tell because you didn’t show your nextcloud config/config.php.
Reverse proxys have to be configured in config/config.php, see the explanation Reverse proxy — Nextcloud latest Administration Manual latest documentation and the example at the bottom Reverse proxy — Nextcloud latest Administration Manual latest documentation :
'trusted_proxies' => ['10.0.0.1'],
If you are in a docker environment it would be something like:
'trusted_proxies' => ['caddy'], (if ‘caddy’ is the name of the caddy docker container)

1 Like

My Config File is also configured correctly

'trusted_proxies' =>
 array (
    0 => '127.0.0.1',
    1 => '::1',
    2 => 'external_ipv6',
    3 => 'external_ipv4',
  )

Also I really don’t understand my current config works with Caddy 2.4.6 and nginx without any changes made but not with Caddy 2.5+.

With nginx and Caddy 2.4.6 notify_push/HBP reports this in the debug log
got remote 1.2.3.4 when trying to set remote 1.2.3.4
With Caddy 2.5+
got remote external_ipv6 when trying to set remote 1.2.3.4

Can you be more specific? external_ipv4/external_ipv6 doesn’t make any sense :thinking:

Francis and I aren’t sure how to help you more without more details. If this is a bug in Caddy, then we need specific instructions to reproduce the problem as minimally as possible. If you want help, then we at least need your full config and logs, without redactions, according to our forum rules. All we can do is guess otherwise, and that will waste your time and ours. :frowning:

1 Like

Sure, I have a VPS with a fixed public IPv4 => external_ipv4 and a fixed public IPv6 => external_ipv6 and a TLD domain => domain. However, for security reasons I do not want to share them, please understand.

I don’t think we can help you then, sorry.

Public domains and IPs are not secret nor do they have anything to do with security.

The notify_push sparked my interest after chatting a bit about this thread with @francislavoie in the internal Slack.
I’ve read about the announcement of it, but couldn’t be bothered to set it up on my employer’s Nextcloud. Plus, I don’t use Nextcloud personally, so I really hadn’t had a reason to play with it.
Until now :sweat_smile:

Anyhow…
While you didn’t provide that many insights on your setup @therealresonix, I managed to set up a little reproduction environment locally with docker-compose.
I tried to keep it similar to your docker-less setup (binding everything on the host directly via --net=host and connecting via localhost):

My directory looks like that:

❯ tree
.
├── Caddyfile
└── docker-compose.yml

0 directories, 2 files

Note: I removed a bunch of stuff from your original Caddyfile. Just limit the scope of things a tiny little bit. Feel free, or rather, I encourage you to re-add those lines I removed! I also used a reverse_proxy to proxy to an apache2 in docker. But more on that later :innocent:

{
	auto_https off
}

# Using http://, because I couldn't be bothered 
# to find out how to tell `notify_push` to skip
# the certificate validation
#
# 127.0.0.1.nip.io will resolve to 127.0.0.1
# and is used so `notify_push` won't yell at us for using
#  1. localhost
#  2. "an invalid dns name" (IP instead of hostname)
#
# Port :8080, because the docker-compose file uses
# `network_mode: host`, so the nextcloud apache container
# will bind to `:80` on the host and thus is already used

http://127.0.0.1.nip.io:8080 {
	# nextcloud
	reverse_proxy localhost:80 {
		trusted_proxies private_ranges
	}

	# notify_push
	# Could also be hosted on a completely seperate domain like 
	# `http://push.127.0.0.1.nip.io:8080`
	handle_path /push/* {
		reverse_proxy localhost:7867
	}
}
services:
  nextcloud:
    image: nextcloud
    environment:
      REDIS_HOST: localhost ## redis is required for `notify_push` to work
      TRUSTED_PROXIES: "127.0.0.1 ::1" ## caddy is connecting via `127.0.0.1` or `::1`
      OVERWRITEPROTOCOL: http ## I couldn't be bothered to find out how to tell `notify_push` to skip the certificate validation
    volumes:
      - ./nextcloud:/var/www/html
    network_mode: host ## binds to :80

  notify_push:
    image: nextcloud
    environment:
      NEXTCLOUD_URL: http://127.0.0.1.nip.io:8080 ## fqdn for clients to connect
    entrypoint:
      - /var/www/html/custom_apps/notify_push/bin/x86_64/notify_push
      - /var/www/html/config/config.php
      - --log-level
      - debug
    volumes:
      - ./nextcloud:/var/www/html
    network_mode: host ## binds to :7867

  redis:
    image: redis:alpine
    network_mode: host ## binds to :6379

The test setup requires the following commands:

## won't be able to start notify_push, but that's expected and okay to ignore
docker-compose up -d 

## open http://127.0.0.1.nip.io:8080 in browser
## and finish setup (only user and password required)

## install notify_push 
docker-compose exec -u www-data nextcloud php occ app:install notify_push

## setup notify_push and run self-check
docker-compose exec -u www-data nextcloud bash -c "php occ notify_push:setup http://127.0.0.1.nip.io:8080/push" 

Now when you open your browser or refresh the Nextcloud page, you should be able to see a websocket connection after a few seconds.
=> Everything works again :upside_down_face:


Now to the why:
caddy v2.5.0 no longer blindly trusts X-Forwarded-For headers from any client.

The release notes under Notable state:

  • Reverse proxy: Incoming X-Forwarded-* headers will no longer be automatically trusted, to prevent spoofing. Now, trusted_proxies must be configured to specify a list of downstream proxies which are trusted to have sent good values. You only need to configure trusted proxies if Caddy is not the first server being connected to. For example, if you have Cloudflare in front of Caddy, then you should configure this with Cloudflare’s list of IP ranges.

The php occ notify_push:setup self-check sets and tests the X-Forwarded-For header, because uses it to preserve the client IP connecting to it, after the setup is done.

Basically:
Client/Browser (Client IP: 10.0.0.100) → Caddy (Client IP: 10.0.0.100) → notify_push (X-Forwarded-For: 10.0.0.100) → Caddy (Client IP: 127.0.0.1, X-Forwarded-For: 10.0.0.100) → Nextcloud (X-Forwarded-For: 127.0.0.1)

So to wrap it up:
All you need to do is extend your fastcgi_php directive:

 php_fastcgi unix//run/php-fpm/nextcloud.sock {
     env modHeadersAvailable true
     env front_controller_active true
     env HTTPS on
+    trusted_proxies private_ranges
 }

or, since you only use localhost to connect to your upstreams, you could also just use

 php_fastcgi unix//run/php-fpm/nextcloud.sock {
     env modHeadersAvailable true
     env front_controller_active true
     env HTTPS on
+    trusted_proxies 127.0.0.1/8 ::1
 }

php_fastcgi (Caddyfile directive) — Caddy Documentation supports all reverse_proxy (Caddyfile directive) — Caddy Documentation subdirectives, where you can find details about the trusted_proxies option.


PS: I don’t think you need external_ipv6 or external_ipv4 in your trusted_proxies array. Though I have to say I am unsure about the 127.0.0.1 and ::1, because theoretically Caddy only ever connects to it via unix socket, not over IP. Might depend on how Nextcloud parses that :woman_shrugging:
Edit: 127.0.0.1 and ::1 needs to stay because of reverse_proxy localhost:7867. I forgot about that lol

1 Like

This line

trusted_proxies private_ranges external_ipv4 external_ipv6

solved the problem, the finished result looks like this

domain {
        file_server
        root * /var/www/html/nextcloud
        php_fastcgi unix//run/php-fpm/nextcloud.sock {
                env modHeadersAvailable true
                env front_controller_active true
                env HTTPS on
                trusted_proxies private_ranges external_ipv4 external_ipv6
}
        redir /.well-known/carddav /remote.php/dav 301
        redir /.well-known/caldav /remote.php/dav 301
        # .htaccess / data / config / ... shouldn't be accessible from outside
        @forbidden {
                path    /.htaccess
                path    /.xml
                path    /3rdparty/*
                path    /config/*
                path    /data/*
                path    /db_structure
                path    /README
                path    /lib/*
                path    /templates/*
                path    /occ
                path    /console.php
                path    /test.php
        }
        respond @forbidden "Access denied" 403 {
                close
        }
        handle_path /push/* {
               reverse_proxy http://127.0.0.1:7867
        }
}
1 Like

A post was split to a new topic: Nextcloud notify_push: Connection timed out

This topic was automatically closed after 30 days. New replies are no longer allowed.