Well, I read your GitHub issue. At first, ownCloud’s CSRF check failed is a very common error. It may not caused by QUIC or php-fpm. Then, try
fastcgi / 127.0.0.1:9000 php
in your Caddyfile (make sure the php-fpm is listening on port 9000).
At last, check if the session.save_path is valid, and the permission/owner of that path is right, which is important to the webapps using sessions to implement login function.
(and another possibility: the implementation of QUIC protocol of Caddy is buggy currently)
Good luck.