I got it running. For future people you need to install php-fpm (ver 7.2+) and modify the /etc/php/7.2/fpm/pool.d/www.conf and add listen=9000 so it will listen on port 9000.
The caddy file is
:8082 {
root * /var/www/domain
file_server
php_fastcgi localhost:9000
}
You can access this on localhost:8082
Thanks for the help!