Traefik as router, Caddy as server, on docker

Keauval · April 15, 2021, 7:35am

Hi everyone

I’m just onboarding on Caddy and I’d like to use it as webserver for php-fpm webapps.
I tried the nextcloud stack from @zilexa (here or there) and I was so surprised with the simplicity and the performances (I know he worked hard to set this) !

As webhoster my current working stack runs like a charm on docker with Traefik routing and giving ACME SSL certificate through http challenge :

[Docker container 1]

Traefik (router/reverse-proxy/certificates)

[Docker container 2]

Nginx (webserver)

PHP FPM

Webapp1 (MariaDB/Redis…)

[Docker container 3]

Nginx (webserver)

PHP FPM

Webapp2 (MariaDB/Redis…)

[…] etc.

I love my stack because I can add webapp without touching to Traefik settings (just adding labels to my webapps). Today I want to try Caddy and perhaps say goodbye to NGINX.

[Docker container 1]

Traefik (router/reverse-proxy/certificates)

[Docker container 2]

Caddy (webserver)

PHP FPM

Webapp1 (MariaDB/Redis…)

[Docker container 3]

Nginx (webserver)

PHP FPM

Webapp2 (MariaDB/Redis…)

[…] etc.

I already tried for bunch of hours to make a new docker-compose file with Traefik - Caddy - PhpFpm - Webapp following examples, but I think I’m missing something I did searches and didn’t find anything about a similar stack working.

I’m still asking myself :

Can Traefik still handle certificates for all my webapps ? Or perhaps be able to keep Traefik handle certificates only for Nginx containers, other one could be handle by Caddy ?
Can this configuration be as simple/dynamic as my actual one ? I think I’ll have to use lucaslorentz’s docker image to use docker’s labels and avoid Caddyfile like in the @zilexa nextcloud example…

Voilà !
Many thanks in advance for your help

francislavoie · April 15, 2021, 7:51am

With Caddy, you can drop Traefik as well. Like you said, with the GitHub - lucaslorentz/caddy-docker-proxy: Caddy as a reverse proxy for Docker plugin you can configure Caddy via docker labels similarly to Traefik, but get all the other advantages of Caddy as well (fastcgi, file server, better automatic HTTPS, etc)

But if you want to play it conservatively, you can just serve on port 80 from your Caddy container, no TLS, and proxy Traefik to Caddy. For example:

:80 {
	root * /var/www/html
	php_fastcgi php-fpm:9000
	file_server
}

Keauval · April 15, 2021, 7:57am

Oww really ?
And How will I serve all my caddy-webapps from my server if they’re all using port :80 ?
Sorry perhaps I’m missing something…

francislavoie · April 15, 2021, 8:00am

What I meant is, either use Caddy to terminate TLS and serve everything, OR, keep Traefik in front and have it terminate TLS (as it currently does for you as I understand) and just use Caddy in between Traefik and Nextcloud (or whatever), without TLS (because if you wanted to enable TLS, it would require trust between Traefik and Caddy, and Caddy can’t solve ACME challenges if Traefik is in front of it)

Keauval · April 15, 2021, 8:05am

That’s perfect !
But :

will I still be able to keep Traefik terminate TLS for some containers (those I’ll keep using with NGINX, for testing/comparing/transition needs) ?
Someone can help me setting that ?

francislavoie · April 15, 2021, 8:11am

Only one thing can terminate TLS. Like I said, go all-in on Caddy and replace Traefik with it, or keep using Traefik to terminate TLS (for everything). Neither of them have the capability (at least by default) to proxy without terminating TLS (i.e. pass through the raw TCP bytes still encrypted). Caddy can do this with the GitHub - mholt/caddy-l4: Layer 4 (TCP/UDP) app for Caddy plugin, but that’s way overkill and advanced for what you need.

If you want to keep Traefik in front, and just use Caddy for fastcgi + file_server, then you’ll have a Caddyfile like I wrote above:

francislavoie:

But if you want to play it conservatively, you can just serve on port 80 from your Caddy container, no TLS, and proxy Traefik to Caddy. For example:
:80 {
	root * /var/www/html
	php_fastcgi php-fpm:9000
	file_server
}

I can’t be any more specific than that, because you weren’t specific about your setup.

Keauval · April 15, 2021, 10:19am

Allright many thanks @francislavoie
I’ll try again to build a new docker-compose file and come back to you with my setup !

Keauval · April 15, 2021, 10:57am

So ! I tried a new setup and get a HTML 502 ERROR
It looks starting not so bad : get my ssl on domain, not sure to have it on webapp1… but sure i’m getting near !

My docker-compose :
(For this first try I didn’t separate my Traefik in another dockercompose… and still using Traefik V1 )

version: '3.7'
networks:
    traefik-network:
services:
    traefik:
        image: traefik:v1.7.12-alpine
        container_name: traefik
        command: --api --docker --configFile=traefik.toml
        ports:
            - 80:80
            - 443:443
            - 8080:8080
        restart: always
        labels:
            - traefik.enabled=true
            - traefik.backend=traefik
            - traefik.frontend.rule=Host:$DOMAIN
            - traefik.port=8080
            - traefik.frontend.entryPoints=http,https
        networks:
            traefik-network:
        volumes:
            - /var/run/docker.sock:/var/run/docker.sock
            - ./traefik.toml:/traefik.toml
            - ./acme.json:/acme.json        

    caddy:
        container_name: caddy
        image: lucaslorentz/caddy-docker-proxy:ci-alpine
        restart: always
        volumes:
            - /var/run/docker.sock:/var/run/docker.sock
            - ./data/webapp1:/var/www/html
        labels:
            traefik.backend: caddywebbapp1
            traefik.frontend.rule: Host:webap1.$DOMAIN
            traefik.docker.network: traefik-network           
            caddy: :80
            caddy.root: "* /var/www/html"
            caddy.php_fastcgi: php-fpm:9000
            caddy.file_server: ""
        networks:
            traefik-network:
        depends_on:
            - webapp1

    webapp1:
        image: wordpress:5.7-php7.4-fpm-alpine
        container_name: webapp1
        restart: unless-stopped
        environment:
            WORDPRESS_DB_HOST: mariadb
            WORDPRESS_DB_USER: user1
            WORDPRESS_DB_PASSWORD: pass1
            WORDPRESS_DB_NAME: db
            WORDPRESS_CONFIG_EXTRA: |
                define('WP_HOME', 'http://webap1.$DOMAIN');
                define('WP_SITEURL', 'http://webap1.$DOMAIN');
        labels:
            - traefik.enable=false
        networks:
            traefik-network:
        volumes:
            - ./data/webapp1:/var/www/html
            - ./php:/usr/local/etc/php/conf.d/php.ini
        depends_on:
            - mariadb

    mariadb:
        image: mariadb:10.2.26
        container_name: mariadb
        command: --max-allowed-packet=128MB
        restart: unless-stopped
        volumes:
            - dbdata:/var/lib/mysql
        labels:
            - traefik.enable=false
        environment:
            MYSQL_ROOT_PASSWORD: pass2
            MYSQL_USER: user1
            MYSQL_PASSWORD: pass1
            MYSQL_DATABASE: db
        networks:
            traefik-network:

volumes:
    dbdata:

My Caddy’s log error :

2021/04/15 10:36:14 [INFO] Running caddy proxy controller
2021/04/15 10:36:14 [INFO] CaddyfilePath:
2021/04/15 10:36:14 [INFO] LabelPrefix: caddy
2021/04/15 10:36:14 [INFO] PollingInterval: 30s
2021/04/15 10:36:14 [INFO] ProcessCaddyfile: true
2021/04/15 10:36:14 [INFO] ProxyServiceTasks: true
2021/04/15 10:36:14 [INFO] IngressNetworks: []
2021/04/15 10:36:14 [INFO] Caddy ContainerID: 372f6f71d43e78e578d37410264581377408bd4d56c1ef6241118ed858cd23ff
2021/04/15 10:36:14 [INFO] Connecting to docker events
2021/04/15 10:36:14 [INFO] IngressNetworksMap: map[ebb4df3cc629afc62a7aee2b1e72c9604f79110fc5660e122b910f449401fe54:true]
2021/04/15 10:36:14 [INFO] Swarm is available: false
2021/04/15 10:36:14 [INFO] Skipping default Caddyfile because no path is set
[INFO] Skipping configs because swarm is not available
[INFO] Skipping services because swarm is not available
2021/04/15 10:36:14 [INFO] New Caddyfile:
:80 {
        file_server
        php_fastcgi php-fpm:9000
        root * /var/www/html
}
2021/04/15 10:36:14 [INFO] New Config JSON:
{"apps":{"http":{"servers":{"srv0":{"listen":[":80"],"routes":[{"handle":[{"handler":"vars","root":"/var/www/html"}]},{"match":[{"file":{"try_files":["{http.request.uri.path}/index.php"]},"not":[{"path":["*/"]}]}],"handle":[{"handler":"static_response","headers":{"Location":["{http.request.uri.path}/"]},"status_code":308}]},{"match":[{"file":{"try_files":["{http.request.uri.path}","{http.request.uri.path}/index.php","index.php"],"split_path":[".php"]}}],"handle":[{"handler":"rewrite","uri":"{http.matchers.file.relative}"}]},{"match":[{"path":["*.php"]}],"handle":[{"handler":"reverse_proxy","transport":{"protocol":"fastcgi","split_path":[".php"]},"upstreams":[{"dial":"php-fpm:9000"}]}]},{"handle":[{"handler":"file_server","hide":["./Caddyfile"]}]}]}}}}}
2021/04/15 10:36:14 [INFO] Sending configuration to localhost
{"level":"info","ts":1618482974.8950067,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_addr":"127.0.0.1:36524","headers":{"Accept-Encoding":["gzip"],"Content-Length":["802"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
{"level":"info","ts":1618482974.8953362,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1618482974.8954828,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003c7260"}
{"level":"info","ts":1618482974.895564,"logger":"http","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv0","http_port":80}
{"level":"info","ts":1618482974.8958702,"logger":"tls","msg":"cleaned up storage units"}
{"level":"info","ts":1618482974.895984,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1618482974.895991,"logger":"admin.api","msg":"load complete"}
2021/04/15 10:36:14 [INFO] Successfully configured localhost
{"level":"info","ts":1618482975.3956406,"logger":"admin","msg":"stopped previous server"}
{"level":"error","ts":1618482996.719878,"logger":"http.log.error","msg":"dialing backend: dial tcp: lookup php-fpm on 127.0.0.11:53: server misbehaving","request":{"remote_addr":"172.31.0.3:51324","proto":"HTTP/1.1","method":"GET","host":"webap1.***MYDOMAIN***","uri":"/","headers":{"X-Forwarded-Host":["webap1.***MYDOMAIN***"],"X-Forwarded-Proto":["https"],"Accept-Language":["fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-User":["?1"],"X-Forwarded-Port":["443"],"X-Real-Ip":["89.86.77.218"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.142 Safari/537.36"],"Sec-Fetch-Mode":["navigate"],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"X-Forwarded-Server":["ecd836bc1e01"],"Sec-Fetch-Site":["none"],"X-Forwarded-For":["89.86.77.218"],"Accept-Encoding":["gzip, deflate, br"],"Cookie":["_ga=GA1.2.1560336775.1602583971; tk_or=%22%22; tk_lr=%22%22"]}},"duration":0.001310205,"status":502,"err_id":"t6hp0kiat","err_trace":"reverseproxy.statusError (reverseproxy.go:783)"}
{"level":"error","ts":1618482998.497844,"logger":"http.log.error","msg":"dialing backend: dial tcp: lookup php-fpm on 127.0.0.11:53: server misbehaving","request":{"remote_addr":"172.31.0.3:51324","proto":"HTTP/1.1","method":"GET","host":"webap1.***MYDOMAIN***","uri":"/","headers":{"Accept-Language":["fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7"],"X-Forwarded-Port":["443"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.142 Safari/537.36"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Dest":["document"],"X-Forwarded-Host":["webap1.***MYDOMAIN***"],"X-Real-Ip":["89.86.77.218"],"Upgrade-Insecure-Requests":["1"],"X-Forwarded-Server":["ecd836bc1e01"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Cache-Control":["max-age=0"],"Cookie":["_ga=GA1.2.1560336775.1602583971; tk_or=%22%22; tk_lr=%22%22"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Site":["cross-site"],"X-Forwarded-For":["89.86.77.218"]}},"duration":0.001172448,"status":502,"err_id":"j00aedkg0","err_trace":"reverseproxy.statusError (reverseproxy.go:783)"}
{"level":"error","ts":1618483041.5941648,"logger":"http.log.error","msg":"dialing backend: dial tcp: lookup php-fpm on 127.0.0.11:53: server misbehaving","request":{"remote_addr":"172.31.0.3:51324","proto":"HTTP/1.1","method":"GET","host":"webap1.***MYDOMAIN***","uri":"/","headers":{"Cache-Control":["max-age=0"],"Sec-Fetch-Dest":["document"],"X-Forwarded-Port":["443"],"X-Forwarded-Proto":["https"],"X-Forwarded-Server":["ecd836bc1e01"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.142 Safari/537.36"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7"],"Cookie":["_ga=GA1.2.1560336775.1602583971; tk_or=%22%22; tk_lr=%22%22"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["cross-site"],"Sec-Fetch-User":["?1"],"X-Forwarded-For":["89.86.77.218"],"X-Forwarded-Host":["webap1.***MYDOMAIN***"],"X-Real-Ip":["89.86.77.218"]}},"duration":0.001553249,"status":502,"err_id":"8ak597gbc","err_trace":"reverseproxy.statusError (reverseproxy.go:783)"}
{"level":"error","ts":1618483042.2238533,"logger":"http.log.error","msg":"dialing backend: dial tcp: lookup php-fpm on 127.0.0.11:53: server misbehaving","request":{"remote_addr":"172.31.0.3:51324","proto":"HTTP/1.1","method":"GET","host":"webap1.***MYDOMAIN***","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.142 Safari/537.36"],"Upgrade-Insecure-Requests":["1"],"X-Forwarded-Server":["ecd836bc1e01"],"Pragma":["no-cache"],"Sec-Fetch-Site":["cross-site"],"X-Forwarded-Host":["webap1.***MYDOMAIN***"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"X-Forwarded-For":["89.86.77.218"],"X-Forwarded-Proto":["https"],"Accept-Language":["fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7"],"Cache-Control":["no-cache"],"Cookie":["_ga=GA1.2.1560336775.1602583971; tk_or=%22%22; tk_lr=%22%22"],"X-Forwarded-Port":["443"],"X-Real-Ip":["89.86.77.218"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"]}},"duration":0.002268641,"status":502,"err_id":"8z47x95r8","err_trace":"reverseproxy.statusError (reverseproxy.go:783)"}

francislavoie · April 15, 2021, 3:50pm

Your container isn’t named php-fpm, it’s named webapp, so you should do:

caddy.php_fastcgi: webapp1:9000

Also I think you’re missing all the - on those labels lines maybe? Not sure how strict yaml is about that.

Keauval · April 16, 2021, 8:17am

Woow !
It works
SOLVED

Yes… yaml is really tolerent

Thank a lot for the help.
Caddy config looks really more simple than NGINX, really interesting.

I’ll try to set my complete set and then try come comparaison/perf tests.
Are you interrested if I let this topic open and give you update ?

Cheers.

francislavoie · April 16, 2021, 2:45pm

Yeah, that’s fine. I’m sure others who land on this thread in the future will be interested in what you ended up with.

system · May 15, 2021, 7:36am

This topic was automatically closed after 30 days. New replies are no longer allowed.