Caddy with cloudflared not routing traffic

alexandzors · November 8, 2021, 5:47pm

1. Caddy version (`caddy version`):

v2.4.3 h1:Y1FaV2N4WO3rBqxSYA8UZsZTQdN+PwcoOcAiZTM8C0I=

2. How I run Caddy:

a. System environment:

Ubuntu Server 20.04.3 LTS
Docker CE - 20.10.9

b. Command:

docker-compose up -d

c. Service/unit/compose file:

docker-compose.yml

version: '3.6'
services:
  caddy:
    restart: always
    logging:
      driver: "json-file"
      options:
        max-size: "500k"
        max-file: "1"
    network_mode: "bridge"
    image: alexandzors/caddy
    ports:
        - 80:80
        - 443:443
        - 10000:10000
    volumes:
        - /opt/docker/caddy/Caddyfile:/etc/caddy/Caddyfile:ro
        - /opt/docker/caddy:/data
        - /opt/docker/caddy:/config
        - /opt/docker/caddy/auth/local/users.json:/etc/caddy/auth/local/users.json
        - /media/plex:/srv

d. My complete Caddyfile or JSON config:

{
        email <email>
#        acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
#        default_sni centaurus
        admin localhost:2019
        #    order filter after encode
        debug
}
(header) {
        header {
                Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
                X-Xss-Protection "1; mode=block"
                X-Content-Type-Options "nosniff"
                X-Frame-Options "DENY"
                Content-Security-Policy "upgrade-insecure-requests"
                Referrer-Policy "strict-origin-when-cross-origin"
                Cache-Control "public, max-age=15, must-revalidate"
                Feature-Policy "accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'self'; camera 'none'; encrypted-media 'none'; fullscreen 'self'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; midi 'none'; payment 'none'; picture-in-picture *; speaker 'none'; sync-xhr 'none'; usb 'none'; vr 'none'"
        }
}
(tls) {
        tls {
                dns cloudflare <key>
        }
}
alexslab.dev:10000 {
        #        import header
        import tls
        #       respond * 503
        log {
                output file /config/logs/lab.log {
                        roll_size 50mb
                        roll_keep 2
                        roll_keep_for 24h
                }
        }
        root /* /config
        file_server {
                index index.html
        }
        route /sonarr* {
                reverse_proxy 192.168.9.3:8989
        }
        route /radarr* {
                reverse_proxy 192.168.9.3:7878
        }
        route /4kradarr* {
                reverse_proxy 192.168.9.3:7879
        }
        route /tautulli* {
                reverse_proxy 192.168.9.3:8181
        }
        #        route /lidarr* {}
        route /bazarr* {
                reverse_proxy 192.168.9.3:6767
        }
}

3. The problem I’m having:

I am trying to get Caddy to respond to requests from cloudflared over port 10000. I moved and my new ISP blocks inbound 80 and 443…

cloudflared config.yml:

tunnel: <id>
credentials-file: /home/alexander/.cloudflared/<id>.json
logfile: cloudflared.log
ingress:
  - service: https://127.0.0.1:10000

CF DNS Config

capella was the original CNAME created when running:
cloudflared tunnel route dns <uid> <name>

4. Error messages and/or full log output:

journalctl -u cloudflared
Nov 08 17:16:53 capella cloudflared[656477]: 2021-11-08T17:16:53Z ERR error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: remote error: tls: internal error"

only caddy log error:
{"level":"debug","ts":1636391925.463325,"logger":"http.stdlib","msg":"http: TLS handshake error from 172.17.0.1:56720: no certificate available for '172.17.0.2'"}

cloudflared tunnel info

NAME:     capella
ID:       <id>
CREATED:  2021-11-08 16:29:40.631129 +0000 UTC

CONNECTOR ID                         CREATED              ARCHITECTURE VERSION   ORIGIN IP    EDGE
<id>                                 2021-11-08T17:16:41Z linux_amd64  2021.11.0 24.115.20.47 2xIAD, 2xORD

curl -kIL --resolve alexslab.dev:127.0.0.1 https://alexslab.dev/sonarr:10000

HTTP/2 502
date: Mon, 08 Nov 2021 17:40:38 GMT
cf-cache-status: DYNAMIC
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=nMYrMx2q8%2FDa7ixIzlOz3cMD93ouAy%2B%2BGwLkXe%2FH6Sdlgr2QjPT2SLF4YO1MZlr39AvAq0N75ifrWD2dRhLlpI%2FgA6xQoht0YjdCbK4FSkF%2Fomk6uxpL%2BFB4hiwhTuA%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 6ab0966e2a1703fc-ORD
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400

curl -kIL --resolve alexslab.dev:127.0.0.1 https://alexslab.dev:10000

HTTP/2 200
accept-ranges: bytes
content-type: text/html; charset=utf-8
etag: "qxee8h14b"
last-modified: Fri, 06 Aug 2021 03:12:17 GMT
server: Caddy
content-length: 1451
date: Mon, 08 Nov 2021 23:11:01 GMT

5. What I already tried:

Forcing new certificates from LE by removing the ./caddy data dir.
Switching between:

url: https://127.0.0.1:10000

and:

ingress:
  - service: https://127.0.0.1:10000

in config.yml

6. Links to relevant resources:

Cloudflare for Teams documentation

Caddy(docker) reverse proxy with dns-01 on Cloudflare - Help - Caddy Community

Whitestrake · November 8, 2021, 11:31pm

G’day @alexandzors,

From this I can infer that cloudflared is not using Server Name Indication (SNI).

We can see the immediate result of this as the “client” is 172.17.0.1 (the Docker network proxy) and the website requested is 172.17.0.2 (most likely the IP address of the Caddy container in the Docker network, and what Caddy will use as the fallback “requested host” when no SNI is present).

That means that Caddy has no idea which website the request is coming in for over HTTPS and it is therefore impossible to select an appropriate certificate to negotiate TLS with.

I did some quick searching and didn’t turn up anything straight away with respect to Argo Tunnels / cloudflared and SNI (or lack thereof).

A workaround in the interim is to un-encrypt communication between cloudflared and Caddy. My recommendation is to put cloudflared in its own container (there exists an official Docker image for the software), inside the Docker network, and then close off host access to the unencrypted Caddy port entirely.

alexandzors · November 8, 2021, 11:37pm

Alright I’ll give a go deploying cloudflared in docker then instead of on the host.

alexandzors · November 9, 2021, 4:32am

Not sure if I am doing this wrong but I can’t get cloudflared to even pass traffic inside a docker network to Caddy.

cloudflared docker-compose.yml

version: '3.6'
services:
  cloudflared:
    restart: always
    logging:
      driver: "json-file"
      options:
        max-size: "500k"
        max-file: "1"
    network_mode: bridge
    image: cloudflare/cloudflared:2021.11.0
    command: 'tunnel --config /etc/cloudflared/config.yml run'
    volumes:
        - /opt/docker/cloudflared:/etc/cloudflared
    user: root

cloudflared config.yml

tunnel: <id>
credentials-file: /etc/cloudflared/<id>.json
logfile: /etc/cloudflared/cloudflared.log
origincert: /etc/cloudflared/cert.pem
ingress:
  - service: https://caddy_caddy_1:10000
connectTimeout: 10s

I have also tried:

ingress:
  - service http://caddy_caddy_1

ingress:
  - service: https://caddy_caddy_1

ingress
  - service: http://caddy_caddy_1:10000

updating Caddyfile accordingly btw

Service still responds with 502 for my host and nothing in the logs from either cloudflared or Caddy. Not sure if I am missing something here.

Edit: cloudflared and caddy are in a docker network together called caddy. As a test I just ran docker network create caddy and then connected both containers to it and restarted them.

Whitestrake · November 9, 2021, 4:48am

They need to be inside the same Docker network.

The easiest way to achieve that is to put them in the same Compose file and let Compose create a default network with the two containers included. (This has the added benefit of being able to refer to Caddy directly by it’s service name; since your Caddy service is called caddy you could then refer to it in the cloudflared config as http://caddy:10000, or whichever port you choose.)

Further to that, you need to:

which is to say that you’ll need to configure Caddy to serve HTTP (without HTTP->S redirection) and configure cloudflared to connect over HTTP on whichever port you’ve configured Caddy for this purpose.

alexandzors · November 13, 2021, 12:01am

So I’ve been playing with this setup for a few days now and I still for the life of me cannot get it to work. I tried even setting up a direct install of Caddy and cloudflared on a linode VM to test it and its still not able to properly pass traffic to Caddy.

~~At this point I’m at a loss. So probably just going to go the wireguard route to tunnel traffic into my caddy install locally from linode.~~

Okay so I tried it again and it seems to now be working. I scraped the entire port stuff and just left my domain as http://alexslab.dev {} and in the config.yml for cloudflared I set the url: parameter to url: http://caddy.

For some reason Caddy was not liking the different port settings. Setting it up as you suggested and reverting to port 80 on the Caddy side got it working.

Testing docker-compose.yml file for anyone curious (you should probably edit this to lock it down and use a non root user for cloudflared):

version: '3.6'
services:
  caddy:
    restart: always
    logging:
      driver: "json-file"
      options:
        max-size: "500k"
        max-file: "1"
    network_mode: bridge
    image: alexandzors/caddy
    ports:
        - 80:80
        - 443:443
    volumes:
        - /opt/docker/caddy/Caddyfile:/etc/caddy/Caddyfile:ro
        - /opt/docker/caddy:/data
        - /opt/docker/caddy:/config
        - /opt/docker/caddy/auth/local/users.json:/etc/caddy/auth/local/users.json
        - /media/plex:/srv

  cloudflared:
    links:
      - caddy
    restart: always
    logging:
      driver: "json-file"
      options:
        max-size: "500k"
        max-file: "1"
    network_mode: bridge
    image: cloudflare/cloudflared:2021.11.0
    command: 'tunnel --config /etc/cloudflared/config.yml run'
    volumes:
        - /opt/docker/cloudflared:/etc/cloudflared
    user: root

system · December 8, 2021, 5:48pm

This topic was automatically closed after 30 days. New replies are no longer allowed.