Cloudflare Tunnels and Caddy DNS Cloudflare

1. The problem I’m having:

I’m trying to use Cloudflare Tunnels to connect the domain insuvion.com to a locally running web app, using Caddy as a reverse proxy. I’m testing out on my MacBook Pro with cloudflared, caddy and a web app (hello world) running in separate docker containers specified by a docker-compose.yml. I’m trying to prove out the pattern on my laptop before deploying to a more robust setup on a server. Caddy fails to get the proper certificates for tls.

2. Error messages and/or full log output:

britz@M10-2198 paragon-ubuntu % docker compose up
[+] Running 3/4
 ✔ Network cloudflared                                                                                                                                         Created                                                                                                     0.0s
 ✔ Network caddy-1                                                                                                                                             Created                                                                                                     0.0s
 ⠋ Container cloudflared                                                                                                                                       Created                                                                                                     0.1s
 ⠋ Container hello-http-1                                                                                                                                      Created                                                                                                     0.1s
 ! hello-http-1 The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested                                                                                                             0.0s
 ⠋ Container caddy-1                                                                                                                                           Created                                                                                                     0.0s
Attaching to caddy-1, cloudflared, hello-http-1
hello-http-1  | httpd started
cloudflared   | 2024-04-02T18:26:17Z INF Starting tunnel tunnelID=<NOTE: REDACTED>
cloudflared   | 2024-04-02T18:26:17Z INF Version 2024.3.0
cloudflared   | 2024-04-02T18:26:17Z INF GOOS: linux, GOVersion: go1.21.5-devel-cf, GoArch: arm64
cloudflared   | 2024-04-02T18:26:17Z INF Settings: map[no-autoupdate:true]
cloudflared   | 2024-04-02T18:26:17Z INF Environmental variables map[TUNNEL_TOKEN:*****]
cloudflared   | 2024-04-02T18:26:17Z INF Generated Connector ID: <NOTE: REDACTED>
cloudflared   | 2024-04-02T18:26:17Z INF Initial protocol quic
cloudflared   | 2024-04-02T18:26:17Z INF ICMP proxy will use 192.168.144.2 as source for IPv4
cloudflared   | 2024-04-02T18:26:17Z INF ICMP proxy will use :: as source for IPv6
cloudflared   | 2024-04-02T18:26:17Z INF Starting metrics server on 127.0.0.1:40797/metrics
caddy-1       | {"level":"info","ts":1712082378.0069342,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy-1       | {"level":"info","ts":1712082378.0080435,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:2019","//[::1]:2019"]}
caddy-1       | {"level":"info","ts":1712082378.0081697,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy-1       | {"level":"info","ts":1712082378.008181,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy-1       | {"level":"info","ts":1712082378.0083408,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x400037fa80"}
caddy-1       | {"level":"info","ts":1712082378.0083816,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddy-1       | {"level":"info","ts":1712082378.0083988,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
caddy-1       | {"level":"info","ts":1712082378.0084596,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
caddy-1       | {"level":"info","ts":1712082378.0085154,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddy-1       | {"level":"info","ts":1712082378.0085177,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["insuvion.com"]}
caddy-1       | {"level":"info","ts":1712082378.0086744,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy-1       | {"level":"info","ts":1712082378.0086799,"msg":"serving initial configuration"}
caddy-1       | {"level":"info","ts":1712082378.0088773,"logger":"tls.obtain","msg":"acquiring lock","identifier":"insuvion.com"}
caddy-1       | {"level":"warn","ts":1712082378.0097108,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"d6e6aced-e476-477d-9c47-572b46514ab6","try_again":1712168778.00971,"try_again_in":86399.999999666}
caddy-1       | {"level":"info","ts":1712082378.0110495,"logger":"tls","msg":"finished cleaning storage units"}
caddy-1       | {"level":"info","ts":1712082378.011407,"logger":"tls.obtain","msg":"lock acquired","identifier":"insuvion.com"}
caddy-1       | {"level":"info","ts":1712082378.0114803,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"insuvion.com"}
caddy-1       | {"level":"info","ts":1712082378.0117629,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["insuvion.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"britz@graham-allen.com"}
caddy-1       | {"level":"info","ts":1712082378.0117815,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["insuvion.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"britz@graham-allen.com"}
cloudflared   | 2024/04/02 18:26:18 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
cloudflared   | 2024-04-02T18:26:18Z INF Registered tunnel connection connIndex=0 connection=92b3753e-3938-4839-acfa-3fdb108b3f9a event=0 ip=198.41.192.27 location=ord10 protocol=quic
cloudflared   | 2024-04-02T18:26:18Z INF Registered tunnel connection connIndex=1 connection=3bf97834-333c-4b6f-9133-139b9e3ca813 event=0 ip=198.41.200.63 location=ord07 protocol=quic
caddy-1       | {"level":"info","ts":1712082378.563633,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"insuvion.com","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
caddy-1       | {"level":"error","ts":1712082378.567124,"logger":"tls.issuance.acme.acme_client","msg":"cleaning up solver","identifier":"insuvion.com","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.insuvion.com\" (usually OK if presenting also failed)"}
caddy-1       | {"level":"error","ts":1712082378.628969,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"insuvion.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[insuvion.com] solving challenges: presenting for challenge: could not determine zone for domain \"_acme-challenge.insuvion.com\": could not find the start of authority for _acme-challenge.insuvion.com.: NOERROR (order=https://acme-v02.api.letsencrypt.org/acme/order/1649009607/257570687737) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
caddy-1       | {"level":"info","ts":1712082378.6293747,"logger":"tls.issuance.zerossl","msg":"waiting on internal rate limiter","identifiers":["insuvion.com"],"ca":"https://acme.zerossl.com/v2/DV90","account":"britz@graham-allen.com"}
caddy-1       | {"level":"info","ts":1712082378.6293857,"logger":"tls.issuance.zerossl","msg":"done waiting on internal rate limiter","identifiers":["insuvion.com"],"ca":"https://acme.zerossl.com/v2/DV90","account":"britz@graham-allen.com"}
caddy-1       | {"level":"info","ts":1712082379.216147,"logger":"tls.issuance.zerossl.acme_client","msg":"trying to solve challenge","identifier":"insuvion.com","challenge_type":"dns-01","ca":"https://acme.zerossl.com/v2/DV90"}
caddy-1       | {"level":"error","ts":1712082379.2223704,"logger":"tls.issuance.zerossl.acme_client","msg":"cleaning up solver","identifier":"insuvion.com","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.insuvion.com\" (usually OK if presenting also failed)"}
caddy-1       | {"level":"error","ts":1712082379.3683999,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"insuvion.com","issuer":"acme.zerossl.com-v2-DV90","error":"[insuvion.com] solving challenges: presenting for challenge: could not determine zone for domain \"_acme-challenge.insuvion.com\": could not find the start of authority for _acme-challenge.insuvion.com.: NOERROR (order=https://acme.zerossl.com/v2/DV90/order/NLR4r7hirWiWox9FLIpj_g) (ca=https://acme.zerossl.com/v2/DV90)"}
caddy-1       | {"level":"error","ts":1712082379.368595,"logger":"tls.obtain","msg":"will retry","error":"[insuvion.com] Obtain: [insuvion.com] solving challenges: presenting for challenge: could not determine zone for domain \"_acme-challenge.insuvion.com\": could not find the start of authority for _acme-challenge.insuvion.com.: NOERROR (order=https://acme.zerossl.com/v2/DV90/order/NLR4r7hirWiWox9FLIpj_g) (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":1.357170371,"max_duration":2592000}
cloudflared   | 2024-04-02T18:26:19Z INF Registered tunnel connection connIndex=2 connection=b8f076a5-520e-45e0-807a-dd589eabaccb event=0 ip=198.41.192.7 location=ord02 protocol=quic
cloudflared   | 2024-04-02T18:26:20Z INF Registered tunnel connection connIndex=3 connection=17a729e9-43a1-4e5f-9634-b8d82536c5a8 event=0 ip=198.41.200.33 location=ord12 protocol=quic
cloudflared   | 2024-04-02T18:26:23Z INF Updated to new configuration config="{\"ingress\":[{\"hostname\":\"insuvion.com\", \"originRequest\":{\"httpHostHeader\":\"insuvion.com\", \"originServerName\":\"insuvion.com\"}, \"service\":\"https://caddy-1:443\"}, {\"service\":\"http_status:404\"}], \"warp-routing\":{\"enabled\":false}}" version=2

After I run docker compose up, if I navigate to https://insuvion.com, I see a 502 Bad Gateway error, similar to the screen here: Troubleshooting Cloudflare 5XX errors · Cloudflare Support docs

…and my logs from docker compose show this:

cloudflared   | 2024-04-02T18:16:55Z ERR  error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: remote error: tls: internal error" connIndex=3 event=1 ingressRule=0 originService=https://caddy-1:443
cloudflared   | 2024-04-02T18:16:55Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: remote error: tls: internal error" connIndex=3 dest=https://insuvion.com/ event=0 ip=198.41.200.73 type=http

3. Caddy version:

My Caddy version is: 2.7.6.

4. How I installed and ran Caddy:

a. System environment:

I’m running caddy and cloudflared in separate docker containers, orchestrated by docker compose (see section C below for the Docker Compose File). I’m running on a MacBook Air M2 on MacOS Ventura.

My docker version is: Docker version 24.0.6, build ed223bc820
My Caddy version is: 2.7.6.

I built a caddy docker image with the following Caddy Dockerfile:

FROM caddy:builder as builder

RUN xcaddy build \
	--with github.com/caddy-dns/cloudflare

FROM caddy:2.7.6-alpine
COPY --from=builder /usr/bin/caddy /usr/bin/caddy

Cloudflare setup:

I have configured Cloudflare following the steps on Set up a tunnel through the dashboard.

  • I created a tunnel called macbook-test in the cloudflare UI, and got the token from the UI.

  • I configured a Public Hostname for insuvion.com, pointed to the service https://caddy-1:443:

  • I put insuvion.com into the “Origin Server Name” and insuvion.com into the “HTTP Host Header” Section in cloudflare:

  • I went to my cloudflare Profile, under cloud flare and created an API token with the Zone.Zone (Read), and Zone.DNS Edit permission for all zones.

I concatenated a few screenshots below of my cloudflare setup:

b. Command:

I run docker compose up to bring up the services.

c. Service/unit/compose file:

services:
  cloudflared:
      image: cloudflare/cloudflared
      container_name: cloudflared
      command: tunnel run
      environment:
        - TUNNEL_TOKEN=<CLOUDFLARE TUNNEL TOKEN (REDACTED)>
      restart: always
      networks:
      - cloudflared

  caddy-1:
    image: caddy-dns-cf:latest
    container_name: caddy-1
    ports:
      - 80:80
      - 443:443
    volumes:
      - caddy-data-1:/data
      - caddy-config-1:/config
      - $PWD/Caddyfile:/etc/caddy/Caddyfile
    depends_on:
      - cloudflared
    networks:
      - caddy-1
      - cloudflared

  hello-http-1:
    image: crccheck/hello-world
    container_name: hello-http-1
    networks:
      - caddy-1

networks:
  caddy-1:
    name: caddy-1
  cloudflared:
    name: cloudflared

volumes:
  caddy-data-1:
    name: caddy-data
  caddy-config-1:
    name: caddy-config

d. My complete Caddy config:

{
	email <EMAIL (REDACTED)>@<EMAIL DOMAIN (REDACTED)>.com
	acme_dns cloudflare <CLOUDFLARE API TOKEN (REDACTED)>
}

insuvion.com {
	reverse_proxy hello-http-1:8000
}

5. Links to relevant resources:

The following Caddyfile also failed, but I didn’t check the logs to make sure it was the same error as above.

{
	<EMAIL (REDACTED) >@<EMAIL DOMAIN (REDACTED)>.com
}

insuvion.com {
	reverse_proxy hello-http-1:8000
	tls {
		dns cloudflare <CLOUDFLARE API TOKEN (REDACTED)>
	}
}

There’s somekind of DNS misconfiguration with your domain, I think. Or with your system resolver. My understanding is that this is complaining about not being able to find an SOA record for your domain.

Thanks for the help. I ended up trying on the server that I would deploy to, and was getting a different error (this time, with Cloudflare…). So, my problems may be a little out of my depth here. I shifted focus to another avenue, but I’ll update back here if I return to it and find a solution.