1. The problem I’m having:
I’m trying to use Cloudflare Tunnels to connect the domain insuvion.com to a locally running web app, using Caddy as a reverse proxy. I’m testing out on my MacBook Pro with cloudflared, caddy and a web app (hello world) running in separate docker containers specified by a docker-compose.yml. I’m trying to prove out the pattern on my laptop before deploying to a more robust setup on a server. Caddy fails to get the proper certificates for tls.
2. Error messages and/or full log output:
britz@M10-2198 paragon-ubuntu % docker compose up
[+] Running 3/4
✔ Network cloudflared Created 0.0s
✔ Network caddy-1 Created 0.0s
⠋ Container cloudflared Created 0.1s
⠋ Container hello-http-1 Created 0.1s
! hello-http-1 The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s
⠋ Container caddy-1 Created 0.0s
Attaching to caddy-1, cloudflared, hello-http-1
hello-http-1 | httpd started
cloudflared | 2024-04-02T18:26:17Z INF Starting tunnel tunnelID=<NOTE: REDACTED>
cloudflared | 2024-04-02T18:26:17Z INF Version 2024.3.0
cloudflared | 2024-04-02T18:26:17Z INF GOOS: linux, GOVersion: go1.21.5-devel-cf, GoArch: arm64
cloudflared | 2024-04-02T18:26:17Z INF Settings: map[no-autoupdate:true]
cloudflared | 2024-04-02T18:26:17Z INF Environmental variables map[TUNNEL_TOKEN:*****]
cloudflared | 2024-04-02T18:26:17Z INF Generated Connector ID: <NOTE: REDACTED>
cloudflared | 2024-04-02T18:26:17Z INF Initial protocol quic
cloudflared | 2024-04-02T18:26:17Z INF ICMP proxy will use 192.168.144.2 as source for IPv4
cloudflared | 2024-04-02T18:26:17Z INF ICMP proxy will use :: as source for IPv6
cloudflared | 2024-04-02T18:26:17Z INF Starting metrics server on 127.0.0.1:40797/metrics
caddy-1 | {"level":"info","ts":1712082378.0069342,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy-1 | {"level":"info","ts":1712082378.0080435,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:2019","//[::1]:2019"]}
caddy-1 | {"level":"info","ts":1712082378.0081697,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy-1 | {"level":"info","ts":1712082378.008181,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy-1 | {"level":"info","ts":1712082378.0083408,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x400037fa80"}
caddy-1 | {"level":"info","ts":1712082378.0083816,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddy-1 | {"level":"info","ts":1712082378.0083988,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
caddy-1 | {"level":"info","ts":1712082378.0084596,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
caddy-1 | {"level":"info","ts":1712082378.0085154,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddy-1 | {"level":"info","ts":1712082378.0085177,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["insuvion.com"]}
caddy-1 | {"level":"info","ts":1712082378.0086744,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy-1 | {"level":"info","ts":1712082378.0086799,"msg":"serving initial configuration"}
caddy-1 | {"level":"info","ts":1712082378.0088773,"logger":"tls.obtain","msg":"acquiring lock","identifier":"insuvion.com"}
caddy-1 | {"level":"warn","ts":1712082378.0097108,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"d6e6aced-e476-477d-9c47-572b46514ab6","try_again":1712168778.00971,"try_again_in":86399.999999666}
caddy-1 | {"level":"info","ts":1712082378.0110495,"logger":"tls","msg":"finished cleaning storage units"}
caddy-1 | {"level":"info","ts":1712082378.011407,"logger":"tls.obtain","msg":"lock acquired","identifier":"insuvion.com"}
caddy-1 | {"level":"info","ts":1712082378.0114803,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"insuvion.com"}
caddy-1 | {"level":"info","ts":1712082378.0117629,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["insuvion.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"britz@graham-allen.com"}
caddy-1 | {"level":"info","ts":1712082378.0117815,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["insuvion.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"britz@graham-allen.com"}
cloudflared | 2024/04/02 18:26:18 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
cloudflared | 2024-04-02T18:26:18Z INF Registered tunnel connection connIndex=0 connection=92b3753e-3938-4839-acfa-3fdb108b3f9a event=0 ip=198.41.192.27 location=ord10 protocol=quic
cloudflared | 2024-04-02T18:26:18Z INF Registered tunnel connection connIndex=1 connection=3bf97834-333c-4b6f-9133-139b9e3ca813 event=0 ip=198.41.200.63 location=ord07 protocol=quic
caddy-1 | {"level":"info","ts":1712082378.563633,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"insuvion.com","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
caddy-1 | {"level":"error","ts":1712082378.567124,"logger":"tls.issuance.acme.acme_client","msg":"cleaning up solver","identifier":"insuvion.com","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.insuvion.com\" (usually OK if presenting also failed)"}
caddy-1 | {"level":"error","ts":1712082378.628969,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"insuvion.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[insuvion.com] solving challenges: presenting for challenge: could not determine zone for domain \"_acme-challenge.insuvion.com\": could not find the start of authority for _acme-challenge.insuvion.com.: NOERROR (order=https://acme-v02.api.letsencrypt.org/acme/order/1649009607/257570687737) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
caddy-1 | {"level":"info","ts":1712082378.6293747,"logger":"tls.issuance.zerossl","msg":"waiting on internal rate limiter","identifiers":["insuvion.com"],"ca":"https://acme.zerossl.com/v2/DV90","account":"britz@graham-allen.com"}
caddy-1 | {"level":"info","ts":1712082378.6293857,"logger":"tls.issuance.zerossl","msg":"done waiting on internal rate limiter","identifiers":["insuvion.com"],"ca":"https://acme.zerossl.com/v2/DV90","account":"britz@graham-allen.com"}
caddy-1 | {"level":"info","ts":1712082379.216147,"logger":"tls.issuance.zerossl.acme_client","msg":"trying to solve challenge","identifier":"insuvion.com","challenge_type":"dns-01","ca":"https://acme.zerossl.com/v2/DV90"}
caddy-1 | {"level":"error","ts":1712082379.2223704,"logger":"tls.issuance.zerossl.acme_client","msg":"cleaning up solver","identifier":"insuvion.com","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.insuvion.com\" (usually OK if presenting also failed)"}
caddy-1 | {"level":"error","ts":1712082379.3683999,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"insuvion.com","issuer":"acme.zerossl.com-v2-DV90","error":"[insuvion.com] solving challenges: presenting for challenge: could not determine zone for domain \"_acme-challenge.insuvion.com\": could not find the start of authority for _acme-challenge.insuvion.com.: NOERROR (order=https://acme.zerossl.com/v2/DV90/order/NLR4r7hirWiWox9FLIpj_g) (ca=https://acme.zerossl.com/v2/DV90)"}
caddy-1 | {"level":"error","ts":1712082379.368595,"logger":"tls.obtain","msg":"will retry","error":"[insuvion.com] Obtain: [insuvion.com] solving challenges: presenting for challenge: could not determine zone for domain \"_acme-challenge.insuvion.com\": could not find the start of authority for _acme-challenge.insuvion.com.: NOERROR (order=https://acme.zerossl.com/v2/DV90/order/NLR4r7hirWiWox9FLIpj_g) (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":1.357170371,"max_duration":2592000}
cloudflared | 2024-04-02T18:26:19Z INF Registered tunnel connection connIndex=2 connection=b8f076a5-520e-45e0-807a-dd589eabaccb event=0 ip=198.41.192.7 location=ord02 protocol=quic
cloudflared | 2024-04-02T18:26:20Z INF Registered tunnel connection connIndex=3 connection=17a729e9-43a1-4e5f-9634-b8d82536c5a8 event=0 ip=198.41.200.33 location=ord12 protocol=quic
cloudflared | 2024-04-02T18:26:23Z INF Updated to new configuration config="{\"ingress\":[{\"hostname\":\"insuvion.com\", \"originRequest\":{\"httpHostHeader\":\"insuvion.com\", \"originServerName\":\"insuvion.com\"}, \"service\":\"https://caddy-1:443\"}, {\"service\":\"http_status:404\"}], \"warp-routing\":{\"enabled\":false}}" version=2
After I run docker compose up, if I navigate to https://insuvion.com, I see a 502 Bad Gateway error, similar to the screen here: Troubleshooting Cloudflare 5XX errors · Cloudflare Support docs
…and my logs from docker compose show this:
cloudflared | 2024-04-02T18:16:55Z ERR error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: remote error: tls: internal error" connIndex=3 event=1 ingressRule=0 originService=https://caddy-1:443
cloudflared | 2024-04-02T18:16:55Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: remote error: tls: internal error" connIndex=3 dest=https://insuvion.com/ event=0 ip=198.41.200.73 type=http
3. Caddy version:
My Caddy version is: 2.7.6.
4. How I installed and ran Caddy:
a. System environment:
I’m running caddy and cloudflared in separate docker containers, orchestrated by docker compose (see section C below for the Docker Compose File). I’m running on a MacBook Air M2 on MacOS Ventura.
My docker version is: Docker version 24.0.6, build ed223bc820
My Caddy version is: 2.7.6.
I built a caddy docker image with the following Caddy Dockerfile:
FROM caddy:builder as builder
RUN xcaddy build \
--with github.com/caddy-dns/cloudflare
FROM caddy:2.7.6-alpine
COPY --from=builder /usr/bin/caddy /usr/bin/caddy
Cloudflare setup:
I have configured Cloudflare following the steps on Set up a tunnel through the dashboard.
-
I created a tunnel called
macbook-testin the cloudflare UI, and got the token from the UI. -
I configured a Public Hostname for
insuvion.com, pointed to the servicehttps://caddy-1:443: -
I put
insuvion.cominto the “Origin Server Name” andinsuvion.cominto the “HTTP Host Header” Section in cloudflare: -
I went to my cloudflare Profile, under cloud flare and created an API token with the Zone.Zone (Read), and Zone.DNS Edit permission for all zones.
I concatenated a few screenshots below of my cloudflare setup:
b. Command:
I run docker compose up to bring up the services.
c. Service/unit/compose file:
services:
cloudflared:
image: cloudflare/cloudflared
container_name: cloudflared
command: tunnel run
environment:
- TUNNEL_TOKEN=<CLOUDFLARE TUNNEL TOKEN (REDACTED)>
restart: always
networks:
- cloudflared
caddy-1:
image: caddy-dns-cf:latest
container_name: caddy-1
ports:
- 80:80
- 443:443
volumes:
- caddy-data-1:/data
- caddy-config-1:/config
- $PWD/Caddyfile:/etc/caddy/Caddyfile
depends_on:
- cloudflared
networks:
- caddy-1
- cloudflared
hello-http-1:
image: crccheck/hello-world
container_name: hello-http-1
networks:
- caddy-1
networks:
caddy-1:
name: caddy-1
cloudflared:
name: cloudflared
volumes:
caddy-data-1:
name: caddy-data
caddy-config-1:
name: caddy-config
d. My complete Caddy config:
{
email <EMAIL (REDACTED)>@<EMAIL DOMAIN (REDACTED)>.com
acme_dns cloudflare <CLOUDFLARE API TOKEN (REDACTED)>
}
insuvion.com {
reverse_proxy hello-http-1:8000
}
5. Links to relevant resources:
The following Caddyfile also failed, but I didn’t check the logs to make sure it was the same error as above.
{
<EMAIL (REDACTED) >@<EMAIL DOMAIN (REDACTED)>.com
}
insuvion.com {
reverse_proxy hello-http-1:8000
tls {
dns cloudflare <CLOUDFLARE API TOKEN (REDACTED)>
}
}