Cloudflare Tunnel with Caddy - problem with config

Hello,

First of all, I just want to say that I like Caddy Server a lot. My intent is to use for a reverse proxy. I’m coming from NGINX Proxy Manager, which is very buggy, to Caddy and I like how it works, how fast it is, reliable and how it’s configured. Also, I tried every wiki and tutorial possible, either here, reddit or other sources, but they don’t seem to apply to my case or configuration. See below for further explanation.

1. The problem I’m having:

When I use the cloudflare tunnel to connect to my Caddy reverse proxy, I’m getting “host error”, which does not happen when I point the cloudflare tunnel to the NGINX Proxy Manager instance that I have running.

Note that locally, all works fine and well with Caddy. The problem is only when it is going through the cloudflare tunnel. An important thing is that I use a wildcard certificate. Also, some services I run with a mTLS certificate. Other services, I don’t. Further down, my Caddyfile will show how I set this up.

My cloudflare tunnel config.yml:

tunnel: <tunnel_UUID>
credentials-file: /etc/cloudflared/<tunnel_UUID>.json

ingress:
  - hostname: '*.mydomain.net'
    service: https://<caddy_lxc_ip>:443
    originRequest:
      noTLSverify: true
      originServerName: '*.mydomain.net'
      httpHostHeader: '*.mydomain.net'
  - service: http_status:404

Note that I tried all sorts of combinations with the options on the originRequest in the configuration, but to no avail.

2. Error messages and/or full log output:

{"level":"info","ts":1738949007.8934052,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1738949007.8949847,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0005c6480"}
{"level":"info","ts":1738949007.8952591,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"warn","ts":1738949007.8952835,"logger":"http","msg":"enabling strict SNI-Host enforcement because TLS client auth is configured","server_id":"srv0"}
{"level":"info","ts":1738949007.8966854,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1738949007.8973846,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"warn","ts":1738949007.897432,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"warn","ts":1738949007.897441,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"info","ts":1738949007.8974454,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1738949007.8974502,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["*.mydomain.net"]}
{"level":"info","ts":1738949007.8979354,"msg":"autosaved config (load with --resume flag)","file":"/root/.config/caddy/autosave.json"}
{"level":"info","ts":1738949007.897953,"msg":"serving initial configuration"}
{"level":"info","ts":1738949007.9025705,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/root/.local/share/caddy","instance":"7ee4f97f-d64d-434a-8010-2d645bd326b9","try_again":1739035407.9025645,"try_again_in":86399.999998044}
{"level":"info","ts":1738949007.902646,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1738956807.9011948,"logger":"tls.cache.maintenance","msg":"reloaded ARI with newer one in storage","identifiers":["*.mydomain.net"],"cert_hash":"6cbd7b39debde2e888762f509c65fe9b307c6bff8651e250bf94ae455855b216","ari_unique_id":"kydGmAOpUWiOmNbEQkjbI79YlNI.AxedazOBd_uPcaTvcY0wAyn3","cert_expiry":1746164729,"next_refresh":1738978194.3478506,"renewal_time":1743637188}
{"level":"info","ts":1738965571.6347935,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/stop","remote_ip":"127.0.0.1","remote_port":"48060","headers":{"Accept-Encoding":["gzip"],"Content-Length":["0"],"Origin":["http://localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
{"level":"warn","ts":1738965571.634828,"logger":"admin.api","msg":"exiting; byeee!! 👋"}
{"level":"info","ts":1738965571.6348436,"logger":"http","msg":"servers shutting down with eternal grace period"}
{"level":"info","ts":1738965571.6362305,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
{"level":"info","ts":1738965571.63625,"logger":"admin.api","msg":"shutdown complete","exit_code":0}
{"level":"info","ts":1738965578.3277547,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1738965578.3284423,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0008ae880"}
{"level":"info","ts":1738965578.3288624,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"warn","ts":1738965578.3288896,"logger":"http","msg":"enabling strict SNI-Host enforcement because TLS client auth is configured","server_id":"srv0"}
{"level":"info","ts":1738965578.3302643,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1738965578.3304112,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"warn","ts":1738965578.3304396,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"warn","ts":1738965578.3304534,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"info","ts":1738965578.330458,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1738965578.3304622,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["*.mydomain.net"]}
{"level":"info","ts":1738965578.3309326,"msg":"autosaved config (load with --resume flag)","file":"/root/.config/caddy/autosave.json"}
{"level":"info","ts":1738965578.3309417,"msg":"serving initial configuration"}
{"level":"info","ts":1738965578.336066,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/root/.local/share/caddy"}
{"level":"info","ts":1738965578.3395813,"logger":"tls","msg":"finished cleaning storage units"}

3. Caddy version:

v2.9.1 h1:OEYiZ7DbCzAWVb6TNEkjRcSCRGHVoZsJinoDR/n9oaY=

4. How I installed and ran Caddy:

a. System environment:

I have a Proxmox server where I have one LXC for the Cloudflare Tunnel (cloudflared) and another LXC with Caddy. Both LXC’s were generated based on the nice people at helper-scripts.com

The Caddy LXC is Debian 12.

b. Command:

sudo caddy fmt --overwrite
sudo caddy adapt
sudo caddy run

c. Service/unit/compose file:

N/A

d. My complete Caddy config:

{
	auto_https prefer_wildcard
	log {
		output file /var/log/access.log
	}
}

(localonly) {
	@denied not remote_ip private_ranges
	abort @denied
}

(mtls) {
	tls {
		client_auth {
			mode require_and_verify
			trust_pool file <file_directory>/ca.pem
		}
	}
}

*.mydomain.net {
	tls {
		dns cloudflare {env.CLOUDFLARE_API_KEY}
	}
}

service1.mydomain.net {
  reverse_proxy <service1_ip>:<service1_port>
}

service2.mydomain.net {
  import localonly
  import mtls
  reserve_proxy <service2_ip>:<service2_port>
}

When I run the service1 from the cloudflare tunnel, I get the host error as mentioned.

5. Links to relevant resources:

I believe you need to explicitly tell Caddy to use the incoming Host header for routing. That can be done with the header_up subdirective.

service1.mydomain.net {
    reverse_proxy <service1_ip>:<service1_port> {
        header_up Host {host}
    }
}

service2.mydomain.net {
    import localonly
    import mtls
    reverse_proxy <service2_ip>:<service2_port> {
        header_up Host {host}
    }
}

That should tell Caddy to forward the incoming Host header from the request to your upstream server (<service1_ip>:<service1_port> or <service2_ip>:<service2_port>)

Hi @TheRettom,

Thank you for your reply.
Just to be clear, the {host} is the IP of the Cloudflare Tunnel container, right?

This is the documentation on the host matcher. There’s different ways of implementing it, so do what works for you.

You can also use the (undocumented) matchSNItoHost: true clause in the cloudflared originRequest config. Then you wouldn’t need the Host configuration nor the noTLSVerify’.

I have this config which allows multiple hostnames to be used on the tunnel and passed through to the local http server. (In my case I’m not using it with Caddy yet but I don’t see why it wouldn’t work.)

ingress:
  - originRequest:
      matchSNItoHost: true
    service: https://localhost
1 Like

None of the suggestions above work, unfortunately.

What does the log say now? The snippet you posted doesn’t seem to include any accesses, as best I can tell.

Is it better if you disable mTLS? I’d figure out one problem at a time.

I’m not sure that having a *.mydomain.net and then service1.mydomain.net clause does what you want, ie getting a certificate from Cloudflare for service1 as well.

That is true. Would the client_auth subdirective be useful for the specified subdomains in this case?

I don’t know what you’re trying to achieve with mTLS so it’s hard to say. As above I suggest turning it off and checking some basics of the tunnel + caddy before trying to add mLTS as well.

Honestly, the mTLS can be disregarded. Right now, I’m only trying to get the services without mTLS to work with Cloudflare tunnel. As suggested, I’m trying to get it to work in small steps.

I will actually try the suggested pattern in Caddy docs (Common Caddyfile Patterns — Caddy Documentation) and let you guys know.

I’m just puzzled because, when I access locally, I have my local DNS server (Adguard / Pi-hole type) pointing to Caddy with the configuration above and all works flawlessly, but when Cloudflare tunnel points to Caddy, it resolves with a “host error”.

At this point, I’m even considering a different solution than the Cloudflare tunnel. I would just prefer not to open ports on my router, for the time being at least.

The logs are not different from my original post.

1 Like

Did you try using header_up Host {host} in your Caddyfile? It should be forwarding the hostname of the client with that.

So, I was able to make it work with the configs below.

tunnel: UUID
credentials-file: /etc/cloudflared/UUID.json

ingress:
  - hostname: '*.mydomain.net'
    service: https://<caddy_lxc_ip>:443
    originRequest:
      originServerName: '*.mydoamin.net'
  - service: http_status:404
{
       acme_dns cloudflared {env.CLOUDFLARE_API_KEY}
}

*.mydomain.net {
     @service1 host service1.mydomain.net 
     
     handle @service1 {
        reverse_proxy <service1_ip>:<service1_port>
     }

    handle {
        abort
    }
}

But what if I want to use the same wildcard certificate and want to be able to allow mTLS locally for a few of the services? This is not possible with the handle @service1 directive.
And when I add the service2 as a second entry into my Caddyfile, then the cloudflare tunnel resolves with a HTTP ERROR 421.

@TheRettom Regarding your suggestion for the header_up Host {host} instruction, I was not sure what the {host} should be, so I tried every different possibility: the <cloudflared_lxc_ip>, the <serviceX_ip>, the domain serviceX.mydomain.net… None of them worked. The Caddy docs may describe what it is, but I don’t see how it could apply to my case.

So you would leave {host} as is, you don’t need to change it. This is a placeholder.

I’m not sure if the handle directive can work with the tls directive (@ is just a name matcher), but you can still add tls to the specified subdomain outside of the name matcher.

You can use the client_auth directive and use require_and_verify to see if that works for mTLS. @francislavoie has a post from a few years ago mentioning other options.

Ok, I misunderstood the header_up Host {host} thing. But still, I tried and it didn’t work.

Outside of the name matcher? In which case? With the handle directive or? Like the example below?

{
       acme_dns cloudflared {env.CLOUDFLARE_API_KEY}
}

*.mydomain.net {
     @service1 host service1.mydomain.net 
     handle @service1 {
        reverse_proxy <service1_ip>:<service1_port> 
     }

    handle {
        abort
    }
}

service1.mydomain.net {
   tls {
      client_auth {
         mode require_and_verify
         trust_pool file /<directory>/<file>.pem
      }
   }
}

Yeah, just like that Caddyfile.

Have your logs changed at all since adding header_up?

header_up is about Caddy working as a reverse proxy to a backend application. It doesn’t affect the interface between cloudflared and Caddy. I don’t think it’s relevant to the problem.

1 Like

That is why I suggested you use matchSNItoHost: true instead of originServerName: '*.mydomain.net'. You are forcing cloudflared to expect the wrong certificate.

So, I tried this:

tunnel: <tunnel_UUID>
credentials-file: /etc/cloudflared/<tunnel_UUID>.json

ingress:
  - service: https://<caddy_lxc_ip>:443
    originRequest:
      matchSNItoHost: true
  - service: http_status:404

With that, the error I’m getting now is:

Ok, I feel like I’m making progress.

I was able to connect with the Caddyfile as in the original post plus this config for cloudflared:

tunnel: <tunnel_UUID>
credentials-file: /etc/cloudflared/<tunnel_UUID>.json
originRequest:
  matchSNItoHost: true

ingress:
  - hostname: '*.mydomain.net'
    service: https://<caddy_lxc_ip>:443
  - service: http_status:404

I was putting the originRequest in the wrong place… My bad.

But the connection is not stable. There are a few endpoints which are reached, and others are not.
Strangely, the endpoint becomes reachable for a while after restarting with sudo caddy stop and sudo caddy start, and after a few seconds, I can’t reach them.

I will investigate further and bring the logs once possible.

These are the errors I’m getting when I can access one endpoint (serviceX) and when I can’t access other endpoint (serviceY) after restarting the caddy service in my Caddy LXC.

{"level":"debug","ts":1739468153.7551732,"logger":"http.log.error","msg":"strict host matching: TLS ServerName (serviceX.mydomain.net) and HTTP Host (serviceY.mydomain.net) values differ","request":{"remote_ip":"<cloudflared_lxc_ip>","remote_port":"40700","client_ip":"<cloudflared_lxc_ip>","proto":"HTTP/1.1","method":"GET","host":"serviceY.mydomain.net","uri":"/","headers":{"Accept-Encoding":["gzip, br"],"Cf-Ipcountry":["US"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Sec-Ch-Ua":["\"Not(A:Brand\";v=\"99\", \"Google Chrome\";v=\"133\", \"Chromium\";v=\"133\""],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"],"Cf-Connecting-Ip":["<cf_connecting_ip>"],"X-Forwarded-For":["<cf_connecting_ip>"],"Cf-Ray":["<cf_ray_id>"],"Connection":["keep-alive"],"User-Agent":["Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Mobile Safari/537.36"],"Accept-Language":["en-US,en;q=0.9,pt-BR;q=0.8,pt;q=0.7,sv;q=0.6,es;q=0.5"],"Purpose":["prefetch"],"Priority":["u=0, i"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Sec-Ch-Ua-Mobile":["?1"],"Cf-Warp-Tag-Id":["<cf_warp_tag_id>"],"Cookie":["REDACTED"],"Sec-Ch-Ua-Platform":["\"Android\""],"Sec-Fetch-Dest":["document"],"Sec-Fetch-User":["?1"],"Cdn-Loop":["cloudflare; loops=1"],"Sec-Purpose":["prefetch;prerender"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"serviceX.mydomain.net"}},"duration":0.000031918,"status":421,"err_id":"mtwe3a18r","err_trace":"caddyhttp.(*Server).enforcementHandler (server.go:479)"}
{"level":"debug","ts":1739468153.836291,"logger":"http.log.error","msg":"strict host matching: TLS ServerName (serviceX.mydomain.net) and HTTP Host (serviceY.mydomain.net) values differ","request":{"remote_ip":"<cloudflared_lxc_ip>","remote_port":"40700","client_ip":"<cloudflared_lxc_ip>","proto":"HTTP/1.1","method":"GET","host":"serviceY.mydomain.net","uri":"/","headers":{"Sec-Fetch-Site":["none"],"Upgrade-Insecure-Requests":["1"],"Sec-Ch-Ua-Mobile":["?1"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Connection":["keep-alive"],"Cf-Warp-Tag-Id":["<cf_warp_tag_id>"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Sec-Purpose":["prefetch;prerender"],"Sec-Fetch-User":["?1"],"X-Forwarded-For":["<cf_connecting_ip>"],"User-Agent":["Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Mobile Safari/537.36"],"Cdn-Loop":["cloudflare; loops=1"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Dest":["document"],"Cf-Ipcountry":["US"],"Cookie":["REDACTED"],"Purpose":["prefetch"],"Accept-Encoding":["gzip, br"],"Priority":["u=0, i"],"X-Forwarded-Proto":["https"],"Sec-Ch-Ua-Platform":["\"Android\""],"Accept-Language":["en-US,en;q=0.9,pt-BR;q=0.8,pt;q=0.7,sv;q=0.6,es;q=0.5"],"Cf-Ray":["<cf_ray_id>"],"Cf-Connecting-Ip":["<cf_connecting_ip>"],"Sec-Ch-Ua":["\"Not(A:Brand\";v=\"99\", \"Google Chrome\";v=\"133\", \"Chromium\";v=\"133\""]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"serviceX.mydomain.net"}},"duration":0.000035829,"status":421,"err_id":"q5bggm8c5","err_trace":"caddyhttp.(*Server).enforcementHandler (server.go:479)"}
{"level":"debug","ts":1739468153.9395714,"logger":"http.log.error","msg":"strict host matching: TLS ServerName (serviceX.mydomain.net) and HTTP Host (serviceY.mydomain.net) values differ","request":{"remote_ip":"<cloudflared_lxc_ip>","remote_port":"40700","client_ip":"<cloudflared_lxc_ip>","proto":"HTTP/1.1","method":"GET","host":"serviceY.mydomain.net","uri":"/","headers":{"Sec-Ch-Ua":["\"Not(A:Brand\";v=\"99\", \"Google Chrome\";v=\"133\", \"Chromium\";v=\"133\""],"Sec-Fetch-Mode":["navigate"],"Accept-Encoding":["gzip, br"],"Cf-Ipcountry":["US"],"Priority":["u=0, i"],"Cf-Ray":["<cf_ray_id>"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"X-Forwarded-Proto":["https"],"Sec-Ch-Ua-Platform":["\"Android\""],"Sec-Fetch-Site":["none"],"Cdn-Loop":["cloudflare; loops=1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"User-Agent":["Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Mobile Safari/537.36"],"X-Forwarded-For":["<cf_connecting_ip>"],"Cookie":["REDACTED"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-User":["?1"],"Upgrade-Insecure-Requests":["1"],"Cf-Warp-Tag-Id":["<cf_warp_tag_id>"],"Sec-Ch-Ua-Mobile":["?1"],"Connection":["keep-alive"],"Accept-Language":["en-US,en;q=0.9,pt-BR;q=0.8,pt;q=0.7,sv;q=0.6,es;q=0.5"],"Cf-Connecting-Ip":["<cf_connecting_ip>"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"serviceX.mydomain.net"}},"duration":0.000051544,"status":421,"err_id":"tazktpdvt","err_trace":"caddyhttp.(*Server).enforcementHandler (server.go:479)"}
{"level":"debug","ts":1739468153.9906244,"logger":"http.log.error","msg":"strict host matching: TLS ServerName (serviceX.mydomain.net) and HTTP Host (serviceY.mydomain.net) values differ","request":{"remote_ip":"<cloudflared_lxc_ip>","remote_port":"40700","client_ip":"<cloudflared_lxc_ip>","proto":"HTTP/1.1","method":"GET","host":"serviceY.mydomain.net","uri":"/","headers":{"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Cf-Ray":["<cf_ray_id>"],"Sec-Ch-Ua-Mobile":["?1"],"Upgrade-Insecure-Requests":["1"],"Connection":["keep-alive"],"Sec-Fetch-Site":["none"],"Cookie":["REDACTED"],"Cdn-Loop":["cloudflare; loops=1"],"X-Forwarded-For":["<cf_connecting_ip>"],"User-Agent":["Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Mobile Safari/537.36"],"Cf-Warp-Tag-Id":["<cf_warp_tag_id>"],"Sec-Fetch-User":["?1"],"X-Forwarded-Proto":["https"],"Accept-Encoding":["gzip, br"],"Cf-Connecting-Ip":["<cf_connecting_ip>"],"Accept-Language":["en-US,en;q=0.9,pt-BR;q=0.8,pt;q=0.7,sv;q=0.6,es;q=0.5"],"Priority":["u=0, i"],"Sec-Fetch-Dest":["document"],"Cf-Ipcountry":["US"],"Sec-Ch-Ua":["\"Not(A:Brand\";v=\"99\", \"Google Chrome\";v=\"133\", \"Chromium\";v=\"133\""],"Sec-Ch-Ua-Platform":["\"Android\""],"Sec-Fetch-Mode":["navigate"],"Cf-Visitor":["{\"scheme\":\"https\"}"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"serviceX.mydomain.net"}},"duration":0.000032687,"status":421,"err_id":"j94k40f1q","err_trace":"caddyhttp.(*Server).enforcementHandler (server.go:479)"}