Caddy + Cloudflare + remote_ip block by ipaddress not working

1. The problem I’m having:

I’ve caddy setup with a block to show maintenance page when coming from any ip other than our public ip, it works find when cloudflare proxy is not enabled. Once cloudflare is enabled it doesn’t work even though all configurations looks right.

Forgot to mention, it always shows the maintenance page i.e index.html from /home/ubuntu/maintenance

The idea is during maintenance want to show the maintenance page for everyone but we should be able to access and test it out internally.

May be we’re configuring something wrong, we know when coming through cloudflare the public ip’s will be coming differently, therefore we tried to add the headers as you see in the file using CF-Connecting-IP but still no luck.

Also tried removing forwarded next to remote_ip

Any help would be appreciated, thanks in advance.

2. Error messages and/or full log output:

no errors, just not showing the right content

3. Caddy version:

v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=

4. How I installed and ran Caddy:

Built using xcaddy with following plugins
xcaddy build --with GitHub - techknowlogick/certmagic-s3 --with GitHub - caddy-dns/cloudflare: Caddy module: dns.providers.cloudflare --with GitHub - caddy-dns/powerdns: Caddy module: dns.providers.powerdns --with GitHub - WeidiDeng/caddy-cloudflare-ip

a. System environment:

Ubuntu 20.04

b. Command:

sudo systemctl start caddy

c. Service/unit/compose file:

No docker, running directly

d. My complete Caddy config:

{
  debug
  log {
    format console
    output file /var/log/caddy/caddy.log {
	roll_size 10mb
        roll_keep 20
        roll_keep_for 720h
    }
  }

  on_demand_tls {
        ask http://127.0.0.1:8080/authorize
        interval 2m
        burst    5
  }

  servers {
      trusted_proxies cloudflare
  }
}

example.org, www.example.org {

  tls admin@example.org {
    dns cloudflare {env.CLOUDFLARE_AUTH_TOKEN}
  }

  header X-Forwarded-For {http.request.header.CF-Connecting-IP}
  header X-Real-IP {http.request.header.CF-Connecting-IP}

  @www header_regexp www Host ^www\.(.*)$
  redir @www https://{re.www.1} permanent
 
  @maintenance {
      path /*
      not remote_ip forwarded 1.2.3.4
  }
  handle @maintenance {
      file_server {
          root /home/ubuntu/maintenance
      }
  }

  reverse_proxy :8080 {
     # this works
  }
}

5. Links to relevant resources:

Tried with commenting these lines
header X-Forwarded-For {http.request.header.CF-Connecting-IP}
header X-Real-IP {http.request.header.CF-Connecting-IP}

Note: 1.2.3.4 replaced with our exact public ip address

This doesn’t make sense. The header directive sets a response header, not a request header. But those are request header names.

The forwarded option is insecure, it doesn’t take into account trusted_proxies.

In the next release of Caddy (or if you build from the master branch for now if you want), we’ve added a new client_ip matcher which does the right thing. It was implemented in https://github.com/caddyserver/caddy/pull/5104. You should also set client_ip_headers beside trusted_proxies in your global options to the header your downstream proxy uses (so CF-Connecting-IP in this case) so that it grabs the correct client IP from there.

Thanks @francislavoie for your fast response.

I can’t find client_ip_headers in the docs, is this part of master and not in the current released version?

Yeah like I said, it’s not released yet, it was just merged this week.

Thanks, is there any other way i can get the expected maintenance scenario done using the current release?

I removed forwarded
not remote_ip 1.2.3.4

Also removed these

 header X-Forwarded-For {http.request.header.CF-Connecting-IP}
 header X-Real-IP {http.request.header.CF-Connecting-IP}

Also is there a release planned anytime sooner?

We don’t have set times for releases. We tend to just release when we feel we have “enough” which isn’t super frequently.

I’d honestly just recommend pinning to a specific commit and building from that commit for now. The latest commit is 2b3046de so you can do xcaddy build 2b3046de --with ...

1 Like

Looks like this needs a go upgrade,

note: module requires Go 1.19```

Our current version:
 ```go version go1.18.1 linux/amd64```

Let me upgrade and try

Yeah, use the latest Go version.

Built 2b3046de and copied, how do i set client_ip_headers

Took a wild guess and set as

      client_ip_headers {http.request.header.CF-Connecting-IP}

Unfortunately that didn’t make any difference, trying the master now…

Just add client_ip_headers CF-Connecting-IP right below your trusted_proxies line.

1 Like

Both the tag 2b3046de and master works like a charm, thank you @francislavoie appreciated.

During this process I also noticed comcast issuing different IPv6 public addresses per device no matter we’re in the same network, I’ll also try again with released version applying correct IPv6 before testing.

Also noticed the actual client_ip’s in the logs, it was only showing proxy ip’s in released version, thanks for that.

Well, those are the same thing. Also that’s not a tag, that’s the commit hash. It’s just that pinning to a specific commit means if you re-run the build later it won’t pull in changes you didn’t expect. Removing the commit hash will default to the latest tagged release instead.

Yep :+1: that’s part of the changes, proper first-party “real IP” handling.

1 Like

For the ip’s in the client_ip argument, can i use a env variable like {$CADDY_WL_IPS} and set the value in ~/.bashrc a value like CADDY_WL_IPS=‘127.0.0.1 1.2.3,4 etc…’

example:
not client_ip {$CADDY_WL_IPS}

Tried few options, couldn’t get to work

{$CADDY_WL_IPS}
{env.CADDY_WL_IPS}

all are coming out as blank, tested by doing
respond "Hello={$CADDY_WL_IPS}"
respond "Hello={env.CADDY_WL_IPS}"

however the interesting part is, i’m using this

dns cloudflare {env.CLOUDFLARE_AUTH_TOKEN}

and that is working, cloudflare plugin. Not sure what’s the issue.

For now i set it in sudo systemctl edit caddy, Environment=" CADDY_WL_IPS=1.2.3.4" and that works.

Hope i’m doing it right, pls chime in.

That would not work because that sets env vars for your user, not the user Caddy actually runs as (i.e. the caddy user).

That’s the correct way of doing it. See the docs: Keep Caddy Running — Caddy Documentation

1 Like

Thank you!

The difference is that the former is replaced at Caddy-adapt time (i.e. once, when the config is loaded) whereas the latter is done at runtime (every time the value is read).

The former allows expanding to multiple tokens because it actually acts like copy-paste before the Caddyfile parsing happens, but the latter can only be a single token so it can’t expand to more than one value.

One reason to use {env.*} is that by default your config gets autosaved to Caddy’s config storage location and if you used the {$ENV} style, the actual value would appear in the autosaved JSON config, but that doesn’t happen with {env.*} because it’s replaced at runtime.

This might matter depending on your policies for data privacy etc because technically the {$ENV} way could leak secret values and {env.*} is safer, so it makes sense to use for the API key. But for a list of IP addresses, it matters less if that gets written elsewhere so it’s fine to use {$ENV} (and it’s your only option if you want more than one value).

Thanks for the detailed explanation, appreciated.

In that case, the first option going to be risky since our storage is configured as S3, means all these might appear in generated json over there (bucket is marked private but still there is a risk if a wrong config or bug it might expose the bucket)

Looks like env. will make the most sense but it has to be set in the caddy profile right?