RemoteAddr and Caddy v2 - help needed

[Austin_Kirsch]

1. Caddy version (caddy version):

2.2.1

2. How I run Caddy:

Running caddy via docker-compose in a debian VM (proxmox host environment)

a. System environment:

latest debian
docker version 19.03.13

b. Command:

docker-compose up -d

c. Service/unit/compose file:

full Dockerfile

RUN xcaddy build v2.2.1 \
        --with github.com/caddy-dns/cloudflare@latest \
        --with github.com/greenpau/caddy-auth-jwt \
        --with github.com/greenpau/caddy-auth-portal \
        --with github.com/kirsch33/format-encoder@latest

full docker-compose.yml

version: "3.7"
services:
  caddy:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: caddy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"

    environment:
      - CLOUDFARE_EMAIL=MY_EMAIL
      - CLOUDFARE_API_TOKEN=MY_TOKEN

    volumes:
      - /web/Caddyfile:/etc/caddy/Caddyfile
      - /web/auth:/authdb
      - /web/site:/srv
      - /var/docks/caddy/data:/data
      - /var/docks/caddy/config:/config

d. My complete Caddyfile or JSON config:

https://greatwhitelab.net {
        route /auth* {
                auth_portal {
                        path /auth
                                backends {
                                        local_backend {
                                                method local
                                                path /authdb/users.json
                                                realm local
                                                }
                                        }
                        cookie_domain greatwhitelab.net
                        jwt {
                                token_name access_token
                                token_secret TOKEN_SECRET
                                token_issuer TOKEN_ISSUER
                                }
                        ui {
                                links {
                                        "Plex" https://plex.greatwhitelab.net
                                        }
                                }
                        }

                }

        route {
                redir https://greatwhitelab.net/auth 302
                }

        tls atki224@g.uky.edu {
                dns cloudflare {env.CLOUDFLARE_API_TOKEN}
                }

        encode gzip
        header {
                header_up X-Forwarded-For {>CF-Connecting-IP}
                Strict-Transport-Security max-age=31536000;
                X-Content-Type-Options nosniff
                X-Frame-Options DENY
                Referrer-Policy no-referrer-when-downgrade
                X-XSS-Protection 1
                }

        log {
                format formatted "{request>headers>X-Forwarded-For} {ts} {request>method} {request>uri} {request>proto} {status}"
                output file /authdb/access.log {
                        roll_size 50MiB
                        roll_keep 5
                        roll_keep_for 48h
                        }
                }
}

3. The problem I’m having:

Apologies for posting twice within short order. I have withdrawn my other thread after researching my problem more. If you read what I posted last night, please ignore and read below instead:

I’m trying to setup fail2ban while using Cloudflare proxy service. I have my auth logs formatted correctly to see the failed attempts and the corresponding client IP by passing the X-Forwarded-For header to the output log however the problem is that IP tables cannot read headers with full SSL enabled… so all I can ever read from ip tables is the Cloudflare sever IP that is being passed as RemoteAddr on the request.

I need to be able to rewrite RemoteAddr to be equal to the value Cloudflare sets in the X-Forwarded-For header for this to work. I understand in Caddy v1 the realip module provided this functionality and I’ve seen some posts mention it but with no follow up, such as the post below:

V2: Equivalent of v1 realip plugin

1. My Caddy version (caddy version): v2 2. How I run Caddy: official docker image a. System environment: Ubuntu 19.10 I’m just wondering if there is a v2 equivalent for the realip plugin. In v1 I used to use realip cloudflare in my caddyfile and I’d like to keep that functionality in v2.

Can anyone shed some light on if the ability to set RemoteAddr was ever added to caddy v2, if so what that may look like? Any pointers to get this working would be helpful.

4. Error messages and/or full log output:

[“ACTUAL_CLIENT_IP”] 2020-10-26T01:40:10Z POST /auth HTTP/1.1 401

5. What I already tried:

See above

[francislavoie]

According to @Mohammed90 I think you can do {request>headers>X-Forwarded-For>[0]} to grab the first element in the array, but you’ll need to try it out and see.

Regarding the realip module, someone just needs to implement it. It’s quite simple, so you might be able to do it yourself if you need:

Extending Caddy - Caddy Documentation

This is the v1 plugin, it would just need to be ported to v2’s plugin system:

GitHub - captncraig/caddy-realip: Real-IP middleware for caddy

[Austin_Kirsch]

Thanks for the reply. Unfortunately I think I will have to go the route of porting the realip module since IP tables only has the ability to look at RemoteAddr.

I will give it a shot this evening.

[matt (Matt Holt)]

Is there any reason the v1 realip plugin can’t be done in v2 simply by using the remote_ip matcher? Is it just setting RemoteAddr?

[francislavoie]

Yeah, nothing in vanilla Caddy allows for overwriting the r.RemoteAddr right now. We’d need a handler to do it. The rest of the realip plugin is just setting up a whitelist for trusted IPs, which can just be done with the remote_ip matcher anyways.

[matt (Matt Holt)]

Ah right. Why does that matter, again? RemoteAddr is an internal construct, not something that external apps can access. They only read headers and stuff, so why is overwriting RemoteAddr specifically so crucial?

[francislavoie]

It makes other pieces of the Caddy pipeline (logs, other plugins) not need to also support handling X-Forwarded-For (which requires parsing the array of headers plus the comma separated IP values, non-trivial to add support for that everywhere).

[Austin_Kirsch]

Matt, I’m a total noob to this stuff so I apologize if I’m wrong or mis-speaking but my understanding is that IP tables can ONLY read RemoteAddr with packets that have end to end SSL, such as my case.

[Austin_Kirsch]

@francislavoie @matt

View my work on porting @captncraig original “realip” plugin over to v2:

GitHub - kirsch33/realip: realip module for Caddy v2

It compiles just fine, moving over to testing now!

[francislavoie]

Cool! I think you’ll need to update your go.mod to point to Caddy v2, it’s still pointing to Caddy v1 currently.

Also please don’t forget to add a LICENSE file to your repo.

Once it’s working, you can sign up for an account here Download Caddy and register your plugin so it shows up in the download page for others to use it

Just as an aside, I noticed you made GitHub - kirsch33/format-encoder, did you consider making a PR against the upstream repo with your change instead of just forking it? That way others could benefit from your changes. I didn’t look at what exactly you changed though, just a thought.

[Austin_Kirsch]

Thank you for the heads up, its been a long night.

Also, regarding my changes to format-encoder, I just added one line to format the time to something other than the plugin default of UNIX-micoseconds. I am just now learning how to use github and what repos are etc, so honeslty I didn’t even know that was an option.

Once I get realip functioning I will try to add proper license files and tie them back to the parent repo so others can see what I’ve been working on.

[Austin_Kirsch]

@francislavoie

Its working!!! Very happy and excited about it. I wouldnt have even attempted were it not for your suggestion so thank you. I will register the plugin so hopefully others can see it to benefit from.

Since so much code changed, I cant really add this as a ‘fork’ of the original plugin correct? What is the best way to get this out there? Ive seen quite a few posts asking about this so just want to make sure anyone who could use it or even make it better is aware.

[francislavoie]

The repo you have is fine. The attribution to the original implementation is enough. Forking is really just a button in github that makes a linked clone of a repo, it’s not something you have to do.

Adding it to the Download Caddy page is probably enough for visibility.

[Austin_Kirsch]

Understood. However, I am unable to register the module with the following error code:

4ac059fa-7e7e-4780-8eab-0427d1171297

Any ideas? @matt

[Austin_Kirsch]

@francislavoie

Hey, sorry to bother about this again. Hoping you may be able to help with an issue. So, it seems to be working OK and i’m not sure if this issue is just me having wrong expectations or if there is a fix. See log output below:

"level":"info","ts":1604254471.782191,
"logger":"http.log.access.log0",
"msg":"handled 
request",
"request":{
"remote_addr":"162.158.74.165:15622",
"proto":"HTTP/1.1",
"method":"GET",
"host":"greatwhitelab.net",
"uri":"/",
"headers":{"Cache-Control":["max-age=0"],
"Sec-Fetch-Dest":["document"],
"Cookie":["__cfduid=XXXXXXXXXXXXXXXXXX"],
"Cdn-Loop":["cloudflare"],
"Cf-Ray":["XXXXXXXXXXXXXXXXXX"],
"X-Forwarded-Proto":["https"],
"Cf-Visitor":["{\"scheme\":\"https\"}"],
"Upgrade-Insecure-Requests":["1"],
"User-Agent":["XXXXXXXXXXXXXXXXXX"],
"X-Forwarded-For":["<MY ACTUAL IP>"],
"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],
"Sec-Fetch-User":["?1"],
"If-Modified-Since":["Sat, 17 Oct 2020 04:21:41 GMT"],
"Accept-Language":["en-US,en;q=0.9"],
"Cf-Request-Id":["XXXXXXXXXXXXXXXXXX"],
"Cf-Connecting-Ip":["<MY ACTUAL IP>"],
"Connection":["Keep-Alive"],
"Accept-Encoding":["gzip"],
"Cf-Ipcountry":["US"],
"Sec-Fetch-Site":["none"],
"Sec-Fetch-Mode":["navigate"]},
"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","proto_mutual":true,"server_name":"greatwhitelab.net"}},
"common_log":"<MY ACTUAL IP> - - [01/Nov/2020:18:14:31 +0000] \"GET / HTTP/1.1\" 304 0","duration":0.000420325,"size":0,"status":304,"resp_headers":{"Referrer-Policy":["no-referrer-when-downgrade"],"Strict-Transport-Security":["max-age=31536000;"],"X-Frame-Options":["DENY"],"Etag":["XXXX"],"Server":["Caddy"],"X-Content-Type-Options":["nosniff"],"X-Xss-Protection":["1"]}}

So, initially “request>remote_addr” is incorrect because 162.158.74.165 is a cloudflare IP. However the common_log output shows the correct output with ? Is this because I am loading in the realip directive too late, if not how can I get the “request>remote_addr” value to be corrected like what common_log is giving me?

[francislavoie]

Ah, that’s because the LoggableHTTPRequest is created before the request is passed through the handlers, so any changes to the request aren’t reflected in the logs.

caddyserver/caddy/blob/937ec342010894f7a6239883b10f6a107ff82c9f/modules/caddyhttp/server.go#L157

The common_log field is created after the handlers though, because it needs the response size and status. So it does get the right value.

This isn’t a problem if all you care about is subsequent handlers having the right value, because they will; it’s just not updated in the logs.

I’m not sure there’s anything you can do in your plugin to fix the log output without a change to Caddy.

@matt what do you think?

[Austin_Kirsch]

I see. That makes sense.

Honestly, at this point I dont think what I want to do is even possible. Keeping in mind I’m learning as I go, but my intention has been to correct “request>remote_addr” so that IP tables can see the correct IP and block accordingly, but my entire approach is flawed because caddy is what I’m relying on to do the IP correction which only happens after the packet is passed through IP tables. So, I think I need to find another solution.

[Austin_Kirsch]

@francislavoie @matt

This will be my last post on this thread but thought I would keep you two in the loop incase this comes up in the future. I figured it out. Just needed to configure fail2ban to use cloudflare API to ban the actual client IP instead of using IP tables at all.

All is working just as I wanted at this point. Turns out, I didn’t even need to port “realip” since I was successfully using “format-encoder” to get my access logs corrected before starting. But regardless, it may be useful to others so I will keep up with it.

Thank you both for your help!

[francislavoie]

Cool!

Since you seem to have figured out how to use fail2ban, do you think you’d be interested in writing a wiki article explaining how to use it alongside Caddy?

There have been quite a few people asking how, but I don’t use it so I haven’t been able to point them in the right direction.

[Austin_Kirsch]

Most definitely! May take me a couple days to find the time to get it written out but I would love to do that. The more guidance out there the better.