Crowdsec parsing caddy access logs and ignoring private (whitelisted) remote_ip

Help
1

1. Output of caddy version:

v2.6.2

2. How I run Caddy:

Caddy runs as a docker container on Ubuntu 22.02 virtualized host. The VM is installed on Proxmox running on a bare-metal NUC.

a. System environment:

Host OS is Ubuntu 22.04.1 LTS
Docker is v20.10.12

b. Command:

FROM caddy:builder AS builder

RUN xcaddy build \
    --with github.com/caddy-dns/cloudflare

FROM caddy:latest

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

docker-compose up -d

c. Service/unit/compose file:

version: '3.9'

services:
  cloudflared:
    image: cloudflare/cloudflared:latest
    container_name: cloudflared
    restart: unless-stopped
    networks:
      net-proxy:
        ipv4_address: 192.168.20.10
    command: tunnel run
    environment:
      TUNNEL_TOKEN: ${TUNNEL_TOKEN}

networks:
  net-proxy:
    external: true

version: '3.9'

services:
  caddy:
    image: caddy:cloudflare
    container_name: caddy
    restart: always
    networks:
      net-proxy:
        ipv4_address: 192.168.20.11
      net-macvlan:
        ipv4_address: 10.1.10.30
    ports:
      - 443:443
    environment:
      TZ: ${TZ}
      PUID: ${PUID}
      PGID: ${PGID}
      CLOUDFLARE_EMAIL: ${CLOUDFLARE_EMAIL}
      CLOUDFLARE_API_TOKEN: ${CLOUDFLARE_API_TOKEN}
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/log/caddy:/var/log/caddy
      - ${DOCKER_APP_DIR}/caddy/Caddyfile:/etc/caddy/Caddyfile
      - ${DOCKER_APP_DIR}/caddy/config:/config
      - ${DOCKER_APP_DIR}/caddy/data:/data

networks:
  net-proxy:
    external: true
  net-macvlan:
    external: true

version: '3.9'

x-crowdsec-common: &crowdsec-common
    restart: always
    security_opt:
        - no-new-privileges:true

x-crowdsec-environment: &crowdsec-environment
    TZ: ${TZ}

services:
    crowdsec:
        image: crowdsecurity/crowdsec:latest
        container_name: crowdsec
        <<: *crowdsec-common
        networks:
            net-proxy:
                ipv4_address: 192.168.20.2
            net-macvlan:
                ipv4_address: 10.1.10.33
        expose:
            - "8080" # API
            - "6060" # Metrics
        environment:
            <<: *crowdsec-environment
            COLLECTIONS: ${COLLECTIONS}
            CUSTOM_HOSTNAME: ${CUSTOM_HOSTNAME}
            GID: ${GID}
        volumes:
            - ${LOCALTIME_DIR}:/etc/localtime:ro
            - ${HOST_LOGS}:/var/log:ro
            - ${DOCKER_APP_DIR}/crowdsec/data:/var/lib/crowdsec/data
            - ${DOCKER_APP_DIR}/crowdsec/config:/etc/crowdsec

    cloudflare-bouncer:
        image: crowdsecurity/cloudflare-bouncer:latest
        container_name: cloudflare-bouncer
        <<: *crowdsec-common
        networks:
            net-proxy:
                ipv4_address: 192.168.20.3
        environment:
            <<: *crowdsec-environment
        volumes:
            - ${LOCALTIME_DIR}:/etc/localtime:ro
            - ${DOCKER_APP_DIR}/crowdsec/cloudflare-bouncer/config.yaml:/etc/crowdsec/bouncers/crowdsec-cloudflare-bouncer.yaml

networks:
    net-proxy:
        external: true
    net-macvlan:
        external: true

d. My complete Caddy config:

## Caddyfile

## Global Parameters
{
        email [redacted]
}

## Snippets
(cloudflare) {
        tls {
                protocols tls1.3
                dns cloudflare {env.CLOUDFLARE_API_TOKEN}
        }
}

(trusted) {
        trusted_proxies \
 173.245.48.0/20 \
 103.21.244.0/22 \
 103.22.200.0/22 \
 103.31.4.0/22 \
 141.101.64.0/18 \
 108.162.192.0/18 \
 190.93.240.0/20 \
 188.114.96.0/20 \
 197.234.240.0/22 \
 198.41.128.0/17 \
 162.158.0.0/15 \
 104.16.0.0/13 \
 104.24.0.0/14 \
 172.64.0.0/13 \
 131.0.72.0/22 \
 10.0.0.0/8 \
 172.16.0.0/12 \
 192.168.0.0/16
}

## Reverse Proxy Hosts

mealie.ws.house {
        import cloudflare

        log {
                output file /var/log/caddy/mealie-access.log
        }

        reverse_proxy mealie:3000 {
                import trusted
        }
}

3. The problem I’m having:

I’ve recently migrated to caddy after being a long-time NPM user. I’ve installed crowdsec as a docker container on the same network as caddy (network, net-proxy), and am having issues with crowdsec parsing my caddy access logs. Crowdsec has /var/log/caddy/*.log configured in acquis.yaml as type = caddy as shown below:

filenames:
 - /var/log/caddy/*.log
labels:
  type: caddy

All external traffic is routed through cloudflared tunnel → caddy → internal service. All access logs show “remote_ip” as my local cloudflared (192.168.20.10), instead of the actual client. As a result, crowdsec sees 192.168.0.0/16 as “whitelisted”, and doesn’t yield any alerts. Below are logs from accessing https://mealie.ws.house from my personal device with an actual IP of 107.127.28.144.

4. Error messages and/or full log output:

Output of /var/log/caddy/mealie-access.log

{"level":"error","ts":1673185496.8189387,"logger":"http.log.access.log6","msg":"handled request","request":{"remote_ip":"192.168.20.10","remote_port":"43890","proto":"HTTP/2.0","method":"POST","host":"mealie.ws.house","uri":"/api/auth/token","headers":{"X-Forwarded-Proto":["https"],"Referer":["https://mealie.ws.house/login"],"Cookie":[],"Accept-Language":["en-US"],"Cf-Ipcountry":["US"],"Cf-Ray":["78655ee99e522275-MIA"],"X-Forwarded-For":["107.127.28.144"],"Cf-Connecting-Ip":["107.127.28.144"],"Content-Type":["multipart/form-data; boundary=----WebKitFormBoundary4Zw8j8KFpAv8Z2JE"],"Origin":["https://mealie.ws.house"],"Accept-Encoding":["gzip"],"User-Agent":["Mozilla/5.0 (iPhone; CPU iPhone OS 16_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Mobile/15E148 Safari/604.1"],"Content-Length":["352"],"Cf-Warp-Tag-Id":["646dcab4-6371-4810-a227-98b5dfe2438a"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept":["application/json, text/plain, */*"],"Cdn-Loop":["cloudflare"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"mealie.ws.house"}},"user_id":"","duration":0.173152772,"size":45,"status":401,"resp_headers":{"Vary":["Accept-Encoding"],"Server":["Caddy","Caddy","uvicorn"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Content-Encoding":["gzip"],"Content-Type":["application/json"],"Date":["Sun, 08 Jan 2023 13:44:55 GMT"]}}

If I exec into crowdsec and run:

cscli --file /var/log/caddy/mealie-access.log --type caddy --verbose

I get the following output:

line: {"level":"error","ts":1673185218.5674167,"logger":"http.log.access.log6","msg":"handled request","request":{"remote_ip":"192.168.20.10","remote_port":"38900","proto":"HTTP/2.0","method":"POST","host":"mealie.ws.house","uri":"/api/auth/token","headers":{"Cf-Connecting-Ip":["107.127.28.144"],"User-Agent":["Mozilla/5.0 (iPhone; CPU iPhone OS 16_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Mobile/15E148 Safari/604.1"],"Referer":["https://mealie.ws.house/login"],"Origin":["https://mealie.ws.house"],"Cf-Ray":["7865581eccdd67bd-MIA"],"Cdn-Loop":["cloudflare"],"Content-Length":["362"],"Accept-Encoding":["gzip"],"Accept-Language":["en-US"],"X-Forwarded-Proto":["https"],"Cf-Ipcountry":["US"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["646dcab4-6371-4810-a227-98b5dfe2438a"],"Content-Type":["multipart/form-data; boundary=----WebKitFormBoundaryKgU09JjZEFVn8iOL"],"X-Forwarded-For":["107.127.28.144"],"Cookie":[],"Accept":["application/json, text/plain, */*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"mealie.ws.house"}},"user_id":"","duration":0.175207567,"size":45,"status":401,"resp_headers":{"Content-Encoding":["gzip"],"Content-Type":["application/json"],"Vary":["Accept-Encoding"],"Date":["Sun, 08 Jan 2023 13:40:17 GMT"],"Server":["Caddy","Caddy","uvicorn"],"Alt-Svc":["h3=\":443\"; ma=2592000"]}}
        ├ s00-raw
        |       ├ 🔴 crowdsecurity/docker-logs
        |       ├ 🟢 crowdsecurity/non-syslog (first_parser)
        |       └ 🔴 crowdsecurity/syslog-logs
        ├ s01-parse
        |       └ 🟢 crowdsecurity/caddy-logs (+19 ~2)
        |               └ update evt.Stage : s01-parse -> s02-enrich
        |               └ create evt.Parsed.request : /api/auth/token
        |               └ create evt.Parsed.tz : GMT
        |               └ create evt.Parsed.year : 2023
        |               └ create evt.Parsed.timestamp : Sun, 08 Jan 2023 13:40:17 GMT
        |               └ create evt.Parsed.day : Sun
        |               └ create evt.Parsed.http_user_agent : Mozilla/5.0 (iPhone; CPU iPhone OS 16_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Mobile/15E148 Safari/604.1
        |               └ create evt.Parsed.month : Jan
        |               └ create evt.Parsed.monthday : 08
        |               └ create evt.Parsed.remote_ip : 192.168.20.10
        |               └ create evt.Parsed.time : 13:40:17
        |               └ create evt.Parsed.verb : POST
        |               └ update evt.StrTime :  -> Sun Jan 08 13:40:17.000000 2023
        |               └ create evt.Meta.log_type : http_access-log
        |               └ create evt.Meta.target_fqdn : mealie.ws.house
        |               └ create evt.Meta.http_path : /api/auth/token
        |               └ create evt.Meta.http_status : 401
        |               └ create evt.Meta.http_user_agent : Mozilla/5.0 (iPhone; CPU iPhone OS 16_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Mobile/15E148 Safari/604.1
        |               └ create evt.Meta.http_verb : POST
        |               └ create evt.Meta.service : http
        |               └ create evt.Meta.source_ip : 192.168.20.10
        ├ s02-enrich
        |       ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~3 [whitelisted])
        |               ├ update evt.Whitelisted : %!s(bool=false) -> true
        |               ├ update evt.WhitelistReason :  -> My IP Ranges
        |               ├ create evt.Enriched.MarshaledTime : 2023-01-08T13:40:17Z
        |               ├ update evt.MarshaledTime :  -> 2023-01-08T13:40:17Z
        |               ├ create evt.Meta.timestamp : 2023-01-08T13:40:17Z
        |       ├ 🟢 crowdsecurity/geoip-enrich (+9)
        |               ├ create evt.Enriched.ASNumber : 0
        |               ├ create evt.Enriched.IsInEU : false
        |               ├ create evt.Enriched.IsoCode :
        |               ├ create evt.Enriched.Latitude : 0.000000
        |               ├ create evt.Enriched.Longitude : 0.000000
        |               ├ create evt.Enriched.ASNNumber : 0
        |               ├ create evt.Enriched.ASNOrg :
        |               ├ create evt.Meta.ASNNumber : 0
        |               ├ create evt.Meta.IsInEU : false
        |       ├ 🟢 crowdsecurity/http-logs (+7)
        |               ├ create evt.Parsed.file_frag : token
        |               ├ create evt.Parsed.file_name : token
        |               ├ create evt.Parsed.impact_completion : true
        |               ├ create evt.Parsed.file_dir : /api/auth/
        |               ├ create evt.Parsed.file_ext :
        |               ├ create evt.Parsed.static_ressource : false
        |               ├ create evt.Meta.http_args_len : 0
        |       └ 🟢 crowdsecurity/whitelists (~1)
        |               └ update evt.WhitelistReason : My IP Ranges -> private ipv4/ipv6 ip/ranges
        └-------- parser failure 🔴

5. What I already tried:

Note, create evt.Parsed.remote_ip and create evt.Meta.source_ip both show my cloudflared local IP of 192.168.20.10. Within crowdsec config.yaml, I’ve set use_forwarded_for_headers: true as described here, but it made no difference.

Is there something I can change in my Caddyfile to make remote_ip reflect the correct X-Forwarded-For client IP, or should I pursue ip_mask for remote_ip to strip from the logs? Thank you for the assistance.

6. Links to relevant resources:

1 Like
2

There’s a crowdsec plugin for Caddy. Is that what you’re looking for?

I don’t use Crowdsec, and it’s not a part of Caddy, so I can’t really give support for it.

1 Like
3

Hi @Jyncus,

I believe the issue is due to the Caddy parser in Crowdsec not reading the X-Forwarded-For headers to obtain source/client IP addresses, similar to the issue I’ve raised here: Problem with Caddy parser when Caddy is behind Cloudflare Proxy · Issue #633 · crowdsecurity/hub · GitHub.

In this issue LaurenceJJones confirmed the use_forwarded_for_headers setting is for the Crowdsec HTTP api only, and does not apply to parsers.

To resolve this we’ll need Crowdsec to implement support for the X-Forwarded-For or CF-Connecting-IP header in the parser, or Caddy to implement an option to set X-Forwarded-For or CF-Connecting-IP as the remote_ip.

4

That’s not possible, because the remote IP is the actual IP address on the TCP packets.

We don’t currently have support for prepending Proxy Protocol bytes as a reverse_proxy client, but that is something we’ll probably eventually support. I’d be surprised if Crowdsec supports that but not the XFF header though.

If proxying to Crowdsec (and not using the plugin I linked), them supporting XFF is the necessary solution to this.

5

This topic was automatically closed after 30 days. New replies are no longer allowed.