Multiple problems after doing change from "gui simply Port forward" to nat/dnat rules with Firewall policies

1. Output of caddy version:

v2.5.2 h1:eCJdLyEyAGzuQTa5Mh3gETnYWDClo1LjtQm2q9RNZrs=

2. How I run Caddy:

Running WinSW as service

a. System environment:

Windows 11

b. Command:

caddy.exe

d. My complete Caddy config:

domain1.com {
        header X-Real-IP {http.request.header.CF-Connecting-IP}
        header X-Forwarded-For {http.request.header.CF-Connecting-IP}
        header X-Forwarded-Host {http.request.hostport}

        reverse_proxy ip:port ip:port ip:port ip:port {
                # streaming
                flush_interval -1

                # load balancing
                lb_policy least_conn
                lb_try_interval 250ms

                # passive health checking
                max_fails 2
                unhealthy_request_count 4

                transport http {
                        dial_timeout 3s
                        keepalive_idle_conns_per_host 4
                }
        }

        encode zstd gzip
        tls mail@mail.com {
                dns cloudflare KEY
                resolvers x.x.x.x # pi-hole
        }
        log {
                format transform [{ts}] - User={user_id} - X-Forwarded-For={request>headers>X-Forwarded-For} - remote_ip={request>remote_ip} Country={request>headers>Cf-Ipcountry} {request>method} {request>headers>X-Forwarded-Proto} {request>host} {request>uri} {request>headers>Referer>[0]} {request>headers>User-Agent>[0]} - {request>proto} {status} {size} - {request>headers} {
                        time_format "02/Jan/2006 15:04:05 -0700"
                }

                output file C:\stuff\caddy\logs\domain.log {
                        roll true # Rotate logs, enabled by default
                        roll_size_mb 5 # Set max size 5 MB
                        roll_gzip true # Whether to compress rolled files
                        roll_keep 2 # Keep at most 2 log files
                        roll_keep_days 7 # Keep log files for 7 days
                }
        }

        @notLocal {
                not remote_ip 192.168.1.0/24
                #Reads Ip from --- header X-Forwarded-For {http.request.header.CF-Connecting-IP}
                not remote_ip forwarded x.x.x.x/public wan ip
        }
        basicauth @notLocal {
                USER PASS
        }
}

http://domain2.com {
        reverse_proxy http://192.168.1.4:9117 {
        }
        encode zstd gzip

        @notLocal {
                not remote_ip 192.168.1.0/24
        }
        basicauth @notLocal {
                USER PASS
        }
}

3. The problem I’m having and What I already tried.

The initiate problem was when I use the Port Forwarding section rules under my router with edgeOS ER12p would get by a lot of bad actors! even after setting up a firewall rule to block them, but more about this later - everything else worked really great and was hassle free and easy to setup more below this picture about this.

I would get hit by a lot of request from various bots,bad ips. So I setup a firewall with daily refreshed blacklisted IP’s, but soon learned that when you use the above settings in the picture, the rules in here will ignore or overrule everything else under Firewall policies and masq/nat/dnat settings under boot/startup

So I started trial and error setting up my router via the nat and dnat section with port forwarding and removing the initiate rules under the default page “Port Forwarding” section which now has most of it up and running.

First I had to add source nat masquerade for lan under source nat

Then I had to setup dnat for wan http/https for port 80,443.

I also added Hairpin LAN dnat rules for 80,443 as I’m inexperience and a noob, don’t do that unless you to try to host the entire internet, you will block your self out from everything, that’s why you can see on the screenshot they are greyed out/disable, luckily I could ssh in and delete the rules and again access to the web again :slight_smile:

Last I had to setup go to Firewall Policies sections and setup for the nat/dnat rule setup I just did, so added 80,443 as shown in the 2 pictures, below the blacklist firewall I created before everything else.

So far all seems to be working from the outside wan to in lan

The rule issues start now when I have setup caddy with cloudflare for incoming connections at domain1dotcom, before the remote_ip would show cloudflare IP now it shows my router IP - 192.168.1.1 in the logs which gives all access to my stuff because I gave the permission as shown in the caddyfile for 192.168.1.0/24

The second issue is that I can no longer go to my domain2dotcom without getting redirected to my router ip and saying certificate has expired, where it also worked without problem before just hitting domain the and it would work as intended.

If I add both ports 80,443 back to the Port Forwarding section all these issues goes away but then I’m back to original problem by bots not getting blocked, so its properly just the matter of something in nat/dnat routes that needs to fixed up?

Domain2dotcom is my running local from the same server as caddy and runs on port 80, caddy does auto https it, I tried disabling this just to test, but it failed no matter what.

I’m really annoyed by this and would love some expert help, and by the history I have in from past topics in here, you guys always seem to have a guide or instructions ready with big expertise.

4. Reference - info & Links

My Router EdgeRouter 12P with latest firmware available v2.0.9-hotfix.4

The setup page wan port running on eth9 and ports 0-7 is in bridge mode.

Routes

I also just found out that all my incoming clients get my router ip 192.168.1.1 so something is up and maybe can explain the reason for one the problems, is it my masquerade that’s setup wrong? or whats going on :S

So I found the cause to all incoming ip addresses getting 192.168.1.1.

The reason being missing filtering source address, after adding 192.168.1.0/24, all clients show their real ip again when connecting to me.

Now the certificate and rerouting from caddy is still an issue.

Hi :slight_smile:

There is a lot to unpack here, and it isn’t immediately clear to me what problems you have successfully fixed yourself, and which remain :thinking:

Also, redacting domain usually makes helping way more time intensive and more difficult.
If you absolutely have to redact a domain name, use example.com.
Just keep in mind, that domain names are public information.
Your domain is public, and any certificate issued for your domain (by a trusted CA that is) will be put in the public https://certificate.transparency.dev/

That being said, could you please provide some curl outputs and say what you would like to see there instead?

An example request/response would be:

❯ curl https://example.com --head
HTTP/2 200 
content-encoding: gzip
accept-ranges: bytes
age: 205485
cache-control: max-age=604800
content-type: text/html; charset=UTF-8
date: Thu, 11 Aug 2022 17:06:08 GMT
etag: "3147526947"
expires: Thu, 18 Aug 2022 17:06:08 GMT
last-modified: Thu, 17 Oct 2019 07:18:26 GMT
server: ECS (mic/9ABC)
x-cache: HIT
content-length: 648

Pretty helpful options include --head, --include and --verbose.
--head does a HEAD requests and only prints the headers, page content.
This is usually good enough.
--include does a usual GET request and prints both headers and page content.
--verbose is very helpful to see how curl tries to connect, negotiates and shows certificate information (if https://).

2 Likes

Thanks for the reply, I have more or less gotten the problems solved for both issues.

My solution for 1 part in comment

Fixed the first issued

The second issued was fixed by doing this by adding wan static ip to this rule for http/https on lan side.
https://i.imgur.com/hDNDWVp.png

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.