Path_regexp + rewrite + dynamic SRV CoreDns

Help
1

1. The problem I’m having:

I want to achieve below:

  1. listen for incoming requests and perform redirection based on a specific path pattern (e.g., /redirect/{subdomain}/remaining_path ).
  2. capture the {subdomain} parameter and {remaining_path} parameter from the path.
  3. Construct the redirection target URL by appending .internal and {remaining_path} parameter to the captured {subdomain} . for example: subdomain.internal/remaining_path
  4. resolve Domain name using SRV record from custom resolver(coredns)

Despite what the Caddy logs indicate, I’m noticing requests in the CoreDNS. It seems that Caddy is trying to retrieve SRV records. However, I’m uncertain why it’s not capturing accurate groups and transmitting them as dynamic configuration when requesting SRV records.

When I attempt a request using the format: http://my_ip:8060/redirect/ff3723fe-cccf-444f-8ac4-0dcb7adb38a7/test, my anticipation was that Caddy would request SRV records for ff3723fe-cccf-444f-8ac4-0dcb7adb38a7.internal. However, it appears that this is not happening as expected.

2. Error messages and/or full log output:

2023/08/28 19:46:35.874 DEBUG   http.handlers.rewrite   rewrote request {"request": {"remote_ip": "49.36.80.36", "remote_port": "60245", "client_ip": "49.36.80.36", "proto": "HTTP/1.1", "method": "GET", "host": "198.203.31.13:8060", "uri": "/redirect/ff3723fe-cccf-444f-8ac4-0dcb7adb38a7/ss", "headers": {"Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"], "Accept-Language": ["en-GB,en-US;q=0.9,en;q=0.8"], "Cache-Control": ["no-cache"], "Upgrade-Insecure-Requests": ["1"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36"], "Accept-Encoding": ["gzip, deflate"], "Connection": ["keep-alive"], "Pragma": ["no-cache"]}}, "method": "GET", "uri": "/"}
2023/08/28 19:46:35.874 DEBUG   http.reverse_proxy.upstreams.srv        refreshing SRV upstreams   {"service": "", "proto": "", "name": "_._tcp.internal"}
2023/08/28 19:46:35.875 ERROR   http.handlers.reverse_proxy     failed getting dynamic upstreams; falling back to static upstreams  {"error": "lookup _._tcp.internal on 127.0.0.53:53: no such host"}
2023/08/28 19:46:35.876 ERROR   http.log.error.log0     no upstreams available  {"request": {"remote_ip": "49.36.80.36", "remote_port": "60245", "client_ip": "49.36.80.36", "proto": "HTTP/1.1", "method": "GET", "host": "198.203.31.13:8060", "uri": "/redirect/ff3723fe-cccf-444f-8ac4-0dcb7adb38a7/ss", "headers": {"Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"], "Accept-Language": ["en-GB,en-US;q=0.9,en;q=0.8"], "Cache-Control": ["no-cache"], "Upgrade-Insecure-Requests": ["1"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36"], "Accept-Encoding": ["gzip, deflate"], "Connection": ["keep-alive"], "Pragma": ["no-cache"]}}, "duration": 0.001905414, "status": 503, "err_id": "f5ah3ij6q", "err_trace": "reverseproxy.(*Handler).proxyLoopIteration (reverseproxy.go:483)"}

Core dns logs:

[INFO] 198.203.31.13:48093 - 18307 "SRV IN _._tcp.internal. udp 44 false 1232" NOERROR qr,aa,rd 103 0.000185916s
[INFO] 198.203.31.13:57968 - 38864 "SRV IN _._tcp.internal.localdomain. udp 56 false 1232" NOERROR qr,aa,rd 139 0.000144252s
[INFO] 198.203.31.13:54471 - 25947 "SRV IN _._tcp.internal. udp 44 false 1232" NOERROR qr,aa,rd 103 0.000310483s
[INFO] 198.203.31.13:51896 - 44105 "SRV IN _._tcp.internal.localdomain. udp 56 false 1232" NOERROR qr,aa,rd 139 0.000162299s
[INFO] 198.203.31.13:33850 - 15057 "SRV IN _._tcp.internal. udp 44 false 1232" NOERROR qr,aa,rd 103 0.000297655s
[INFO] 198.203.31.13:52474 - 58465 "SRV IN _._tcp.internal.localdomain. udp 56 false 1232" NOERROR qr,aa,rd 139 0.000229997s
[INFO] 198.203.31.13:43975 - 59738 "SRV IN _._tcp.internal. udp 44 false 1232" NOERROR qr,aa,rd 103 0.000225959s
[INFO] 198.203.31.13:56601 - 6942 "SRV IN _._tcp.internal.localdomain. udp 56 false 1232" NOERROR qr,aa,rd 139 0.000138709s

3. Caddy version:

v2.7.4

4. How I installed and ran Caddy:

a. System environment:

Ubuntu

b. Command:

/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

C. My complete Caddy config:

{
        debug
}
:8060 {
        log {
                output file /var/log/caddy/access.log
        }
        @capture path_regexp capture /redirect/([0-9a-zA-Z-]+)(?:/(.*))?$
        rewrite /redirect/* /{re.capture.2}
        reverse_proxy {
                dynamic srv _{re.capture.1}._tcp.internal {
                        resolvers udp/198.203.31.13:53
                }
                transport http {
                        resolvers udp/198.203.31.13:53
                }
        }
}

D. My complete CoreDns config:

.:53 {
    errors
    health {
        lameduck 5s
    }
    ready
    template ANY ANY {
        rcode NXDOMAIN
        policy up
        answer ""
    }
    log
    hosts {

ff3723fe-cccf-444f-8ac4-0dcb7adb38a7.internal 198.232.23.23
_ff3723fe-cccf-444f-8ac4-0dcb7adb38a7._tcp.service.local. 3600 IN SRV 10 10 2500 ff3723fe-cccf-444f-8ac4-0dcb7adb38a7.internal.
    }
    forward . /etc/resolv.conf
    cache
    reload 5s
}
2

The problem is you’re never actually using your @capture matcher on any handlers. Did you mean to use it with rewrite?

@capture path_regexp capture /redirect/([0-9a-zA-Z-]+)(?:/(.*))?$
rewrite @capture /{re.capture.2}

The problem would be obvious if you adapt your config to JSON, you’ll see your matcher isn’t in the final JSON config at all. Run caddy adapt --pretty --config Caddyfile to see.

1 Like
3

@francislavoie, referring to the documentation, the rewrite syntax is as follows: rewrite matcher to

In my case, I’m using the following rewrite: rewrite /redirect/* /{re.capture.2}

This should match “/redirect/*” and replace it with the “remaining_path” captured in the second group of the named matcher “@capture”.

Is my understanding correct?

1 Like
4

@capture is your matcher, use that instead of /redirect/*. See Request matchers (Caddyfile) — Caddy Documentation

If you declare a named matcher but never use it, it never exists in the final config. It needs to actually be used somewhere for it to do anything at all.

1 Like
5

@francislavoie, I appreciate your assistance. It was incredibly helpful, and now things seem to be functioning as expected. However, I’m still encountering error logs related to the resolver. Oddly, I’m receiving requests in CoreDNS, so I’m perplexed as to why the error is referencing the local nameserver instead of the actual nameserver being utilized.

2023/08/29 05:15:29.552 DEBUG   http.handlers.rewrite   rewrote request {"request": {"remote_ip": "49.36.80.36", "remote_port": "52603", "client_ip": "49.36.80.36", "proto": "HTTP/1.1", "method": "GET", "host": "198.203.31.13:8060", "uri": "/redirect/ff3723fe-cccf-444f-8ac4-0dcb7adb38a7/sas", "headers": {"Accept-Language": ["en-GB,en-US;q=0.9,en;q=0.8"], "Connection": ["keep-alive"], "Cache-Control": ["max-age=0"], "Upgrade-Insecure-Requests": ["1"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"], "Accept-Encoding": ["gzip, deflate"]}}, "method": "GET", "uri": "/sas"}
2023/08/29 05:15:29.552 DEBUG   http.reverse_proxy.upstreams.srv        refreshing SRV upstreams   {"service": "", "proto": "", "name": "_ff3723fe-cccf-444f-8ac4-0dcb7adb38a7._tcp.tabless.internal"}
2023/08/29 05:15:29.554 ERROR   http.handlers.reverse_proxy     failed getting dynamic upstreams; falling back to static upstreams  {"error": "lookup _ff3723fe-cccf-444f-8ac4-0dcb7adb38a7._tcp.tabless.internal on 127.0.0.53:53: no such host"}
2023/08/29 05:15:29.554 ERROR   http.log.error.log0     no upstreams available  {"request": {"remote_ip": "49.36.80.36", "remote_port": "52603", "client_ip": "49.36.80.36", "proto": "HTTP/1.1", "method": "GET", "host": "198.203.31.13:8060", "uri": "/redirect/ff3723fe-cccf-444f-8ac4-0dcb7adb38a7/sas", "headers": {"Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"], "Accept-Encoding": ["gzip, deflate"], "Accept-Language": ["en-GB,en-US;q=0.9,en;q=0.8"], "Connection": ["keep-alive"], "Cache-Control": ["max-age=0"], "Upgrade-Insecure-Requests": ["1"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36"]}}, "duration": 0.00167631, "status": 503, "err_id": "fqsbphdk0", "err_trace": "reverseproxy.(*Handler).proxyLoopIteration (reverseproxy.go:483)"}
1 Like
6

This is an artifact of the Go runtime, the language which Caddy is written in. The runtime consults the server specified in /etc/resolv.conf regardless of the resolver address we pass to the Go runtime. I’d argue this is a bug in the Go runtime, but we’ll have to take it with the Go team and hear from them.

I’m surprised it’s only come up now after so long! The only workaround I can think of is to reverse the logic in your Corefile, so resolv.conf points at CoreDNS, then CoreDNS takes falling requests to the upstream which the earlier resolv.conf used to point at.

1 Like
7

@Mohammed90, I’m equally surprised by the situation. After rectifying the capture named group, my Caddy requests are being redirected as intended. It seems the runtime is using the specified resolver address. However, the error log still persists, referencing the server specified in /etc/resolv.conf.

Regarding the workaround you mentioned:

The only workaround I can think of is to reverse the logic in your Corefile, so resolv.conf points at CoreDNS, then CoreDNS takes failing requests to the upstream which the earlier resolv.conf used to point at.

Could you clarify if you were suggesting updating the resolv.conf within the Caddy container to include the nameserver of CoreDNS?

1 Like
8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.