Caddy 2.5: Dynamic upstreams and consul SRV DNS

Help
1

Hi

We use srv+ upstream config (like it was described in Docs for upcoming v2.5.0 release (#216) · caddyserver/website@a1ddadf · GitHub) with consul.

Trying to migrate to v2.5 and the new dynamic upstream config failed so far.

loading upstream source module: loading module 'srv': provision http.reverse_proxy.upstreams.srv: address 172.17.0.1: missing port in address

What we’ve tried:

reverse_proxy {
  dynamic srv {
    name service.consul
    service prometheus
    proto http
    refresh 10s
    resolvers 172.17.0.1
    dial_timeout 1s
    dial_fallback_delay -1s
  }
}

DNS SRV record:

 dig  +short srv prometheus.service.consul
1 1 29346 c0a8029f.addr.isys.consul.

Our current working config:

reverse_proxy {
  to srv+http://prometheus.service.consul
  ...
}
2 Likes
2

What port is the resolver listening on? Make sure to add that to the resolver address.

3

Port 53

There is a local dnsmasq which allows to query consul dns with port 53. We also have a config for the docker containers to set the resolver:

dns_servers = [ "172.17.0.1" ]

We use this since while already in various services.

4

What Matt is saying, try resolvers 172.17.0.1:53.

I agree that it shouldn’t be necessary to explicitly put the port for resolvers, 53 should be assumed if not specified. But that’ll come in a later release.

1 Like
5

I guess that’s a good idea.

6

@matt @francislavoie
got it.

Seems this did the trick!

2 Likes
7
1 Like
8

Sorry to say but there is still an issue with reverse proxy:

adminer.example.com {
  route {
    authorize with mypolicy
    reverse_proxy {
      # to srv+http://adminer.service.consul
      dynamic srv {
        name service.consul
        service adminer
        proto tcp
        refresh 10s
        resolvers 172.17.0.1:53
        dial_timeout 1s
        dial_fallback_delay -1s
      }
    }
  }
}

results in:


{"level":"debug","ts":1651850474.5743804,"logger":"http.reverse_proxy.upstreams.srv","msg":"refreshing SRV upstreams","service":"adminer","proto":"tcp","name":"service.consul"}

{"level":"error","ts":1651850474.5781791,"logger":"http.handlers.reverse_proxy","msg":"failed getting dynamic upstreams; falling back to static upstreams","error":"lookup _adminer._tcp.service.consul on 172.17.0.1:53: server misbehaving"}

Then I tried:

adminer.example.com {
  route {
    reverse_proxy {
      # to srv+http://adminer.service.consul
      dynamic srv adminer.service.consul {
        refresh 10s
        resolvers 172.17.0.1:53
        dial_timeout 1s
        dial_fallback_delay -1s
      }
    }
  }
}

results in:

{"level":"debug","ts":1651851023.367271,"logger":"http.reverse_proxy.upstreams.srv","msg":"refreshing SRV upstreams","service":"","proto":"","name":"adminer.service.consul"}

{"level":"error","ts":1651851023.3705013,"logger":"http.handlers.reverse_proxy","msg":"failed getting dynamic upstreams; falling back to static upstreams","error":"lookup adminer.service.consul on 172.17.0.1:53: server misbehaving"}

not sure what is failing here. But what I want it to query is:

dig +short srv adminer.service.consul
1 1 22766 0a170c95.addr.preprod.consul.

I looked into the code base on the comments int he code, the second example should work, right?

9

Are you sure 172.17.0.1:53 reaches your DNS server?

Are you running in Docker or something? Docker might be using that IP range for its own containers, causing conflicts.

This doesn’t seem like an issue with Caddy specifically, that error is saying Caddy received an unexpected response.

10

It’s running in docker but I am pretty sure 172.17.0.1:53 is reachable (but I’ll verify again) Note, that we used srv+http://adminer.service.consul before which still works.

11

I checked and when I use an IP or port which does not exist:

{"level":"error","ts":1651852643.0701892,"logger":"http.handlers.reverse_proxy","msg":"failed getting dynamic upstreams; falling back to static upstreams","error":"lookup adminer.service.consul on 172.17.0.1:53: dial tcp 172.18.0.1:53: i/o timeout"}

or port nothing listens on:

{"level":"error","ts":1651852820.2347252,"logger":"http.handlers.reverse_proxy","msg":"failed getting dynamic upstreams; falling back to static upstreams","error":"lookup adminer.service.consul on 172.17.0.1:53: dial tcp 172.17.0.1:54: connect: connection refused"}

What I don’t understand is, why does it say “on 172.17.0.1:53:” ?

12

I got it, skipping the resolvers and it works!

adminer.example.com {
  route {
    reverse_proxy {
      dynamic srv {
        name adminer.service.consul
        refresh 10s
        dial_timeout 1s
        dial_fallback_delay -1s
      }
    }
  }
}

:person_facepalming:

3 Likes
13

This topic was automatically closed after 29 days. New replies are no longer allowed.