Caddy with consul

[pipedev (Muritala David)]

Here’s a formatted issue for the Caddy GitHub repository regarding Consul integration:

1. The problem I’m having:

I’m trying to use Caddy as a reverse proxy with Consul service discovery. Specifically, I want to reverse proxy to a service registered in Consul using the service name (web.service.consul). When I try to use the Consul DNS name in my Caddyfile like this:

nginx.brimble.io {
    reverse_proxy web.service.consul
}

The proxy doesn’t work. I can confirm that the service is properly registered in Consul and resolvable via dig:

$ dig @localhost -p 8600 web.service.consul

;; ANSWER SECTION:
web.service.consul.     0       IN      A       X.X.X.138

2. Error messages and/or full log output:

2025/01/01 11:41:55 [ERROR] [nginx.brimble.io] failed to lookup service address: lookup web.service.consul: no such host
2025/01/01 11:41:55 [ERROR] [nginx.brimble.io] reverse proxy upstream could not be resolved: no addresses available

3. Caddy version:

$ caddy version
v2.7.6 h1:xxx

4. How I installed and ran Caddy:

a. System environment:

• OS: Ubuntu 22.04 LTS
• Architecture: amd64
• Consul version: 1.16.6

b. Command:

sudo systemctl start caddy

c. My complete Caddy config:

nginx.brimble.io {
    reverse_proxy {
        to web.service.consul:3333
    }
}

5. Links to relevant resources:

• DNS usage overview | Consul | HashiCorp Developer
• reverse_proxy (Caddyfile directive) — Caddy Documentation
• JSON Config Structure - Caddy Documentation

I’ve confirmed that:

1. The service is properly registered in Consul
2. The service is healthy (passing health checks)
3. I can resolve the service using dig @localhost -p 8600
4. I can curl the service directly using its IP:port

I’m looking for guidance on how to properly configure Caddy to work with Consul’s DNS service discovery. If there’s a better approach or a recommended plugin for this use case, I’d appreciate that information as well.

[timelordx]

While you can resolve the service name via dig by talking directly to the consul service (port 8600), have you tried to do the same by asking the OS resolver? Caddy is using the default OS resolver.

Make sure that your Caddy OS can resolve the name

$ dig web.service.consul

[pipedev (Muritala David)]

I fixed it by doing this

sudo mkdir -p /etc/systemd/resolved.conf.d/

sudo tee /etc/systemd/resolved.conf.d/consul.conf << EOF
[Resolve]
DNS=127.0.0.1:8600
Domains=~consul
EOF

sudo systemctl restart systemd-resolved

But on my caddyfile, i have to specify the port, the application is running on. How can i prevent this ?

nginx.brimble.io {
 reverse_proxy {
        to web.service.consul:3333
}
}

Because i can have the container deployed on a nomad cluster with 3 different ports

[Mohammed90 (Mohammed Al Sahaf)]

You can also use resolvers sub-directive of the http transport in reverse_proxy

reverse_proxy (Caddyfile directive) - Caddy Documentation

[pipedev (Muritala David)]

nginx.brimble.io {
    reverse_proxy {
        to web.service.consul
        transport http {
            resolvers 127.0.0.53
        }
    }
}

I did this, but no positive effect, i just got to the default caddy works! page

[Mohammed90 (Mohammed Al Sahaf)]

I don’t follow. Unless your upstream runs Caddy with the default page, then you’re not running the config you think you’re running.

@pipedev

i just got to the default caddy works! page

First, I see in some places you’re using web.service.consul while in others using web.service.consul:3333. Which one is it? A/AAAA lookups don’t provide port numbers. For that you’ll need SRV lookups.

Second, your top post shows you’re using @localhost -p 8600 to resolve the address. You can just configure Caddy with resolvers localhost:8600.

[pipedev (Muritala David)]

I now use this configuration

nginx.brimble.io {
    reverse_proxy {
        to web.service.consul
        transport http {
            resolvers localhost:8600
        }
    }
}

root@loadbalancer:/etc/caddy# dig @localhost -p 8600 web.service.consul SRV

; <<>> DiG 9.18.28-0ubuntu0.22.04.1-Ubuntu <<>> @localhost -p 8600 web.service.consul SRV
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8471
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 4
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;web.service.consul.            IN      SRV

;; ANSWER SECTION:
web.service.consul.     0       IN      SRV     1 1 3333 d1618a8a.addr.dc1.consul.

;; ADDITIONAL SECTION:
d1618a8a.addr.dc1.consul. 0     IN      A       X.X.X.138
server-1.node.dc1.consul. 0     IN      TXT     "consul-version=1.16.6"
server-1.node.dc1.consul. 0     IN      TXT     "consul-network-segment="

;; Query time: 20 msec
;; SERVER: ::1#8600(localhost) (UDP)
;; WHEN: Wed Jan 01 22:38:24 UTC 2025
;; MSG SIZE  rcvd: 191

But i keep having this page, but if i attach the port 3333. It resolves correctly