Caddy service failed due to following error

1. My Caddy version (caddy -version):

v1.0.3

2. How I run Caddy:

systemctl start caddy

a. System environment:

CentOS Linux release 7.7.1908
Intalled caddy by “yum install -y caddy”
Created conf at /etc/caddy/conf.d/[domain].conf
[domain] {
root /var/www/[domain]
}
Tried to run caddy as a system service
systemctl start caddy

b. Command:

systemctl start caddy

c. Service/unit/compose file:

[Unit]
Description=Caddy HTTP/2 web server
Documentation=https://caddyserver.com/docs
After=network.target

[Service]
User=caddy
Group=caddy
Environment=CADDYPATH=/var/lib/caddy
EnvironmentFile=-/etc/caddy/envfile
ExecStartPre=/usr/bin/caddy -conf /etc/caddy/caddy.conf -validate
ExecStart=/usr/bin/caddy -conf /etc/caddy/caddy.conf -root /tmp -agree
ExecReload=/usr/bin/kill -USR1 $MAINPID
KillMode=mixed
KillSignal=SIGQUIT
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectHome=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile:

[domain] {
  root /var/www/[domain]
}

3. The problem I’m having:

Failed to run caddy service

4. Error messages and/or full log output:

Nov 23 19:56:18 systemd[1]: Starting Caddy HTTP/2 web server…
Nov 23 19:56:18 systemd[1]: Started Caddy HTTP/2 web server.
Nov 23 19:56:18 caddy[19584]: Activating privacy features… 2019/11/23 19:56:18 get Agreement URL: Get https://acme-v02.api.letsencrypt.org/directory: dial tcp: lookup acme-v02.api.letsencrypt.org on [::1]:53: read udp [::1]:57972->[::1]:53: read: connection refused
Nov 23 19:56:18 systemd[1]: caddy.service: main process exited, code=exited, status=1/FAILURE
Nov 23 19:56:18 systemd[1]: Unit caddy.service entered failed state.
Nov 23 19:56:18 systemd[1]: caddy.service failed.

5. What I already tried:

systemctl restart caddy

6. Links to relevant resources:

N/A

Hi @CaddyOk, welcome to the Caddy community.

This part indicates that Caddy attempted to look up the IP address for acme-v02.api.letsencrypt.org by connecting to the local host ([::1] in IPv6) on port 53, which was refused. With no DNS, Caddy can’t connect to LetsEncrypt.

Why is Caddy trying to connect to the local host to resolve DNS? Not sure. Caddy uses DNS servers specified by the operating system, so make sure that’s all working as expected and you can nslookup acme-v02.api.letsencrypt.org without issues and it should be all good from there.

Hi @Whitestrake,
Thank you for your reply.
nslookup acme-v02.api.letsencrypt.org looks all good:

Server:		8.8.8.8
Address:	8.8.8.8#53

Non-authoritative answer:
acme-v02.api.letsencrypt.org	canonical name = prod.api.letsencrypt.org.
prod.api.letsencrypt.org	canonical name = ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com.
Name:	ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com
Address: 172.65.32.248
Name:	ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com
Address: 2606:4700:60:0:f53d:5624:85c7:3a2c

However, caddy service still fails because of the error.

Can you post your /etc/resolv.conf?

Sure

# Generated by SolusVM
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 2001:4860:4860::8888
nameserver 2001:4860:4860::8844

I’m stumped.

Golang’s standard net package should resolve DNS directly from those servers in your resolv.conf (see: https://golang.org/pkg/net/#hdr-Name_Resolution). Caddy doesn’t do DNS resolution manually for this stuff, it relies on the standard library.

But that error is saying it’s trying to connect to itself for DNS, which doesn’t make sense with that resolv.conf. Maybe it’s falling back to cgo for some reason - question is why would cgo try that?

Maybe try Environment=GODEBUG=netdns=go in your service unit file and try it again to explicitly force the default, see if you’re still getting this error.

1 Like

Just a guess: is it possible Caddy’s binding to a different network interface that’s set up to use different DNS?

1 Like

I don’t know how getaddrinfo and getnameinfo via cgo work - I assume it’s possible, but the default for Go programs using the net library is to use the servers listed in /etc/resolv.conf directly. So I expect it to ignore DNS servers configured per interface.

I worked around the issue by reinstalling caddy via curl https://getcaddy.com | bash -s personal, and followed instructions in here https://github.com/caddyserver/caddy/tree/master/dist/init/linux-systemd to setup service. This time the service went through that step without issues.

2 Likes