Caddy service enters failed state

1. Caddy version (caddy version):

v2.4.6 h1:HGkGICFGvyrodcqOOclHKfvJC0qTU7vny/7FhYp9hNw=

2. How I run Caddy:

systemctl start caddy (Failed)
caddy start (Success)

a. System environment:

Centos7

b. Command:

systemctl start caddy (Failed)
caddy start (Success)

c. Service/unit/compose file:

caddy.service

#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

aaa.aaa{
        reverse_proxy 127.0.0.1:29584 {
        }
}

aaa.aaa.aaa{
        reverse_proxy 127.0.0.1:12580 {
        }
}


aaa.aaa.aaa{
        reverse_proxy aaa.aaa.aaa.aaa:aaa{
                header_up Host aaa.aaa.aaa.aaa
        }
}

aaa.aaa.aaa{
        reverse_proxy aaa.aaa.aaa.aaa:aaa{
                header_up Host aaa.aaa.aaa.aaa
        }
}


aaa.aaa.aaa{
        tls /aaa/aaa/aaa/aaa/server.pem /aaa/aaa/aaa/https/server.key
        reverse_proxy /jquery-3.3* https://127.0.0.1:8443 {
               transport http {
                        tls
                        tls_insecure_skip_verify
                }
        }
}

3. The problem I’m having:

I’m trying to start caddy service by using systemctl start caddy, when caddy is initally installed, the default Caddyfile fits well and the caddy service lauched correctly, but when I use the Caddyfile above, the caddy service couldn’t launch. However, using the same command mentioned in the file caddy.service goes well.

4. Error messages and/or full log output:

systemctl status caddy -l goes like this:

● caddy.service - Caddy
   Loaded: loaded (/usr/lib/systemd/system/caddy.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2022-01-13 13:23:56 CST; 14min ago
     Docs: https://caddyserver.com/docs/
  Process: 1312 ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile (code=exited, status=1/FAILURE)
 Main PID: 1312 (code=exited, status=1/FAILURE)

Jan 13 13:23:56 important-pony-1.localdomain caddy[1312]: runtime.GOARCH=amd64
Jan 13 13:23:56 important-pony-1.localdomain caddy[1312]: runtime.Compiler=gc
Jan 13 13:23:56 important-pony-1.localdomain caddy[1312]: runtime.NumCPU=2
Jan 13 13:23:56 important-pony-1.localdomain caddy[1312]: runtime.GOMAXPROCS=2
Jan 13 13:23:56 important-pony-1.localdomain caddy[1312]: runtime.Version=go1.16.8
Jan 13 13:23:56 important-pony-1.localdomain caddy[1312]: os.Getwd=/
Jan 13 13:23:56 important-pony-1.localdomain systemd[1]: Started Caddy.
Jan 13 13:23:56 important-pony-1.localdomain systemd[1]: caddy.service: main process exited, code=exited, status=1/FAILURE
Jan 13 13:23:56 important-pony-1.localdomain systemd[1]: Unit caddy.service entered failed state.
Jan 13 13:23:56 important-pony-1.localdomain systemd[1]: caddy.service failed.

I tried the /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile mentioned above and this time everything goes well:

2022/01/13 05:39:34.312 INFO    using provided configuration    {"config_file": "/etc/caddy/Caddyfile", "config_adapter": ""}
2022/01/13 05:39:34.316 WARN    input is not formatted with 'caddy fmt' {"adapter": "caddyfile", "file": "/etc/caddy/Caddyfile", "line": 11}
2022/01/13 05:39:34.319 INFO    admin   admin endpoint started  {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2022/01/13 05:39:34.322 WARN    tls     stapling OCSP   {"error": "no OCSP stapling for [cloudflare origin certificate *.aaa.aaa aaa.aaa]: no URL to issuing certificate"}
2022/01/13 05:39:34.323 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc00037e070"}
2022/01/13 05:39:34.324 INFO    http    server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server {"server_name": "srv1", "http_port": 80}
2022/01/13 05:39:34.326 INFO    http    skipping automatic certificate management because one or more matching certificates are already loaded  {"domain": "aaa.aaa.aaa", "server_name": "srv0"}
2022/01/13 05:39:34.326 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2022/01/13 05:39:34.327 INFO    tls     cleaning storage unit   {"description": "FileStorage:/root/.local/share/caddy"}
2022/01/13 05:39:34.327 INFO    http    enabling automatic TLS certificate management   {"domains": ["aaa.aaa.aaa", "aaa.aaa"]}
2022/01/13 05:39:34.331 INFO    tls     finished cleaning storage units
2022/01/13 05:39:34.346 INFO    autosaved config (load with --resume flag)      {"file": "/root/.config/caddy/autosave.json"}
2022/01/13 05:39:34.346 INFO    serving initial configuration

5. What I already tried:

I’ve tried reinstall caddy and I found the default given Caddyfile fits well with the systemctl command , I guess there’s something wrong with my Caddyfile but I couldn’t figure out why it happened. Any hint would be appreicated.

This is strange. How did you install Caddy? What are the permissions on /etc/caddy/Caddyfile? Run ls -al /etc/caddy and report the output.

➜  caddy ls -al /etc/caddy        
total 16
drwxr-xr-x   2 root root 4096 Jan 13 13:37 .
drwxr-xr-x. 86 root root 4096 Jan 13 13:12 ..
-rw-r--r--   1 root root  619 Jan 13 13:37 Caddyfile
-rw-r--r--   1 root root  618 Jan 13 12:38 Caddyfile.rpmsave

I install caddy via yum, just yum install caddy

Those look fine. What were the full logs? Run journalctl -r -u caddy and share the full output

-- Logs begin at Thu 2022-01-13 13:12:14 CST, end at Fri 2022-01-14 09:45:57 CST. --
Jan 14 09:45:36 important-pony-1.localdomain systemd[1]: caddy.service failed.
Jan 14 09:45:36 important-pony-1.localdomain systemd[1]: Unit caddy.service entered failed state.
Jan 14 09:45:36 important-pony-1.localdomain systemd[1]: caddy.service: main process exited, code=exited, status=1/FAILURE
Jan 14 09:45:36 important-pony-1.localdomain systemd[1]: Started Caddy.
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: {"level":"info","ts":1642124736.5124917,"logger":"tls.cache.maintenance","msg":"stopped background certificat
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: {"level":"info","ts":1642124736.512439,"logger":"tls.cache.maintenance","msg":"started background certificate
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: {"level":"info","ts":1642124736.5118148,"logger":"admin","msg":"admin endpoint started","address":"tcp/localh
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: {"level":"warn","ts":1642124736.5006769,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile"
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: {"level":"info","ts":1642124736.4945745,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyf
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: SHELL=/sbin/nologin
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: USER=caddy
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: LOGNAME=caddy
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: HOME=/var/lib/caddy
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: NOTIFY_SOCKET=/run/systemd/notify
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: LANG=en_US.UTF-8
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: os.Getwd=/
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: runtime.Version=go1.16.8
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: runtime.GOMAXPROCS=2
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: runtime.NumCPU=2
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: runtime.Compiler=gc
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: runtime.GOARCH=amd64
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: runtime.GOOS=linux
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: caddy.Version=v2.4.6 h1:HGkGICFGvyrodcqOOclHKfvJC0qTU7vny/7FhYp9hNw=
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: caddy.HomeDir=/var/lib/caddy
Jan 14 09:45:36 important-pony-1.localdomain systemd[1]: Starting Caddy...
Jan 13 13:23:56 important-pony-1.localdomain systemd[1]: caddy.service failed.
Jan 13 13:23:56 important-pony-1.localdomain systemd[1]: Unit caddy.service entered failed state.
Jan 13 13:23:56 important-pony-1.localdomain systemd[1]: caddy.service: main process exited, code=exited, status=1/FAILURE

Wait, what? The dates on those logs don’t make sense :thinking: It goes from Jan 14 to Jan 13 :exploding_head: time traveler?

1 Like

I’m suspecting the storage location is not writable/accessible. Can you run these commands?

  • ls -al /var/lib/caddy
  • ls -al /var/lib/caddy/.local/share/caddy

Information pertaining to Caddy’s systemd service is documented here.

1 Like

The log mentioned above include 2 attempts of launching caddy service, one in Jan 13 when I manually reboot the Centos, which is just part of the whole log. One in Jan 14 when I wake up receive the mail and test caddy again and get the same error log.

So for each failure, caddy output the same log goes like this:

Jan 14 09:45:36 important-pony-1.localdomain systemd[1]: caddy.service failed.
Jan 14 09:45:36 important-pony-1.localdomain systemd[1]: Unit caddy.service entered failed state.
Jan 14 09:45:36 important-pony-1.localdomain systemd[1]: caddy.service: main process exited, code=exited, status=1/FAILURE
Jan 14 09:45:36 important-pony-1.localdomain systemd[1]: Started Caddy.
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: {"level":"info","ts":1642124736.5124917,"logger":"tls.cache.maintenance","msg":"stopped background certificat
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: {"level":"info","ts":1642124736.512439,"logger":"tls.cache.maintenance","msg":"started background certificate
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: {"level":"info","ts":1642124736.5118148,"logger":"admin","msg":"admin endpoint started","address":"tcp/localh
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: {"level":"warn","ts":1642124736.5006769,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile"
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: {"level":"info","ts":1642124736.4945745,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyf
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: SHELL=/sbin/nologin
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: USER=caddy
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: LOGNAME=caddy
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: HOME=/var/lib/caddy
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: NOTIFY_SOCKET=/run/systemd/notify
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: LANG=en_US.UTF-8
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: os.Getwd=/
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: runtime.Version=go1.16.8
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: runtime.GOMAXPROCS=2
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: runtime.NumCPU=2
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: runtime.Compiler=gc
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: runtime.GOARCH=amd64
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: runtime.GOOS=linux
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: caddy.Version=v2.4.6 h1:HGkGICFGvyrodcqOOclHKfvJC0qTU7vny/7FhYp9hNw=
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Jan 14 09:45:36 important-pony-1.localdomain caddy[8865]: caddy.HomeDir=/var/lib/caddy
Jan 14 09:45:36 important-pony-1.localdomain systemd[1]: Starting Caddy...

They’re in reverse order I believe.

In that case, caddy should emit an error message before trying to provision a cert :thinking: