Lots of tls.issuance.acme Errors

Lots of errors pouring into the log file that seem to be harmless. The site is working fine.

Not sure what I am doing wrong to get all these errors? The logs get quite large after a day or so…

1. Caddy version (caddy version):

v2.4.6 h1:HGkGICFGvyrodcqOOclHKfvJC0qTU7vny/7FhYp9hNw=

2. How I run Caddy:

a. System environment:

Ubuntu 20.04.3 LTS (aka focal)
systemd 245 (245.4-4ubuntu3.14)
Running on a DigitalOcean droplet
INBOUND:
ICMP
HTTP (80)
HTTPS (443)

OUTBOUND:
ICMP
TCP (all ports)
UDP (all ports)

b. Command:

sudo cp myCaddyfile /etc/caddy/Caddyfile
sudo systemctl reload caddy.service

c. Service/unit/compose file:

caddy.service
# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

Caddyfile
# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace the line below with your
# domain name.
# :80 
# Set this path to your site's directory.
# root * /usr/share/caddy

# Enable the static file server.
# file_server

# Another common task is to set up a reverse proxy:
# reverse_proxy localhost:3000

# Or serve a PHP site through php-fpm:
# php_fastcgi localhost:9000

# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile

http://meris.xyz {
  redir https://www.meris.xyz{uri} permanent
}


(meris_rproxy) {
  handle {
    encode gzip
    reverse_proxy {
      to 0.0.0.0:{srv_port}
      header_up ext-host {meris_host}
      header_up host {host}
      header_up origin {origin}
    }
  }
}

# handle direct DNS queries
www.meris.xyz {
  # send all ddp requests to shared server @admin.test.meris.xyz
  map {host} {meris_port} {srv_port} {meris_host} {
#    "stest.meris.xyz" 5880 4880 "stest.meris.xyz"
#    "stage.meris.xyz" 5280 4280 "stage.meris.xyz"
    "www.meris.xyz" 5080 4080 "www.meris.xyz"
  }

  import meris_rproxy

  # all queries logged to stderr by default
}


# =============
# ADMIN Routing
# =============

admin.meris.xyz {
  map {host} {srv_port} {meris_host} {
#    "admin.stest.meris.xyz" 5880 "admin.stest.meris.xyz"
#    "admin.stage.meris.xyz" 5280 "admin.stage.meris.xyz"
    "admin.meris.xyz" 5080 "admin.meris.xyz"
  }

  import meris_rproxy
}

# MINIO

https://b.meris.xyz, https://blob.meris.xyz {
      reverse_proxy {
      to 0.0.0.0:9000
      header_up X-Forwarded-Proto {scheme}
      header_up X-Forwarded-Host {host}
      header_up Host {host}
      health_uri /minio/health/ready
    }
}

3. The problem I’m having:

tls.issuance.acme is pouring out errors.
Many of the errors are for domains not listed in my Caddyfile except within comments and one domain is not in the file at all.
Here is a sampling of them:

Error samples
# this looks right
Jan  4 21:26:13 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"info","ts":1641349573.679473,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["b.meris.xyz","blob.meris.xyz","www.meris.xyz","admin.meris.xyz"]}
# only used as http://
Jan  4 21:26:32 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349592.0677156,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"meris.xyz","error":"no information found to solve challenge for identifier: meris.xyz"}
# OK these should be challenged
Jan  4 21:26:34 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349594.1415145,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"www.meris.xyz","error":"no information found to solve challenge for identifier: www.meris.xyz"}
Jan  4 21:26:39 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349599.2948058,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"admin.meris.xyz","error":"no information found to solve challenge for identifier: admin.meris.xyz"}
Jan  4 21:26:34 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349594.106386,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"b.meris.xyz","error":"no information found to solve challenge for identifier: b.meris.xyz"}
# whoa!  These domains are mentioned behind comments
# whey are they being challenged?
Jan  4 21:26:32 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349592.856338,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"admin.stest.meris.xyz","error":"no information found to solve challenge for identifier: admin.stest.meris.xyz"}
Jan  4 21:26:39 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349599.3018298,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"admin.stage.meris.xyz","error":"no information found to solve challenge for identifier: admin.stage.meris.xyz"}
Jan  4 21:26:42 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349602.25449,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"stest.meris.xyz","error":"no information found to solve challenge for identifier: stest.meris.xyz"}
Jan  4 21:26:47 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349607.1742175,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"stage.meris.xyz","error":"no information found to solve challenge for identifier: stage.meris.xyz"}
# whoa! whoa!  this domain is not even in my Caddyfile
# I had it in previous Caddyfiles but have since
# reloaded and then I stopped the service and restarted
# then I rebooted.  This old name is still haning around
Jan  4 21:26:34 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349594.112324,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"bconsole.meris.xyz","error":"no information found to solve challenge for identifier: bconsole.meris.xyz"}

4. Error messages and/or full log output:

Note: tried to use debug directive but it failed

Jan  4 21:22:04 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1771]: reload: adapting config using caddyfile: /etc/caddy/Caddyfile:45: unrecognized directive: debug
5 minutes of logs
Jan  4 21:26:13 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1978]: {"level":"info","ts":1641349573.6682084,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Jan  4 21:26:13 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1978]: 2022/01/04 21:26:13 [WARNING] Unnecessary header_up ('X-Forwarded-Proto' field): the reverse proxy's default behavior is to pass headers to the upstream
Jan  4 21:26:13 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1978]: {"level":"warn","ts":1641349573.672293,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":27}
Jan  4 21:26:13 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"info","ts":1641349573.6735933,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_addr":"127.0.0.1:43844","headers":{"Accept-Encoding":["gzip"],"Content-Length":["1913"],"Content-Type":["application/json"],"Origin":["localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
Jan  4 21:26:13 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"info","ts":1641349573.676234,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Jan  4 21:26:13 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"info","ts":1641349573.6768236,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Jan  4 21:26:13 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"info","ts":1641349573.6771376,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Jan  4 21:26:13 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"info","ts":1641349573.6773958,"logger":"http","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
Jan  4 21:26:13 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"info","ts":1641349573.679473,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["b.meris.xyz","blob.meris.xyz","www.meris.xyz","admin.meris.xyz"]}
Jan  4 21:26:13 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"info","ts":1641349573.6845386,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0000f4a10"}
Jan  4 21:26:13 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"info","ts":1641349573.6854987,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0000f4540"}
Jan  4 21:26:13 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"info","ts":1641349573.6865418,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Jan  4 21:26:13 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"info","ts":1641349573.6879818,"logger":"admin.api","msg":"load complete"}
Jan  4 21:26:13 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"info","ts":1641349573.6998959,"logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
Jan  4 21:26:32 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349592.0677156,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"meris.xyz","error":"no information found to solve challenge for identifier: meris.xyz"}
Jan  4 21:26:32 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349592.0679114,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"meris.xyz","error":"no information found to solve challenge for identifier: meris.xyz"}
Jan  4 21:26:32 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349592.856338,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"admin.stest.meris.xyz","error":"no information found to solve challenge for identifier: admin.stest.meris.xyz"}
Jan  4 21:26:32 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349592.856437,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"admin.stest.meris.xyz","error":"no information found to solve challenge for identifier: admin.stest.meris.xyz"}
Jan  4 21:26:34 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349594.1062756,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"b.meris.xyz","error":"no information found to solve challenge for identifier: b.meris.xyz"}
Jan  4 21:26:34 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349594.106386,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"b.meris.xyz","error":"no information found to solve challenge for identifier: b.meris.xyz"}
Jan  4 21:26:34 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349594.112324,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"bconsole.meris.xyz","error":"no information found to solve challenge for identifier: bconsole.meris.xyz"}
Jan  4 21:26:34 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349594.1129026,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"bconsole.meris.xyz","error":"no information found to solve challenge for identifier: bconsole.meris.xyz"}
Jan  4 21:26:34 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349594.1345665,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"b.meris.xyz","error":"no information found to solve challenge for identifier: b.meris.xyz"}
Jan  4 21:26:34 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349594.1351318,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"b.meris.xyz","error":"no information found to solve challenge for identifier: b.meris.xyz"}
Jan  4 21:26:34 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349594.1415145,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"www.meris.xyz","error":"no information found to solve challenge for identifier: www.meris.xyz"}
Jan  4 21:26:34 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349594.142327,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"www.meris.xyz","error":"no information found to solve challenge for identifier: www.meris.xyz"}
Jan  4 21:26:34 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349594.1703396,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"www.meris.xyz","error":"no information found to solve challenge for identifier: www.meris.xyz"}
Jan  4 21:26:34 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349594.1708992,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"www.meris.xyz","error":"no information found to solve challenge for identifier: www.meris.xyz"}
Jan  4 21:26:39 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349599.2640316,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"admin.meris.xyz","error":"no information found to solve challenge for identifier: admin.meris.xyz"}
Jan  4 21:26:39 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349599.2641172,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"admin.meris.xyz","error":"no information found to solve challenge for identifier: admin.meris.xyz"}
Jan  4 21:26:39 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349599.2942057,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"admin.meris.xyz","error":"no information found to solve challenge for identifier: admin.meris.xyz"}
Jan  4 21:26:39 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349599.2948058,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"admin.meris.xyz","error":"no information found to solve challenge for identifier: admin.meris.xyz"}
Jan  4 21:26:39 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349599.3018298,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"admin.stage.meris.xyz","error":"no information found to solve challenge for identifier: admin.stage.meris.xyz"}
Jan  4 21:26:39 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349599.3019164,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"admin.stage.meris.xyz","error":"no information found to solve challenge for identifier: admin.stage.meris.xyz"}
Jan  4 21:26:40 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349600.1123993,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"meris.xyz","error":"no information found to solve challenge for identifier: meris.xyz"}
Jan  4 21:26:40 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349600.1124938,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"meris.xyz","error":"no information found to solve challenge for identifier: meris.xyz"}
Jan  4 21:26:42 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349602.25449,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"stest.meris.xyz","error":"no information found to solve challenge for identifier: stest.meris.xyz"}
Jan  4 21:26:42 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349602.254576,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"stest.meris.xyz","error":"no information found to solve challenge for identifier: stest.meris.xyz"}
Jan  4 21:26:47 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349607.1740549,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"stage.meris.xyz","error":"no information found to solve challenge for identifier: stage.meris.xyz"}
Jan  4 21:26:47 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349607.1742175,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"stage.meris.xyz","error":"no information found to solve challenge for identifier: stage.meris.xyz"}
Jan  4 21:27:35 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349655.8039985,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"meris.xyz","error":"no information found to solve challenge for identifier: meris.xyz"}
Jan  4 21:27:35 ubuntu-s-1vcpu-1gb-amd-nyc1-01 caddy[1369]: {"level":"error","ts":1641349655.8041055,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"meris.xyz","error":"no information found to solve challenge for identifier: meris.xyz"}

5. What I already tried:

  1. Trimmed down the Caddyfile to its basics. I have been trying to reuse a single Caddyfile so I got rid of all of the unused stuff for this server.
  2. In addition to executing the systemctl reload caddy.service I have tried
systemctl stop caddy.service
systemctl start caddy.service

Finally tried:

sudo reboot

I still keep getting the errors

6. Links to relevant resources:

I’ve seen something like this myself when a challenge fails. I think the acme ca is still trying to validate the challenge? Will look into this more soon if I can.

debug is not a directive, it’s a global option. It goes in a block at the top of your Caddyfile. See the docs to understand the structure:

FYI, you can remove these, they’re redundant.

That’s definitely pretty strange that you’re still getting challenges for these.

How long ago has it been since you’ve had them enabled in your config?

It would be interesting to find out which IP addresses those requests are coming from. I don’t think Let’s Encrypt tends to keep trying for too long, so it might be either a bug on their end or someone else’s server trying to make weird requests to your server (maybe somekind of weird caching layer trying to refresh some old request results? I dunno)

1 Like

@francislavoie, thanks for the feedback.

  1. I was able to set the debug option within the options block.
  2. Removed my redundant header_up directives.

Regarding the errors, here is a timeline:

  1. 01/04/21 ~20:00 - ghost domains were removed from Caddyfile
  2. 01/04/21 21:22 - Server was last restarted
  3. 01/05/21 06:02 - Last error for any ghost domain
  4. 01/05/21 11:41 - Server restarted with debug option, no tls.issuance.acme errors (initially)
  5. 01/05/21 11:43 - First http.stdlib error
  6. 01/05/21 12:00 - First tls & tls.issuance.acme errors
  7. Then the tls.issuance.acme errors start flooding in again

You can see the full debug log file here:

1 Like

Was caddy restarted forcefully? If so I can see how maybe it didn’t deactivate the challenges in time.

This is my restart command:

sudo systemctl reload caddy.service

plus between 01/04/21 20:00 & 01/04/21 21:22 I executed a few of these:

sudo systemctl stop caddy.service
sudo systemctl start caddy.service

plus I rebooted twice.

My understanding is that systemctl will perform graceful shutdowns?

It should, yeah…

I wonder, did Caddy ever attempt to use ZeroSSL to get a cert after it may have failed from Let’s Encrypt? I still wonder where the requests are coming from. (Enabling access logging would help.)

@matt , heading out on a vacation back the end of the month. If it is still firing errors I will enable access logging and post the logs.

@matt, @francislavoie, I got back from vacation.

I just pulled in the snapshot of the latest errors with access logs turned on. It looks like acme.zerossl.com is making a challenge request?

And it looks like it has been cropping up at least once/twice per hour for the last 3 weeks.

I do not think I ever reference zerossl anywhere?

@matt, @francislavoie, any thoughts?

Meant to reply to @matt, @francislavoie. Any thoughts?

I got rid of the errors by clearing out the certificates folder at caddy home e.g. rm -rf /var/lib/caddy/.local/share/caddy/* and then restarting. No errors since.

I guess some zerossl remnants were laying around in certificates…

BTW: the only way I found that was thank to this comment

Would be nice to add a note to Conventions — Caddy Documentation something along the lines of “remember $HOME is /var/lib/caddy for the caddy user. when run as a service.” That took a while to figure out…

1 Like

Actually there’s a note about that here:

But fair point we could mention that on the Conventions page too.

1 Like

Well, this depends on how the service was configured. We do recommend knowing about the environment you run Caddy in. That’s why we recommend running caddy with --environ when run as a service (as our official service unit file does).

Glad you figured it out. And yeah, if Caddy has certificates from a particular CA already, it will retry from that CA first in the future.

@matt, @francislavoie,

I do use the the official service unit file with the --environ option. I am not sure how this helps solve this problem?

When I type caddy environ | sort I get this:

output
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus                                                                                                                                                                                        HOME=/home/meris                                                                                                                                                                                                                             LANG=C.UTF-8                                                                                                                                                                                                                                 LESSCLOSE=/usr/bin/lesspipe %s %s                                                                                                                                                                                                            LESSOPEN=| /usr/bin/lesspipe %s                                                                                                                                                                                                              LOGNAME=meris                                                                                                                                                                                                                                LS_COLORS=rs=0:di=01;33:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:                                                                                                                                                          MOTD_SHOWN=pam                                                                                                                                                                                                                               OLDPWD=/home/meris                                                                                                                                                                                                                           PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin                                                                                                                                      PWD=/home/meris/live/logs                                                                                                                                                                                                                    SHELL=/usr/bin/bash                                                                                                                                                                                                                          SHLVL=0                                                                                                                                                                                                                                      SSH_CLIENT=50.35.89.2 61534 22                                                                                                                                                                                                               SSH_CONNECTION=50.35.89.2 61534 143.244.170.208 22                                                                                                                                                                                           SSH_TTY=/dev/pts/0                                                                                                                                                                                                                           TERM=screen-256color                                                                                                                                                                                                                         USER=meris                                                                                                                                                                                                                                   XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop                                                                                                                                                                             XDG_RUNTIME_DIR=/run/user/1000                                                                                                                                                                                                               XDG_SESSION_CLASS=user                                                                                                                                                                                                                       XDG_SESSION_ID=67                                                                                                                                                                                                                            XDG_SESSION_TYPE=tty                                                                                                                                                                                                                         _=/usr/bin/caddy                                                                                                                                                                                                                             caddy.AppConfigDir=/home/meris/.config/caddy                                                                                                                                                                                                 caddy.AppDataDir=/home/meris/.local/share/caddy                                                                                                                                                                                              caddy.ConfigAutosavePath=/home/meris/.config/caddy/autosave.json                                                                                                                                                                             caddy.HomeDir=/home/meris                                                                                                                                                                                                                    caddy.Version=v2.4.6 h1:HGkGICFGvyrodcqOOclHKfvJC0qTU7vny/7FhYp9hNw=                                                                                                                                                                         os.Getwd=/home/meris/live/logs                                                                                                                                                                                                               runtime.Compiler=gc                                                                                                                                                                                                                          runtime.GOARCH=amd64                                                                                                                                                                                                                         runtime.GOMAXPROCS=1                                                                                                                                                                                                                         runtime.GOOS=linux                                                                                                                                                                                                                           runtime.NumCPU=1                                                                                                                                                                                                                             runtime.Version=go1.17.2

HOME=/home/meris & there is no XDG_DATA_HOME value?

Do I get the caddy environment for the service a different way?

When you run caddy environ as your current user, you see the environment if Caddy would run as your current user. That’s not really useful. What you want to know is the environment of the user you’ll actually be running Caddy as, i.e. the caddy user since you’re running it as a systemd service. This is why the --environ option exists for the caddy run command.

1 Like

@francislavoie, thanks for the info. But I am a little stuck and I think I have a more generic linux question perhaps you can answer?

I cannot su caddy, caddy is just the user for the service but it really does not have an account. How do I get the environment for the caddy user?

I tried adding a password sudo passwd caddy, but when I su caddy it replies with This account is currently not available.

Thanks!

Also, another question/observation…

What is the point of caddy environ? essentially it is the same as running printenv?

When I run caddy run --environ --config Caddyfile I think it would be useful if the command caddy environ would output the environment of the current running caddy? Regardless of who started it?

Then if no caddy was running caddy environ could respond with no running caddyserver.

Yeah, the caddy user from our distro package isn’t a shell user (can’t log in as it). It can only run commands. Use sudo -u instead.

(The command is caddy environ btw.)

The “environment of the current running caddy” == “the environment of who started it” by definition. If that’s a problem you can take it up with the Linux project. :man_shrugging: