Permission Denied error when Caddy try to save the certificate

[job_noam (Noam)]

Hi,

1. Caddy version (caddy version):

v2.4.3

2. How I run Caddy:

1. We have 5 servers running behind an AWS network load balancer.
2. They share the same storage using AWS EFS.
3. Every server has the same Caddyfile but he doesn’t locate it in the shared storage (EFS). Just in the local storage of the server.

a. System environment:

Ubuntu 20.04.2 LTS (GNU/Linux 5.8.0-1041-aws x86_64)
Simple install without Docker

b. Command:

Auto Caddy services are running in the background

d. My complete Caddyfile or JSON config:

    {
        #debug
        storage file_system {
                root /mnt/efs/caddy_data
        }
        log HHHH {
            output file /var/caddy_log/requests.log {
                roll_size 200mb
                roll_keep 3
                roll_keep_for 24h
            }
            level WARN
        }

        # TLS Options
        email HHHH @ HHHH .com
        on_demand_tls {
                ask      https://www. HHHH .com/isDomainValid.php
        }
    }

    https://

    tls {
        on_demand
        issuer acme {
                email HHHH @ HHHH .com
        }
        issuer zerossl HHHH {
                email HHHH @ HHHH .com
        }
    }

    reverse_proxy 76.223. HHHH . HHHH {
        header_down -proxy-cache
    }

3. The problem I’m having:

I saw in my logs that when Caddy tried to save the certificate he get from Let’sEncrypt it get this error:
Feb 08 17:01:57 ip-172-30-3-253 caddy[481]: {"level":"error","ts":1644339717.556735,"logger":"tls.obtain","msg":"will retry","error":"[test20. myDomain .com] Obtain: saving assets: mkdir /mnt/efs/caddy/certificates/acme-v02.api.letsencrypt.org-directory/test20. myDomain .com: permission denied","attempt":1,"retrying_in":60,"elapsed":0.715147592,"max_duration":2592000}

ZeroSSL folder is working and doesn’t send any error.

1. Can I run something that I will make sure I’m right and Caddy has an issue with this folder?
2. What do I need to run to make sure Caddy user will have full permission on my folders?

Thank you.

[matt (Matt Holt)]

What is the output of ls -l /mnt/efs/caddy/certificates ?

[job_noam (Noam)]

drwxr-xr-x 6348 nobody nogroup 268288 Jan 1 1970 acme-v02.api.letsencrypt.org-directory
drwx------ 47618 caddy caddy 2365440 Feb 8 21:24 acme.zerossl.com-v2-dv90

[matt (Matt Holt)]

Well that’s the problem, somehow the file mode of the folder containing the certs for LE got reset: it has no owner, no group, and no timestamp. Simply reset the folder’s mode (and all of its contents) to have the same owner and permission bits as the rest of the contents.

(To clarify, this doesn’t actually have anything to do with Caddy. And was not caused by Caddy. A simple run of chown and chmod commands should fix it.)

[sarge (Mark Sargent)]

@job_noam I would be interested in how this got into this state. Did you manage to get a root cause?

[job_noam (Noam)]

Really don’t know, most of the time I’m not changing anything in the server because of my little knowledge
So, I don’t know what to say.
Anyway, that commend solve the issue for me:
sudo chown -R caddy:caddy /mnt/efs/caddy/certificates/acme-v02.api.letsencrypt.org-directory

hope it will help someone.