Caddy.storage.file_system Permission issues

1. The problem I’m having:

I have been testing around a bit and wanted to start from scratch, I have uninstalled caddy including all certificates that were previously created in /var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/duelify.com/duelify.com.crt etc.

I’m trying to run Caddy service with the caddy-dns/cloudflare plugin. But it seems to me that there are either permission problems or Caddy doesn’t create the certificates in the given storage file system path.

systemctl status caddy.service

2. Error messages and/or full log output:

Jun 08 08:23:34 new-duelify caddy[23034]: {"level":"info","ts":1686212614.428661,"msg":"using provided configuration","config_file":"/etc/caddy/caddy.json","config_adapter":""}
Jun 08 08:23:34 new-duelify caddy[23034]: {"level":"info","ts":1686212614.4300416,"msg":"redirected default logger","from":"stderr","to":"/var/log/caddy/error.log"}
Jun 08 08:23:34 new-duelify caddy[23034]: Error: loading initial config: loading new config: tls app module: start: automate: managing [*.duelify.com]: automate: manage [*.duelify.com]: *.duelify.com: caching certificate: open /root/tls/certificates/acme-v02.api.letsencrypt.org-directory/wildcard_.duelify.com/wildcard_.duelify.com.key: permission denied
Jun 08 08:23:34 new-duelify systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
Jun 08 08:23:34 new-duelify systemd[1]: caddy.service: Failed with result 'exit-code'.
Jun 08 08:23:34 new-duelify systemd[1]: Failed to start Caddy.

/var/log/caddy/error.log

{"level":"warn","ts":1686212614.430258,"logger":"admin","msg":"admin endpoint disabled"}
{"level":"warn","ts":1686212614.4312956,"logger":"http","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"h1","http_port":80}
{"level":"info","ts":1686212614.431734,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0001d6ee0"}
{"level":"info","ts":1686212614.431749,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0001d6ee0"}

3. Caddy version:

v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=

4. How I installed and ran Caddy:

mkdir ~/tls
chown caddy:caddy ~/tls/
apt install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
apt update
apt install caddy
systemctl stop caddy
sudo curl -o /usr/bin/caddy "https://caddyserver.com/api/download?os=linux&arch=amd64&p=github.com%2Fcaddy-dns%2Fcloudflare&p=github.com%2FWeidiDeng%2Fcaddy-cloudflare-ip&p=github.com%2Fmastercactapus%2Fcaddy2-proxyprotocol&p=github.com%2Fimgk%2Fcaddy-trojan"
sed -i "s/Caddyfile/caddy.json/" /lib/systemd/system/caddy.service
systemctl daemon-reload
systemctl start caddy

a. System environment:

Debian 11

c. Service/unit/compose file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/caddy.json
ExecReload=/usr/bin/caddy reload --config /etc/caddy/caddy.json --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateDevices=yes
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

CLOUDFLARE_API_TOKEN in Service has to be redacted to be able to post this here.

d. My complete Caddy config:

{
  "admin": {
    "disabled": true
  },
  "logging": {
    "logs": {
      "default": {
        "writer": {
          "output": "file",
          "filename": "/var/log/caddy/error.log"
        },
        "level": "ERROR"
      }
    }
  },
  "storage": {
    "module": "file_system",
    "root": "/root/tls"
  },
  "apps": {
    "http": {
      "servers": {
        "h1": {
          "listen": [
            ":80"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "static_response",
                  "headers": {
                    "Location": [
                      "https://{http.request.host}{http.request.uri}"
                    ]
                  },
                  "status_code": 301
                }
              ]
            }
          ]
        },
        "h1h2c": {
          "listen": [
            "127.0.0.1:88"
          ],
          "listener_wrappers": [
            {
              "wrapper": "proxy_protocol"
            }
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "headers",
                  "response": {
                    "set": {
                      "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains; preload"
                      ]
                    }
                  }
                },
                {
                  "handler": "file_server",
                  "root": "/srv/http/default"
                }
              ]
            }
          ],
          "protocols": [
            "h1",
            "h2c"
          ]
        }
      }
    },
    "tls": {
      "certificates": {
        "automate": [
          "*.duelify.com"
        ]
      },
      "automation": {
        "policies": [
          {
            "issuers": [
              {
                "module": "acme",
                "email": "myemail.com",
                "challenges": {
                  "dns": {
                    "provider": {
                      "name": "cloudflare",
                      "api_token": "{env.CLOUDFLARE_API_TOKEN}"
                    }
                  }
                }
              },
              {
                "module": "zerossl",
                "email": "myemail.com",
                "challenges": {
                  "dns": {
                    "provider": {
                      "name": "cloudflare",
                      "api_token": "{env.CLOUDFLARE_API_TOKEN}"
                    }
                  }
                }
              }
            ]
          }
        ]
      }
    }
  }
}

Thank you

Caddy runs as the caddy user, not as root. It doesn’t have permission to write to /root.

Why are you overriding the storage path? Leave it as the default and it will work fine.

Thank you for your reply.

I’m following this particular example to utilise Caddy and Xray:

The problem is that in this example both caddy and Xray accessing the same path to use the certificates.
I’m not sure if this is the best approach.

The recommended way to deal with that is to use events to copy certificates to somewhere else where your other software can read them. See the events app

Thanks for that. Sorry is there any example you could share for events.
Looking at the docs it seems it’s a fairly new and experimental feature. I couldn’t find any example for it on the internet.

I can’t find a list of supported events.

This is what ChatGPT has been suggesting. Is this a fair approach?

{
  "apps": {
    "http": { 
      // ... Your existing HTTP server configuration ...
    },
    "events": {
      "hooks": {
        "tls_handshake_completed": [
          {
            "handler": "exec",
            "command": "/path/to/your/script.sh"
          }
        ]
      }
    }
  }
}
#!/bin/bash
cp /var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/wildcard_.duelify.com/* /path/to/destination

I don’t think that’s quite right.

Sorry, yeah, events are not well-documented yet but they will be on our new website.

Ad-hoc docs are in the release notes for 2.6: Release v2.6.0 · caddyserver/caddy · GitHub

Here’s a list:

Hi Matt,

Thanks for the example.
That’s brilliant for restarting the Xray service whenever new certificates have been obtained by Caddy.

However the issue remains that how do we give access to these certificates to the XRay service in the first place?

Is there a way to add a handler beforehand to copy the certificates in place first?

"handlers": [
    {
        "handler": "exec",
        "command": "cp",
        "args": ["/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/wildcard_.duelify.com/*", "/home/xray/tls/"]
    },
{
	"handler": "exec",
	"command": "systemctl",
       "args": ["reload", "xray"]
}
]

But given the path, it seems rather messy.
Maybe the best approach is still to stick to old fashioned ACME script and crontab.

If you can configure xray to read the cert files where they are already at, you could avoid a copy. (You’ll need to make sure it has permission to read those files.)

Otherwise I think the copy + reload is a fine solution. (Again, check file permissions.) What’s wrong with it?

Francis left me a note (he’s busy with $DAYJOB today) so I’ll expand on it here.

The idea is that you’d use the event payload which includes the actual path of the certificate. You’re hard-coding the path, which will not always work. Caddy may use ZeroSSL instead of Let’s Encrypt, for example, which has a different path.

I actually need to see if we have placeholders that contain the path to the certificate file(s). That’d be ideal.

Ok so I think the data we had in the event wasn’t super useful for this, until now:

So once this commit gets released, you’ll be able to use these placeholders:

{event.data.storage_path}
{event.data.certificate_path}
{event.data.private_key_path}
{event.data.metadata_path}

Like so:

{
    "handler": "exec",
    "command": "cp",
    "args": ["{event.data.storage_path}/*", "/home/xray/tls/"]
}

Hi Matt,

That’s amazing. Thank you. What a quick turn around. Yes, I share the concern with you and Francis.

I still have to test this to see if this works. If it does then driving the certificate updates through caddy’s caddy-dns/cloudflare plugin would make sense. Otherwise the classic script is more powerful. Please allow me to explain:

You see on my other test domain gymmotivator.com I have used the classic acme script.

curl https://get.acme.sh | sh
export CF_Token="sdfsdfsdfljlbjkljlkjsdfoiwje" 
acme.sh --issue -d gymmotivator.com -d *. gymmotivator.com --dns dns_cf
mkdir /etc/ssl/vpn 
acme.sh --install-cert -d gymmotivator.com -d *.gymmotivator.com --fullchain-file /etc/ssl/vpn/cert.pem --key-file /etc/ssl/vpn/privkey.key --reloadcmd "chown root:caddy -R /etc/ssl/vpn && systemctl restart xray && systemctl restart caddy"

The acme script runs as root and can store the certificates under the shared directory /etc/ssl/vpn.
I can then accordingly change the permissions to allow both services to access the certificates easily.

When caddy caddy-dns/cloudflare plugin takes care of certificates, it runs them under caddy’s user permission. Caddy can only create the certificates under its own home /var/lib/caddy/, which gives no execution and read permissions to outsiders. In other words the certificates are trapped and can only be used by caddy itself.

I have tried to play with links:
sudo ln -f -s /var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/duelify.com/duelify.com.key /etc/ssl/vpn/duelify.com.key

But this is pointless, since xray user still can’t access the target of the link.

After 3 days of trying I was about to give up, when your commit arrived at the perfect timing. :slight_smile:

I can see the commit is on cert magic repo. Will there anytime soon a mini release of caddy incorporating this? Not sure if I can take a fork to get a peak preview beforehand.

Thank you,

Tangential, but important: I happened to discover a 0-day RCE in acme.sh just last night, and it was patched this morning, so you might want to upgrade if you’re running that, especially as root :sweat_smile:

(It’s a zero-day because a group was already actively exploiting it.)

You can copy the files outside the folder though, and/or change the permissions. In theory if you just change the permissions on the certificate and key you want to share with another program, it should work just fine. I can’t imagine why the other program wouldn’t be able to read it then.

If you can build from source, you can test it real easily. Just clone down the caddy repo and run go get github.com/caddyserver/certmagic@master and it should give you the latest commit. Then you can run go build and run the resulting binary!

Otherwise just wait a little bit for me to tag the second 2.7 beta, which I hope will happen early next week (Monday-ish) or so. Would love for you to test it out!

Tangential, but important: I happened to discover a 0-day RCE in acme.sh just last night, and it was patched this morning, so you might want to upgrade if you’re running that, especially as root :sweat_smile:

Wow I read about your findings on that thread. Amazing job. Not only you have discovered an important flaw, you also forced a dodgy organisation into hiding. Well done.

Given how you experienced you are with internet security, I have to ask you for your advice this regarding please:

You can copy the files outside the folder though, and/or change the permissions. In theory if you just change the permissions on the certificate and key you want to share with another program, it should work just fine. I can’t imagine why the other program wouldn’t be able to read it then.

I thought at first it would be as simple too, but then I got into trouble:

First I gave group reading rights to the certificate. The certificate belongs to caddy:caddy.

$ chmod -R g+r /var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/duelify.com/duelify.com.key

Proof that xray user also belongs to caddy group:

$ sudo su xray
$ groups
xray caddy

However permission is still denied to xray:

$ ls -lt /var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/duelify.com/duelify.com.key
ls: cannot access '/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/duelify.com/duelify.com.key': Permission denied

In Linux to access a file, the user needs to have execute permissions on all the parent directories of the file. This is because accessing a file is seen as traversing to the file location, which requires execute permissions on the directories.

$ exit
$ chmod g+x /var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/duelify.com
$ chmod g+x /var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory
$ chmod g+x /var/lib/caddy/.local/share/caddy/certificates
$ chmod g+x /var/lib/caddy/.local/share/caddy
$ chmod g+x /var/lib/caddy/.local/share
$ chmod g+x /var/lib/caddy/.local

Now it works:

$ su xray
$ ls -lt /var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/duelify.com/duelify.com.key
-rw-r----- 1 caddy caddy 227 Jun 9 13:08 /var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/duelify.com/duelify.com.key

I’m a bit cautious when it comes to security and I don’t know if this is a safe approach. Am I overthinking it?

However I still think your commit would still be the best solution, because the path under /var/lib/caddy/.local/share/caddy/certificates/ could end up to be different if it’s executed by ZeroSSL instead of letsencrypt.

If you can build from source, you can test it real easily. Just clone down the caddy repo and run go get github.com/caddyserver/certmagic@master and it should give you the latest commit. Then you can run go build and run the resulting binary!

Otherwise just wait a little bit for me to tag the second 2.7 beta, which I hope will happen early next week (Monday-ish) or so. Would love for you to test it out!

Of course. I would love to test it.

I never worked with go before. Is this approach correct?

git clone https://github.com/caddyserver/caddy
cd caddy
go get github.com/caddyserver/certmagic@master
cd cmd/caddy
go build

I can see now a caddy binary. However I still need to integrate the caddy-dns/cloudflare and mastercactapus/caddy2-proxyprotocol plugins with this self built binary, otherwise my setup won’t work.

Any advice, how I could do that? Otherwise I can also wait for your 2.7 Beta to come out next week. It’s no problem.

Thanks,
Houman

1 Like

Thank you for doing your homework :pray: Looks like good progress.

I wonder if creating a hard link would be easier in your case, so you don’t have to give +x on all parent directories (that seems silly that you’d need to do that, since you already know the path, you just need to read the file, the dir perms shouldn’t matter IMO?):

Ah, if you need plugins, you can use xcaddy, something like this:

$ xcaddy build --with github.com/caddy-dns/cloudflare --with github.com/mastercactapus/caddy2-proxyprotocol --with github.com/caddyserver/certmagic@master

I think that will give you what we want. You can use ./caddy build-info to verify…

Hi Matt,

Sorry for the delay. It was a long day at work.

I was just able to look into it. I ran latest xcaddy arm build on Mac.

xcaddy build --with github.com/caddy-dns/cloudflare --with github.com/mastercactapus/caddy2-proxyprotocol --with github.com/caddyserver/certmagic@master

2023/06/12 18:53:56 [INFO] exec (timeout=-2562047h47m16.854775808s): /opt/homebrew/bin/go build -o /Users/houmie/Projects/caddy/caddy -ldflags -w -s -trimpath
# github.com/caddyserver/caddy/v2/modules/caddytls
../../../go/pkg/mod/github.com/caddyserver/caddy/v2@v2.6.4/modules/caddytls/automation.go:266:3: unknown field 'Managers' in struct literal of type certmagic.Config
2023/06/12 18:54:13 [INFO] Cleaning up temporary folder: /Users/houmie/Projects/caddy/buildenv_2023-06-12-1853.3707030395
2023/06/12 18:54:13 [FATAL] exit status 2

It seems there is a problem with certmagic.Config on the master branch as well as tag v1.18.0.

Thank you for that link from unix.stackexchange.com. I need to let that sink in. It seems unusual. :smiley:

1 Like

Oh, sorry – we need to specify the latest beta version of Caddy since it defaults to the latest stable, which is too old, since the latest CertMagic has a breaking change (it is not yet v1.0).

$ xcaddy build master --with ...

Give that a try!

No problem. This is what I have tried:

$ systemctl stop caddy
$ xcaddy build master --with github.com/caddy-dns/cloudflare --with github.com/mastercactapus/caddy2-proxyprotocol --with github.com/caddyserver/certmagic@master
...
2023/06/13 09:14:42 [INFO] exec (timeout=-2562047h47m16.854775808s): /usr/local/go/bin/go build -o /root/caddy -ldflags -w -s -trimpath
2023/06/13 09:17:24 [INFO] Build complete: ./caddy
2023/06/13 09:17:24 [INFO] Cleaning up temporary folder: /tmp/buildenv_2023-06-13-0909.682840620

$ cp caddy /usr/bin/
$ systemctl start caddy

Sadly it fails with this error:

Jun 13 09:20:03 new-duelify systemd[1]: Starting Caddy...
Jun 13 09:20:03 new-duelify caddy[63781]: panic: module already registered: caddy.listeners.proxy_protocol
Jun 13 09:20:03 new-duelify caddy[63781]: goroutine 1 [running]:
Jun 13 09:20:03 new-duelify caddy[63781]: github.com/caddyserver/caddy/v2.RegisterModule({0x1ed4260?, 0xc000508fc0?})
Jun 13 09:20:03 new-duelify caddy[63781]:         github.com/caddyserver/caddy/v2@v2.7.0-beta.1.0.20230612153522-2ddb7171440c/modules.go:156 +0x1d9
Jun 13 09:20:03 new-duelify caddy[63781]: github.com/mastercactapus/caddy2-proxyprotocol.init.0()
Jun 13 09:20:03 new-duelify caddy[63781]:         github.com/mastercactapus/caddy2-proxyprotocol@v0.0.2/caddy.go:9 +0x4c

Same error shows when running the binary directly from prompt.
./caddy

The error isn’t very descriptive what it could be.

1 Like

Remove github.com/mastercactapus/caddy2-proxyprotocol, you don’t need it anymore. Caddy ships with proxy_protocol by default now, since v2.7.0-beta.1

We haven’t pushed the docs update yet, but you can see it mentioned here Docs for v2.7.0 by francislavoie · Pull Request #322 · caddyserver/website · GitHub

1 Like

Ah thanks. That makes perfectly sense to include that plugin going forwards.

I succeeded building and running caddy again.

$ /usr/bin/caddy version
v2.7.0-beta.1.0.20230612153522-2ddb7171440c h1:WOJoV9b1sf+KudXzTxyDwdzRvOXUontOk3JlvSrWtAE=

As root I created a new directory for xray User:

mkdir /home/xray/tls2
chown xray:caddy /home/xray/tls2
chmod -R g+rw /home/xray/tls2

Now the caddy user should be able to read and write to that directory.

Then I went to:

cd /var/lib/caddy/.local/share/caddy/certificates/
rm -r acme-v02.api.letsencrypt.org-directory/

Now restarting caddy should fetch the certs again and trigger the events.
But it throws an error:

Jun 13 21:42:12 new-duelify caddy[70418]: Error: loading initial config: loading new config: loading events app module: provision events: loading event subscriber modules: position 0: loading module 'exec': unknown module: events.handlers.exec
Jun 13 21:42:12 new-duelify systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
Jun 13 21:42:12 new-duelify systemd[1]: caddy.service: Failed with result 'exit-code'.
Jun 13 21:42:12 new-duelify systemd[1]: Failed to start Caddy.

caddy.json:

"apps": {
    "events": {
      "subscriptions": [
        {
          "events": [
            "cert_obtained"
          ],
          "handlers": [
            {
              "handler": "exec",
              "command": "cp",
              "args": ["{event.data.storage_path}/*", "/home/xray/tls2/"]
            }
          ]
        }
      ]
    },
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            "127.0.0.1:5005"
          ],
     ...

I have a feeling xcaddy didn’t pick the commit, as it doesn’t know about events.

This is how I compiled it:

xcaddy build master --with github.com/caddy-dns/cloudflare --with github.com/caddyserver/certmagic@master

Thanks,
Houman

1 Like