Getting bind: permission denied even with setcap in effect

Help
1

I updated my Arch Linux server and Caddy refused to start. After this, I updated Caddy to 0.9.4, but no help.
I get this:
failed to get certificate: [mysite.org] error presenting token: Could not start HTTPS server for challenge → listen tcp :443: bind: permission denied

getcap result verifies everything is in order:
/usr/bin/caddy = cap_net_bind_service+eip

It runs fine, if I launch the ExecStart command manually as root:
sudo /usr/bin/caddy -log stdout -agree=true -conf=/etc/caddy/caddy.conf -root=/var/tmp

Any ideas?

2

I found some clues that this could be because filesystems mounted with nosuid.
Here is the output of mount:

/dev/sda2 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
devtmpfs on /dev type devtmpfs (rw,relatime,size=2009724k,nr_inodes=502431,mode=755)
sysfs on /sys type sysfs (rw,relatime)
proc on /proc type proc (rw,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw)
devpts on /dev/pts type devpts (rw,relatime,mode=600,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=32,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)
mqueue on /dev/mqueue type mqueue (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev)
/dev/sda3 on /home type ext4 (rw,relatime,data=ordered)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=402048k,mode=700,uid=1000,gid=1000)

To test this theory, I could set the flag SECBIT_NO_SETUID_FIXUP. This can be set with the command capsh, but unfortunately the man docs are lacking for precisely this thing:
–secbits=N XXX - need to document this feature.

Apparently I have to give it the value in hexadecimal format. So how do I find out the hex representation for SECBIT_NO_SETUID_FIXUP?

This http://lxr.free-electrons.com/source/include/uapi/linux/securebits.h

just says

#define SECBIT_NO_SETUID_FIXUP (issecure_mask(SECURE_NO_SETUID_FIXUP))

So how to derive the bit value from this:
#define issecure_mask(X) (1 << (X))

Edit: I tried capsh --print and this is the output:

Current: =
Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend
Securebits: 00/0x0/1’b0
secure-noroot: no (unlocked)
secure-no-suid-fixup: no (unlocked)
secure-keep-caps: no (unlocked)
uid=1000(myuser)
gid=1000(myuser)
groups=10(wheel),1000(myuser)

3

Ok I got advice about the secbits. Still no clear idea how I would be able to use them. Apparently I need to pass 0x04 to activate SECBIT_NO_SETUID_FIXUP.

Capsh is a debugging tool and I am having a hard time running Caddy with it.
Tried sudo capsh --keep=1 --secbits=0x04 --user=http – -c “/usr/bin/caddy”
but it just said
2016/12/29 23:44:26 [ERROR] Unable to make new certificate storage path: mkdir /root/.caddy: permission denied

4

The $HOME environment variable visible to Caddy has a value of /root and Caddy needs permission to write in that folder.

5

Yeah, I’m too tired to think :frowning:
I tried to find information on how I could remove the nosuid from /sys/fs/cgroup/systemd to test the nosuid theory, but I failed. It seems to be the only mount that could be the culprit.

Edit: discussed this with some gurus and came to the conclusion that this nosuid thing is a red herring. It would apply, if the filesystem /usr/bin/caddy resides in would have the nosuid flag. This was tested by copying capsh executable to /tmp, which had nosuid flag, giving capsh itself the bind capabilities: it printed Current: =. Then doing mount /tmp -oremount,suid and setting the cap again, it now printed Current: = cap_net_bind_service+ep. It worked fine from the start with a copy of capsh in /usr/bin.

6

Ok, another test: I was able to run Caddy successfully as user http without systemd with this command:

sudo CADDYPATH=/etc/ssl/caddy su -s /bin/sh http -c “/usr/bin/caddy -log stdout -agree=true -conf=/etc/caddy/caddy.conf -root=/var/tmp”

The user http is “nologin”, so I had to do it that way. Evidently it is really running as http because without the CADDYPATH it always tried to look for the certs in /srv/http which is the home for http.

So the problem only occurs when running through systemd! I thought I could use strace after figuring out how to run Caddy directly as http, but apparently I would have to strace through systemd, however the heck that works… sigh.

Edit: apparently I can’t use strace as it will drop capabilities.
I think the thing I have to solve is this old problem of not being able to set the caps in the systemd service: Starting with systemd: Failed at step NAMESPACE spawning /usr/local/bin/caddy: No such file or directory - #9 by Beluga
If I don’t comment out the caps settings, it still fails like in that post.

Edit2: I verified the problem is due to systemd upgrade by downgrading to 231. Now if I run it like in the past (the caps stuff commented out), it works fine.

1 Like
7

I got help from #systemd

I have this capshtest service to keep things simple:

[Service]
ExecStart=/sbin/capsh --print
AmbientCapabilities=CAP_NET_BIND_SERVICE
User=nobody

I also gave the capsh binary the setcap treatment.

I ran strace -f -p 1 which traces PID 1 + its children and then started my capshtest service in another terminal. This is the actual failure:

prctl(PR_CAP_AMBIENT, PR_CAP_AMBIENT_RAISE, CAP_NET_BIND_SERVICE, 0, 0) = -1 EINVAL (Invalid argument)

The man page says:
“prctl() is called with a first argument describing what to do (with values defined in <linux/prctl.h>), and further arguments with a significance depending on the first one.”

So why the invalid argument? I could not find any solution to this.

8

Searching for those issues on Github’s issue tracker will get you some more pointers. :wink:

  1. Try this with systemd v229 and later:
    https://github.com/mholt/caddy/blob/da8ae9e511a94d828877d6c12467f4dc8e2a4383/dist/init/linux-systemd/caddy.service
  2. Else, use capsh like this (the i is important):
    https://github.com/mholt/caddy/issues/528#issuecomment-261125303

Adjust www-data and 1000 to your environment. It could be user http on Arch Linux, or www, or nginx, or ….

Do not use setcap on the binary.

9

[quote=“wmark, post:8, topic:1198”]
Try this with systemd v229 and later:https://github.com/mholt/caddy/blob/da8ae9e511a94d828877d6c12467f4dc8e2a4383/dist/init/linux-systemd/caddy.service[/quote]

This is what I have been trying to do and failing innumerable times.

Tried it (staying with systemd 231, adjusted user and group ids, did setcap -r for Caddy binary, caps in caddy.service stay commented).
It says capsh is unable to set capabilities.
Here is the relevant strace output:

[pid 26543] capget({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, NULL) = 0
[pid 26543] capget({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, {effective=0, permitted=0, inheritable=0}) = 0
[pid 26543] capget({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, NULL) = 0
[pid 26543] capset({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, {effective=1<<CAP_SETPCAP, permitted=0, inheritable=0}) = -1 EPERM (Operation not permitted)
[pid 26543] capset({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, {effective=1<<CAP_NET_BIND_SERVICE|1<<CAP_LEASE, permitted=1<<CAP_NET_BIND_SERVICE|1<<CAP_LEASE, inheritable=1<<CAP_NET_BIND_SERVICE|1<<CAP_LEASE}) = -1 EPERM (Operation not permitted)
[pid 26543] write(2, “Unable to set capabilities [–ca”…, 71) = 71

Thank you for trying to help me with this horrible problem.

10

No offense, but I doubt you’ve used that version. (The file has been updated since, but I’ve linked to and meant the verbatim one.)

Don’t use setcap, though it wouldn’t have made any difference.
capsh does not allow you to add capabilities. Being root or a privileged user you use it to start something with a reduced capability set. It won’t work for user www-data or similar.

Don’t use systemd+capsh — just systemd.

11

I’ll try to explain myself better.

The differences of your file vs. master is:

  • master has StartLimitInterval=8640010 and - StartLimitBurst=5
  • master has the caps commented (the only relevant thing)

The caps in systemd have given me grief since I first tried using them in August 2016 and that is the whole point of this thread: I cannot stay with systemd 231 forever and from 232 onwards this stuff must work. I must get systemd working with the caps bits uncommented, if I want to update to systemd 232.

You said “don’t use setcap”, even though I only wanted to point out I removed the caps from the binary with setcap -r. It was a mistake to try to cram the capsh solution to systemd’s ExecStart and I understand that now. I misread your advice.

So do you have any idea, how I can debug the relevant error prctl(PR_CAP_AMBIENT, PR_CAP_AMBIENT_RAISE, CAP_NET_BIND_SERVICE, 0, 0) = -1 EINVAL (Invalid argument) ?

12

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.