Build from source and include custom modification

Help
1

I want to modify https://github.com/caddyserver/caddy/blob/2cac3c5491e6428441ecf668cc4f5a86e67ed9b3/modules/caddyhttp/fileserver/browse.go#L148 to allow for more than 10000 files. So I did the following:

  • git clone https://github.com/caddyserver/caddy
  • modify browse.go
  • cd caddy/cmd/caddy/
  • add extra modules to main.go
  • go mod init caddy
  • go mod tidy
  • go build
  • setcap caddy and put it in /usr/bin/

However, when I run caddy and browse a large directory, I still see a limit of 10000 files.

Screenshot from 2023-09-09 17-25-54

Why does not the modified code get used when doing go build? I have cleared ~/.cache/go-build/ and also tried go build -x -a.

2

You can either just cd cmd/caddy and run go build there, or you could use xcaddy with xcaddy build --with github.com/caddyserver/caddy/v2=../path/to/your/checkout or if you commit to your fork you can do xcaddy build --with github.com/caddyserver/caddy/v2=github.com/your-fork/caddy/v2

1 Like
3

You can either just cd cmd/caddy and run go build there

This did not work. It seems that go simply downloads all sources from github and does not use the sources in upper directory.

you could use xcaddy with xcaddy build --with github.com/caddyserver/caddy/v2=../path/to/your/checkout

This worked! Thank you :slight_smile:

Screenshot_20230910_100055_Opera
Screenshot_20230910_100055_Opera1440×1249 81.4 KB

4

I do have a query about performance. It takes several minutes to generate the file listing with such large directories. Is there some way to improve this?

There seems to be several lookups per file (most are symlinks to files in subdirs)

10:41:59.755329 newfstatat(AT_FDCWD</>, "/var/www/domains/mirrors.tnonline.net/gentoo/gentoo-distfiles/distfiles/tl-beamerthemejltree-2021.tar.xz", {st_mode=S_IFLNK|0777, st_size=35, ...}, AT_SYMLINK_NOFOLLOW) = 0 <0.000040>
10:41:59.755418 newfstatat(AT_FDCWD</>, "/var/www/domains/mirrors.tnonline.net/gentoo/gentoo-distfiles/distfiles/tl-beamerthemejltree-2021.tar.xz", {st_mode=S_IFREG|0644, st_size=1500, ...}, 0) = 0 <0.000049>
10:41:59.755532 newfstatat(AT_FDCWD</>, "/var/www/domains/mirrors.tnonline.net/gentoo/gentoo-distfiles/distfiles/tl-beamerthemejltree-2021.tar.xz", {st_mode=S_IFREG|0644, st_size=1500, ...}, 0) = 0 <0.000014>
10:41:59.755603 newfstatat(AT_FDCWD</>, ".", {st_mode=S_IFDIR|0755, st_size=318, ...},
0) = 0 <0.000025>
10:41:59.755672 newfstatat(AT_FDCWD</>, "/", {st_mode=S_IFDIR|0755, st_size=318, ...},
0) = 0 <0.000021>
10:41:59.755754 newfstatat(AT_FDCWD</>, "/var/www/domains/mirrors.tnonline.net/gentoo/gentoo-distfiles/distfiles/tl-beamerthemenirma-2021.tar.xz", {st_mode=S_IFLNK|0777, st_size=34, ...}, AT_SYMLINK_NOFOLLOW) = 0 <0.000024>
10:41:59.755837 newfstatat(AT_FDCWD</>, "/var/www/domains/mirrors.tnonline.net/gentoo/gentoo-distfiles/distfiles/tl-beamerthemenirma-2021.tar.xz", {st_mode=S_IFREG|0644, st_size=2704, ...}, 0) = 0 <0.000053>
10:41:59.755942 newfstatat(AT_FDCWD</>, "/var/www/domains/mirrors.tnonline.net/gentoo/gentoo-distfiles/distfiles/tl-beamerthemenirma-2021.tar.xz", {st_mode=S_IFREG|0644, st_size=2704, ...}, 0) = 0 <0.000024>
10:41:59.756026 newfstatat(AT_FDCWD</>, ".", {st_mode=S_IFDIR|0755, st_size=318, ...},
0) = 0 <0.000011>
10:41:59.756091 newfstatat(AT_FDCWD</>, "/", {st_mode=S_IFDIR|0755, st_size=318, ...},
0) = 0 <0.000025>

For example do we need to lookup symlink targets instead of simply displaying a link in the dirlisting (like ls -l does). (Enhancement: file-server browser, allow for listing symlinks as symlinks · Issue #5810 · caddyserver/caddy · GitHub)

Also, is it nessecary to do fstatat on . and / after each file?

The complete file list is available at https://mirrors.tnonline.net/gentoo/gentoo-distfiles.ls-laFR.txt

5

I think you’re not running the correct executable after building it. How are you running it?

6

Well, go ouputs the binary in cmd/caddy. I simply replace the /usr/bin/caddy and starts the caddy service with rc-service caddy start. And yes, I do not have any other caddy binaries in the system.

xcaddy also produces a caddy binary in current directory. I do the same with this and copy it to /usr/bin/caddy, which works good.

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.