I’m working on a Homebrew formula (package) for Caddy 2 so that Caddy 2 can be built and installed automatically with the Homebrew package manager. The Homebrew team unfortunately won’t accept the formula while Caddy 2 is still in beta, but it’s helpful for me nonetheless. And once 2.0.0 is released, we can immediately submit the formula to Homebrew.
The requirement of Homebrew is that all formulae must be built from source. The formula for Caddy 1 uses just go build, but that doesn’t embed the version information and therefore isn’t ideal.
I’ve tried the following installation commands now according to the instructions in your README:
cd cmd/caddy
go mod init caddy
go get github.com/caddyserver/caddy/v2@v#{version}
go build -o #{path}/caddy
It works great, but unfortunately the Homebrew formula linter won’t accept build steps that include go get. Which makes sense, as the tarball is already downloaded by Homebrew.
So the question is: Is there another way to make a Caddy build with the version number, but based on the locally downloaded repo? Unfortunately I don’t have any Go experience, so I’m hoping you can help me here.
Ah, now I get how it was supposed to work for the Caddy 1 Homebrew formula: It uses an -ldflags -X github.com/caddyserver/caddy/caddy/caddymain.gitTag=#{version} argument for go build inspired by the old build helper.
So the updated question is:
Is this build argument not supported anymore at all?
If it is still supported: What would it need to look like for Caddy 2?
If not: Is there an alternative way to “inject” the version information into the build process?
Hi Lukas, thanks for working on a homebrew formula!
Frankly, that’s a stupid policy. It represents a fundamental misunderstanding of what go get does. go get isn’t for downloading the project’s code, it’s for the project’s dependencies – for downloading versions and pinning them. The main Caddy module/package should be downloaded with git clone as usual (or whatever other means they use to download the source). go get is primarily used to modify go.mod and go.sum.
If I were you, I would try to convince the Homebrew team that forbidding go get is harmful, because go get invokes the Go module security tooling, which computes checksums over downloaded dependencies and ensures that your supply chain was not compromised. By not restricting the use of the Go toolchain, Homebrew is making it much easier for attackers to deploy malicious code by skipping checksum validations. Homebrew has deliberately skipped checksum validations before, putting millions at risk… so I’m hopeful they’ll change their policies here but I’m not optimistic, given their history. This could be a good chance for them to change their reputation. (More info)
FWIW, the way we build Caddy with version information is necessary due to a bug in Go.
I think there’s a misunderstanding though: Homebrew doesn’t prevent formulae from fetching their dependencies with go get, it just doesn’t allow calling go get directly from the formula script as Homebrew assumes that go get is being used to manually fetch dependencies that should instead be managed by the project that is being built itself.
The error message is:
Do not use `go get`. Please ask upstream to implement Go vendoring
Which is a false-positive, but well – that’s what static analysis can do.
The question is now: Can we find a pragmatic solution for the formula and/or the Caddy 2 code base? I have read the thread about the Go bug already, see my other comment above. It looks like it had worked with the -ldflags injection before, but doesn’t anymore.
Because the goal is exactly what you write: The Caddy source should still be downloaded from Git or tarball. I only need a way to tell go build which version information to embed into the binary. I have access to the version number in the formula build script, I only need a way to pass it to the build – ideally without having to use go get to avoid stupid discussions with the Homebrew team.
Here’s the draft of the v2 formula if you want to play around with it yourself:
class Caddy < Formula
desc "Alternative general-purpose HTTP/2 web server"
homepage "https://caddyserver.com/"
url "https://github.com/caddyserver/caddy/archive/v2.0.0-beta.15.tar.gz"
sha256 "9d1992f80a7c8c4a6c3784396d5345e8cb64c412581eee14cce1921258e2cc9d"
head "https://github.com/caddyserver/caddy.git", :branch => "v2"
depends_on "go" => :build
def install
ENV["GOOS"] = "darwin"
ENV["GOARCH"] = "amd64"
if build.head?
revision = version.commit
else
revision = "v#{version}"
end
Dir.chdir(buildpath/"cmd/caddy") do
system "go", "mod", "init", "caddy"
system "go", "get", "github.com/caddyserver/caddy/v2@#{revision}"
system "go", "build", "-o", bin/"caddy"
end
end
plist_options :manual => "caddy run --config #{HOMEBREW_PREFIX}/etc/Caddyfile"
def plist; <<~EOS
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>KeepAlive</key>
<true/>
<key>Label</key>
<string>#{plist_name}</string>
<key>ProgramArguments</key>
<array>
<string>#{opt_bin}/caddy</string>
<string>run</string>
<string>--config</string>
<string>#{etc}/Caddyfile</string>
</array>
<key>RunAtLoad</key>
<true/>
</dict>
</plist>
EOS
end
test do
require "socket"
server = TCPServer.new(0)
port = server.addr[1]
server.close
(testpath/"Caddyfile").write <<~EOS
127.0.0.1:#{port}
respond "Hello, world!"
EOS
pid = fork do
exec bin/"caddy", "run", "--config", "#{testpath}/Caddyfile"
end
sleep 2
assert_match "Hello, world!", shell_output("curl -s 127.0.0.1:#{port}")
assert_match "\":#{port}\"", shell_output("curl -s 127.0.0.1:2019/config/apps/http/servers/srv0/listen/0")
ensure
Process.kill("SIGINT", pid)
Process.wait(pid)
end
end
It works really well (including HEAD builds and brew test), but won’t be accepted in its current form because of the linting error (brew audit --new-formula caddy.rb).
So, wait, if you can’t call go get from the formula, how are you supposed to call go get? Calling go get is necessary AFAIK to build Caddy properly.
I’ve asked on the Gophers Slack but no good solutions so far.
Correct, because version information is embedded directly into the binary from Go modules… with the caveat of that bug.
To make sure I have this straight: the Caddy repo (source) is already on the machine, extracted into a folder, at the exact commit (version) that is tagged for release? If so, I suppose adding replace to the go.mod file could work, but — and I’m not sure on this, we’d have to verify — I believe that sidesteps all the checksum verification and kind of defeats the purpose of module security.
The best solution here is either for Go to fix their bug or for Homebrew to fix their policy. Actually, both.
But if replace doesn’t nullify the benefits of the checksum db, then we might be able to use that as a workaround.
I don’t think so. Calling just go build from the cmd/caddy directory works just fine as it will trigger the dependency download in the background. But of course that won’t embed version information.
Yes! Homebrew downloads the tarball, validates that against the SHA256 checksum and extracts it. It then runs build commands inside the source tree.
Sounds like something we could try. Tell me more: Which commands would I need to call? As I don’t have Go experience, it would be ideal if you sent me the list of shell commands that I only need to embed into the formula.
What I still don’t understand: Why did the workaround with passing -ldflags work in the past, but does not work anymore? Go had the same bug back then, didn’t it?
Yeah, but unfortunately that’s not a proper build. In future 2.x versions, Caddy will need that information to do security checks and upgrades and stuff. We already rely on it heavily for troubleshooting/debugging purposes. So we need the version information.
That was pre-Go-modules, when version information was set like this in code:
var caddyVersion = "v0.11.3"
This was terrible because the source code has to change in order to set the version! The -ldflags trick told Go to change that variable’s value at compile-time. But with Go modules, the compiler embeds module information directly into the binary: debug package - runtime/debug - Go Packages
The first one at least still shows the version, which is the important part. Not sure what’s going on in that second one… the second one would not be acceptable. The hash would be nice to have as well, to show that we are running a binary with a verified code base.
I’m still trying to figure out if using replace nullifies the benefits of the checksum db, i.e. turns off module security. If so, then this will not be an acceptable solution either, I’m afraid. (The Go Modules wiki page isn’t clear on this point, that I can find so far.)
It’s a build with the version being the commit hash, in this case 1324da2 (which is being used if brew install --HEAD caddy is being run to install the latest development version from the repo). The result is the same if I run go get github.com/caddyserver/caddy/v2@1324da2 instead of the replace dance. So if it’s unacceptable, it’s a general bug in Caddy or Go.
I’m keeping both variants (with go get and with the replace dance) for now. It all becomes relevant when the stable Caddy 2.0.0 is released at the earliest, so we have some time until then. If you come across a better solution until then, please let me know.
Ohh, you’re right – I was referring to /private/tmp/caddy-20200301-58177-itm3dn on the RHS. But now that you mention it, maybe the second line is fine too.
Sounds good, thanks! If you are up to it, I would encourage trying to get Homebrew to fix their policy regarding go get in the meantime. It would make things so much easier (and more secure, I’m pretty sure).
If that makes sense from your perspective, sure! Would probably be better if you were the one to propose that though as I don’t have Go experience and therefore can’t speak for the Go community.
Maybe they are willing to change their policy if we explain that there are valid use-cases for go get even if Go vendoring is used.
Well, I can’t speak for the Go community either, but I’d be willing to lend my voice in some petition for a change of that policy.
In chatting with some other Go programmers, they appear to have confirmed that checksums for local replacements using replace do not appear in the go.sum file.
I’m not sure what the implications of this are yet, I need to do some more reading.
In talking to people smarter than me (in the Gophers Slack about modules), I think I got the information I was looking for (though I haven’t done a bunch of tests, etc., to verify it myself).
It seems that replace should be safe in our case. Whatever is git clone'd (or downloaded tarball) will be verified, but we have to trust the actual replace line to point to the right local folder (which we can trust if we also trust the downloader / script that pulls the Caddy source code). The final binary will know this replacement took place (as you saw when you ran caddy version), but go build would not have succeeded if the checksums failed, so we can still trust the resulting binary.
The key is that we need to trust the downloader and whatever person/script added the replace to go.mod.
Yeah, that’s a necessary prerequisite anyway as a manipulated Homebrew formula could manipulate the binary in any way it wants. That’s the case with every package manager though.
Which method do you prefer? I think the script would be even cleaner as that would give us the proper hash that a) doesn’t look as weird and b) can be validated. And it would simplify the manual builds as well.
The issue is: The formula would need to clone that repo just for that one file. The same for everyone who wants to build Caddy from source. And also the script would need an additional argument for the base path of the Caddy repo as it cannot autodetect it if it’s outside of the repo. That makes it more complex.
If you still think putting it into the separate repo is the best way, I’m happy to send a PR.
