Is my config okay or do I need to install the Cloudflare module?

Help
1

1. The problem I’m having:

I am hosting a VPS with Ubuntu 22 where I have set up Immich, Jellyfin and Audiobookshelf with Docker Compose. I was earlier using Wireguard (set up with PiVPN script) to access various services but wanted to connect without Wireguard also, so looked into reverse proxy. Caddy seemed the simplest, so followed the instructions on Caddy’s official docs for stable release (running version 2.8.4 right now). Then I updated the Caddyfile with the following information.

immich.domain.com {
        reverse_proxy 10.0.0.58:1234
}

abs.domain.com {
        reverse_proxy 10.0.0.58:2345
}

jellyfin.domain.com {
        reverse_proxy 10.0.0.58:3456
}

This is the entire Caddy file. And now when I go to the Immich, Audiobookshelf or Jellyfin subdomains, I can access those services without problems.

Cloudflare is managing the DNS for my domain. Earlier I was using DNS proxy (orange cloud) on Cloudflare but I guess it goes against their T&C to proxy all these service’s data through them, so I just turned off DNS proxy and now I have DNS only (grey cloud). While setting up Caddy while following some YouTube tutorial and a tutorial on a website, I saw them mention Cloudflare API key. From what I understood, with this, I won’t have to manually refresh the Let’s Encrypt cert every 90 days. So I created an API key. However, those tutorials showed using the Cloudflare module and that’s where I am stuck.

tls {
  dns cloudflare {my_api_token}
}

I added the code above under each subdomain entry, so it looked like this.

immich.domain.com {
        reverse_proxy 10.0.0.58:1234
tls {
  dns cloudflare {my_api_token}
}
}

abs.domain.com {
        reverse_proxy 10.0.0.58:2345
tls {
  dns cloudflare {my_api_token}
}
}

jellyfin.domain.com {
        reverse_proxy 10.0.0.58:3456
tls {
  dns cloudflare {CF_API_TOKEN}
}
}

However, when I try reloading the config, I get this error.

Job for caddy.service failed.
See "systemctl status caddy.service" and "journalctl -xeu caddy.service" for details.

Do I need to even include the Cloudflare token? If yes, do I need to uninstall Caddy and then install some other version of Caddy with the Cloudflare module built in? I tried following the instructions mentioned here, i.e., went to the Caddy download page, chose Linux arm64 structure and then chose Cloudflare. Then opened terminal and ran the following command.

sudo curl -o /usr/bin/caddy -L https://caddyserver.com/api/download?os=linux&arch=arm64&p=github.com%2Fcaddy-dns%2Fcloudflare&idempotency=14732388869855

However, when I run this command, terminal freezes and I see this response.

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: Failed to create the file /usr/bin/caddy: Text file busy
100  5921    0  5921    0     0   6222      0 --:--:-- --:--:-- --:--:--  6232
curl: (23) Failure writing output to destination

If I have to reinstall Caddy, could you please tell me how to remove the current installation? I’m fairly a newbie at this, so please bear with me.

Also, I opened ports 80 and 443 for Caddy. But right now it’s open to the whole world. Can I limit access to my websites to a few IP addresses and a couple of CIDRs with firewall rules?

2. Error messages and/or full log output:

3. Caddy version: v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy:

Followed instructions on Install — Caddy Documentation for the stable release

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

a. System environment:

Ubuntu 22.04.4 LTS on Oracle Cloud with Oracle Cloud Ampere CPU (free tier), Docker Engine Community version 26.1.3, Docker Compose version 2.27.0

b. Command:

sudo nano /etc/caddy/Caddyfile to edit the Caddy file
sudo systemctl reload caddy to reload Caddy

c. Service/unit/compose file:

d. My complete Caddy config:

immich.domain.com {
        reverse_proxy 10.0.0.58:1234
}

abs.domain.com {
        reverse_proxy 10.0.0.58:2345
}

jellyfin.domain.com {
        reverse_proxy 10.0.0.58:3456
}

5. Links to relevant resources:

2

Howdy @carrynewb! There’s a fair bit of meat to your post so I’ll try run through really quick.

Not exactly. ACME certificate requisition doesn’t require your DNS provider’s API access at all. If your website is internet-facing, it can just try port 80 or port 443 and talk directly to your server. Caddy will still automatically update your certificate on a schedule and you won’t need to manually intervene for this to happen.

A Cloudflare API key enables the use of DNS validation (as opposed to HTTP or TLS-ALPN validation), which is a different process that doesn’t require your webserver to be publicly accessible.

Strictly speaking, no. It should work just fine without it. Even behind the “orange cloud” - Caddy might have a very brief delay while it grapples with that, but it’ll figure it out pretty quick and get it sorted.

Yes. You’ll need to replace the Caddy binary that’s currently installed with one with the module.

You might be able to do this really easily by running: sudo caddy add-package github.com/caddy-dns/cloudflare

This might be because Caddy is running already. You might need to stop it first before replacing it with that curl command.

Since you installed with apt, you should be able to remove with: sudo apt remove caddy

But you may want to leave it installed because it comes with systemd service configuration, etc.; it might be preferable just to replace the binary, i.e. with that curl command or with caddy add-package.

You could definitely do this!

It’s possible in Caddy itself, using request matchers to reject requests from IPs that aren’t allowed. It’s far, far more efficient to do this at your firewall, though, which is beyond the scope of Caddy.

This introduces a bit of a problem, though; it WILL break LetsEncrypt validation on ports 80/443 unless you specifically allow all the LetsEncrypt IPs. Then you will need the Cloudflare API key and the module.

1 Like
3

Not exactly. ACME certificate requisition doesn’t require your DNS provider’s API access at all. If your website is internet-facing, it can just try port 80 or port 443 and talk directly to your server. Caddy will still automatically update your certificate on a schedule and you won’t need to manually intervene for this to happen.
A Cloudflare API key enables the use of DNS validation (as opposed to HTTP or TLS-ALPN validation), which is a different process that doesn’t require your webserver to be publicly accessible.

So if I understood this correctly, Caddy can totally renew the Let’s Encrypt cert without the API key since my server has access ports 80 and 443 open, even behind the Cloudflare orange cloud. What’s the use of the API key and DNS validation then? Could you share a setup where it’ll be useful?

You might be able to do this really easily by running:
sudo caddy add-package github.com/caddy-dns/cloudflare

Thank you. This worked wonderfully. I ran this command, then just rebooted the system once and ran this command to see non-standard modules, and Cloudflare showed up.

caddy list-modules --skip-standard

Then I modified the Caddyfile and it looks like this now. By the way, are those curly brackets supposed to be there?

immich.domain.com {
        reverse_proxy 10.0.0.58:1234
tls {
  dns cloudflare {a1b2c3d4e5f6g7}
}
}

abs.domain.com {
        reverse_proxy 10.0.0.58:2345
tls {
  dns cloudflare {a1b2c3d4e5f6g7}
}
}

jellyfin.domain.com {
        reverse_proxy 10.0.0.58:3456
tls {
  dns cloudflare {a1b2c3d4e5f6g7}
}
}

And then I ran sudo systemctl reload caddy and didn’t get any error, which is great. Is there a way to know if I have set this up correctly, though? I visited Cloudflare’s DNS area but I’m not seeing any entries there that I didn’t make.

This introduces a bit of a problem, though; it WILL break LetsEncrypt validation on ports 80/443 unless you specifically allow all the LetsEncrypt IPs. Then you will need the Cloudflare API key and the module.

I did a bit of Googling and looks like the LetsEncrypt IPs is a dead-end. That pushed me down a rabbit hole where I discovered Crowdsec, Authentik, Fail2ban and Authelia, and it was so overwhelming. Any tips for which of these would offer the protection I am looking for while also being easy/relatively easy to set up in my use case?

Also, let’s say I don’t want to deal with any of these potential solutions and just want renew the cert manually every 85 days or so like I do with my AdGuard Home. So, let’s say, I limit access to ports 80 and 443 for a few IP addresses and a few CIDRs. On every 80th or 85th day, can I disable the firewall and run some command to get Caddy to update the cert at that time only? If yes, how do I do it?

4

So I did some more looking around and made a couple of changes to the Caddyfile. It now looks like this.

{
acme_dns cloudflare a1b2c3d4e5f6g7h8i9j10
email myemail@domain.com
}

immich.domain.com {
        reverse_proxy 10.0.0.58:1234
}

abs.domain.com {
        reverse_proxy 10.0.0.58:2345
}

jellyfin.domain.com {
        reverse_proxy 10.0.0.58:3456
}

And then I ran sudo find /var/lib/caddy/.local/share/caddy/certificates -type f -exec rm {} \; to remove the currently issued certs (found it when I searched Google for instructions to force renew certs using Caddy. And then just restarted Caddy with sudo systemctl restart caddy. When I visited the DNS records page of Cloudflare, it showed three more TXT entries there. However, after about 5 minutes, those TXT entries were gone. Is this expected?

Also, what would happen if I add another subdomain for another service, let’s say Uptime Kuma (uptime.domain.com) and then restart Caddy? Will Caddy try to generate certs for every domain or only for uptime.domain.com?

5

One detail, you need to make sure to turn off Always Use HTTPS · Cloudflare SSL/TLS docs in Cloudflare so that it doesn’t attempt to redirect HTTP requests from Let’s Encrypt to HTTPS, breaking the ACME HTTP validation. Also ACME TLS-ALPN cannot work while using orange-cloud because Cloudflare intercepts the TLS connection using their own certs, so the connection with the ALPN stuff doesn’t reach Caddy.

The two main usecases are either when you need a wildcard certificate, OR if your server isn’t publicly accessible (but you still want a publicly-trusted cert). The ACME DNS challenge is the only way to prove that you control the entire domain and not just a single domain, so it’s the only way to get a wildcard cert.

Many users of Cloudflare just use the DNS challenge anyway because they don’t realize they can turn off “Always Use HTTPS” (it’s not obvious, and we don’t document it well enough) but many also want wildcard certs for a variety of reasons.

Keep in mind that using that command, next time there’s a Caddy upgrade your /usr/bin/caddy will be replaced with the one from the apt repo. To resolve that, follow these instructions instead to set up a diversion: Build from source — Caddy Documentation

No they should not be. The braces are special syntax for Caddy placeholders. If you’re using environment variables, then you could use {env.CLOUDFLARE_TOKEN} to have it replaced at runtime with the value from the env var. But if not, it should just be the raw token value with no braces.

If you can make requests and it reaches Caddy and you get a successful response, you’re good to go. Look at your logs for any possible warnings or errors.

Yeah, LE doesn’t publish their IP ranges on purpose. They don’t want people to depend on them because then it makes people’s setups brittle if they later add more IP ranges that they issue from. They recently started performing challenge verification from two new regions recently, and some users had breakages because they were relying on not-officially-published IP range lists.

IMO you don’t need any of those. Only use them if you have an actual need to use them, don’t just use them blindly out of paranoia.

Why tho? Just let Caddy automate it as it normally does. Caddy starts attempting renewal once the cert is 2/3 of the way through its lifetime (so after 60 days) and keeps trying until expiry, with backoff. There’s no reason to manage certs manually anymore, in 2024. ACME solves that completely. Always automate everything you can automate. It removes the human factor. Humans are imperfect.

I don’t recommend that. The hassle is not worth the benefit.

Yes, that’s how the DNS challenge works. Caddy writes a TXT record for each domain with a special value given to Caddy by LE, then LE does a DNS lookup to find that value, to prove that “yeah ok the server I talked to does have control of that domain”. Then Caddy cleans it up afterwards by removing the TXT records once it’s no longer needed.

Only for the new one, because you already have valid certs in storage for the other domains.

But don’t “restart” Caddy, you should “reload” Caddy so that you don’t have downtime (a restart involves stopping then starting Caddy, a reload is an in-memory config change).

1 Like
6

Thanks a ton for the detailed answer. I greatly appreciate it.

One detail, you need to make sure to turn off Always Use HTTPS.

Did that. Turned it on yesterday and turned it off now.

Keep in mind that using that command, next time there’s a Caddy upgrade your /usr/bin/caddy will be replaced with the one from the apt repo. To resolve that, follow these instructions instead to set up a diversion: Build from source — Caddy Documentation

So if I am getting this correctly, if I caddy upgrade when there’s a new version release, I’ll run into trouble. To not run into trouble either now or when there’s an upgrade, I should go to that link and run each command one by one. Right?

On the same page, the following is mentioned.

To upgrade Caddy after this point, you may run caddy upgrade. This attempts to download a build with the same plugins as your current build, with the latest version of Caddy, then replace the current binary with the new one.

Does it mean after the first time I upgrade by following instructions there, moving forward whenever there is an upgrade, I should only caddy upgrade?

IMO you don’t need any of those. Only use them if you have an actual need to use them, don’t just use them blindly out of paranoia.

Actually, until now I had DNS Proxy on in Cloudflare and Cloudflare is showing me bot activity in Security under Events. That’s why I was looking into it.

SCR-20240607-csvh
SCR-20240607-csvh2070×2208 414 KB

I don’t recommend that. The hassle is not worth the benefit.

I understand that and I agree with you. One thing that held me back from switching from a VPN based solution to a reverse proxy based solution was the effort needed for backend tasks like this. However, is this a possible solution to the bot or DDoS concern that I mentioned (restricting and unrestricting ports 80 and 443)?

The two main usecases are either when you need a wildcard certificate, OR if your server isn’t publicly accessible (but you still want a publicly-trusted cert).

I see. My use case doesn’t fit either of the two situations. So I guess I have no reason to have it on. However, I did set it up properly a few hours ago while I was looking for setting up Crowdsec with Caddy, when I updated this thread. After that, I came across this Reddit post and followed the instructions on my test server which worked fine. So I tried doing the same on my main server as well. In my main server, now the config looks like this.

{
acme_dns cloudflare a1b2c3d4e5f6g7h8i9j10
email myemail@domain.com
}

{
  crowdsec {
    api_key my_long_api_key
  }
}

immich.domain.com {
        reverse_proxy 10.0.0.58:1234
}

abs.domain.com {
        reverse_proxy 10.0.0.58:2345
}

jellyfin.domain.com {
        reverse_proxy 10.0.0.58:3456
}

As per the instructions on the Reddit post, I needed to sudo systemctl restart caddy. When I try to do that now, I get this error.

Job for caddy.service failed because the control process exited with error code.
See "systemctl status caddy.service" and "journalctl -xeu caddy.service" for details.

However, if I comment out the first four lines of the Caddyfile with the api key and the email address, Caddy restarts just fine. I wasn’t sure if the order was important, so I tried putting the api key and email address after the crowdsec code as well, and I am still getting the error.

7

Rather if you sudo apt upgrade then it’ll replace your current /usr/bin/caddy with the latest vanilla build of Caddy, losing the plugins you added.

If you run caddy upgrade it downloads the latest version but with the plugins currently installed in the binary, so if you lost the plugins due to sudo apt upgrade you’d need to add-package again to get them back.

If you set up a diversion then you can safely do caddy upgrade since it replaces the custom binary only, leaving the vanilla one alone, and the custom one “overlays” the vanilla one (i.e. /usr/bin/caddy becomes a symlink to /usr/bin/caddy.custom and the vanilla one is a separate file).

Don’t worry about bots/crawlers. They’re harmless 99.99% of the time. It’s only a problem if they actually try to target you en-masse because they think they can gain something from it. But you’re just a random person with a home server, that’s not valuable to them.

Well, do what that tells you – check Caddy’s logs, it’ll say why it failed to start. Also read Keep Caddy Running — Caddy Documentation

2 Likes
8

So I was setting up another container using Docker compose and saw there were some things that I could upgrade. Without thinking much, I ended up sudo apt update && sudo apt upgrade -y and right after that, I remembered that I needed to do some modifications for Caddy. I went to the page you had linked and ran sudo dpkg-divert --divert /usr/bin/caddy.default --rename /usr/bin/caddy and got a response Leaving 'local diversion of /usr/bin/caddy to /usr/bin/caddy.default'. After this, however, when I ran sudo mv ./caddy /usr/bin/caddy.custom, I got the response mv: cannot stat './caddy': No such file or directory. I am at the /home/ubuntu directory while running these commands. Caddy is still running in the background since I can access domains that are behind the proxy. And running sudo systemctl status caddy shows that Caddy is active.

9

Those steps assume you have downloaded or built a custom build it Caddy and have it in your current directory (i.e. ./caddy) ready to move to where it needs to go. You can download the build from Download Caddy or build it with xcaddy. Or make a copy of the vanilla binary with cp then run ./caddy add-package which is kinda the same as downloading it from Download Caddy. Then you can continue with those steps.

1 Like
10

I tried following the steps mentioned. I am currently using cloudflare and crowdsec plugins. I set the path with export PATH=$PATH:/home/ubuntu/go/bin and then did a build with xcaddy

xcaddy build \
    --with github.com/hslatman/caddy-crowdsec-bouncer
    --with github.com/caddy-dns/cloudflare

However, after this, I got this error.

"2024/06/07 17:18:40 [INFO] absolute output file path: /home/ubuntu/caddy
2024/06/07 17:18:40 [INFO] Temporary folder: /tmp/buildenv_2024-06-07-1718.4114256224
2024/06/07 17:18:40 [INFO] Writing main module: /tmp/buildenv_2024-06-07-1718.4114256224/main.go
package main

import (
        caddycmd "github.com/caddyserver/caddy/v2/cmd"

        // plug in Caddy modules here
        _ "github.com/caddyserver/caddy/v2/modules/standard"
        _ "github.com/hslatman/caddy-crowdsec-bouncer"
)

func main() {
        caddycmd.Main()
}
2024/06/07 17:18:40 [INFO] Initializing Go module
2024/06/07 17:18:40 [INFO] exec (timeout=0s): /usr/bin/go mod init caddy 
go: creating new go.mod: module caddy
go: to add module requirements and sums:
        go mod tidy
2024/06/07 17:18:40 [INFO] Pinning versions
2024/06/07 17:18:40 [INFO] exec (timeout=0s): /usr/bin/go get -d -v github.com/caddyserver/caddy/v2 
go: downloading github.com/caddyserver/caddy v1.0.5
go: downloading github.com/caddyserver/caddy/v2 v2.8.4
go: downloading github.com/caddyserver/certmagic v0.21.3
go: downloading github.com/cespare/xxhash/v2 v2.2.0
go: downloading github.com/google/uuid v1.6.0
go: downloading github.com/cespare/xxhash v1.1.0
go: downloading github.com/prometheus/client_golang v1.19.1
go: downloading github.com/quic-go/quic-go v0.44.0
go: downloading go.uber.org/zap v1.27.0
go: downloading go.uber.org/zap/exp v0.2.0
go: downloading golang.org/x/sys v0.20.0
go: downloading golang.org/x/term v0.20.0
go: downloading golang.org/x/time v0.5.0
go: downloading github.com/beorn7/perks v1.0.1
go: downloading github.com/prometheus/client_model v0.5.0
go: downloading github.com/prometheus/common v0.48.0
go: downloading github.com/prometheus/procfs v0.12.0
go: downloading google.golang.org/protobuf v1.34.1
go: downloading github.com/caddyserver/zerossl v0.1.3
go: downloading github.com/klauspost/cpuid/v2 v2.2.7
go: downloading github.com/libdns/libdns v0.2.2
go: downloading github.com/mholt/acmez/v2 v2.0.1
go: downloading github.com/miekg/dns v1.1.59
go: downloading github.com/zeebo/blake3 v0.2.3
go: downloading golang.org/x/crypto v0.23.0
go: downloading golang.org/x/net v0.25.0
go: downloading github.com/quic-go/qpack v0.4.0
go: downloading github.com/onsi/ginkgo/v2 v2.13.2
go: downloading go.uber.org/mock v0.4.0
go: downloading golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842
go: downloading go.uber.org/multierr v1.11.0
go: downloading golang.org/x/tools v0.21.0
go: downloading golang.org/x/text v0.15.0
go: downloading golang.org/x/mod v0.17.0
go: downloading golang.org/x/sync v0.7.0
go: downloading github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572
go: downloading github.com/google/pprof v0.0.0-20231212022811-ec68065c825e
github.com/caddyserver/caddy/v2 imports
        log/slog: package log/slog is not in GOROOT (/usr/lib/go-1.18/src/log/slog)
github.com/caddyserver/caddy/v2 imports
        github.com/quic-go/quic-go imports
        github.com/quic-go/quic-go/internal/wire imports
        slices: package slices is not in GOROOT (/usr/lib/go-1.18/src/slices)
2024/06/07 17:19:39 [FATAL] exit status 1
--with: command not found"

I also went to caddy download page to download the custom binary and tried to download it with curl https://caddyserver.com/api/download?os=linux&arch=arm64&p=github.com%2Fcaddy-dns%2Fcloudflare&p=github.com%2Fhslatman%2Fcaddy-crowdsec-bouncer&idempotency=28084318048133 but got this response

[1] 3057
[2] 3058
[3] 3059
[4] 3060
ubuntu@my-media:~$ Warning: Binary output can mess up your terminal. Use "--output -" to tell 
Warning: curl to output it to your terminal anyway, or consider "--output

Could you tell me in simpler terms what I’m doing wrong? Thank you.

11

You’re probably using too old a Go version. Don’t use the Go version from your OS’s package manager, it’s usually way too old. I like to use GitHub - udhos/update-golang: update-golang is a script to easily fetch and install new Golang releases with minimum system intrusion to install it, automates the process with a simple shell script.

Curl is telling you what to do, you need to use --output to tell curl where to write the file. If you just use curl without any flags, it doesn’t try to download a file, it just tries to write the output to stdout which is not useful for you in this case.

1 Like
12

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.