Global directives on custom ports

Help
1

1. The problem I’m having:

Switch to Caddy they said. It’ll be easy they said. Jokes aside, I’m having a lot of trouble switching from Apache to Caddy. Namely with global directives. I understand my use case is very non-traditional, which is why I’m here asking for help.

I have a lot of websites being served by Apache. A mixture of reverse proxies and local directories. In order to safely test out my configuration with Caddy, I’ve decided to run both Apache and Caddy simultaneously for the duration of the transition. Having both serve the same websites on different ports allows me to test both versions side by side in the browser.

Well Caddy doesn’t seem to like being told to listen on custom ports. I’ve seen other posts discussing this and they’re always advised against doing it, instead of being given an actual solution. And sure in some cases there might be better ways, but not in mine. And the HTTPS issue is easily countered with a DNS plugin - provided it’s configured correctly.

I couldn’t make the official installer page work with the plugins I chose, so I compiled it myself. But no mater what combination of directives I used I can’t get it to work.

As you can see, I also moved global headers into a separate file. Please let me know if there’s better ways to address any of these challenges.

2. Error messages and/or full log output:

Error: adapting config using caddyfile: server block without any key is global configuration, and if used, it must be first

3. Caddy version:

v2.8.1 h1:UVWB6J5f/GwHPyvdTrm0uM7YhfaWb4Ztdrp/z6ROHsM=

4. How I installed and ran Caddy:

Compiled with plugins.

a. System environment:

AlmaLinux 9, amd64, systemd.

b. Command:

systemctl start caddy

c. Service/unit/compose file:

[Unit]
Description=Caddy Web Server
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
User=apache
Group=apache
ExecStart=/etc/caddy/caddy run --config /etc/caddy/Caddyfile --adapter caddyfile
ExecReload=/etc/caddy/caddy run --config /etc/caddy/Caddyfile --adapter caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

{
        debug
        storage file_system /var/www/html
        log default {
                output file /var/log/caddy/caddy.log {
                        roll_keep_for 168h
                }
                include
                level debug
                format json
        }
        http_port 8888
        https_port 4444
        email [redacted]
        acme_dns cloudflare [redacted]
}
import headers.caddy

//site blocks

5. Links to relevant resources:

2

Please show your entire configuration. I’m not sure what headers.caddy is meant to contain, and it’s relevant to the error you might be getting. Are you sure that’s the entire error message? Show all your logs.

Please use the latest version, v2.8.4

Which plugins? How did you build it? What was your command you used to build?

Unfortunately your post is quite vague about the problems you’re having. You alluded to a variety of issues but barely actually showed evidence of those problems, so I’m not sure how to help here. We need more detail to work with.

1 Like
3

So I take it my syntax is actually correct? Because that was my reasoning for posting here. I guess I’ll go try compiling the latest version. I’ll be sure to document the entire process for you too.

These are the contents of headers.caddy:

{
  X-Frame-Options "DENY"
  X-Content-Type-Options "nosniff"
  Referrer-Policy "strict-origin-when-cross-origin"
  Content-Security-Policy "default-src 'self'; style-src 'self' 'unsafe-inline'; object-src: none;"
}

And these are the plugins I’m trying to get work:

mercure
vulcain
cloudflare
replace-response
security
crowdsec
jwt
webhook
git
geoip
basic auth filter
frankenphp
redis
cbrotli
minifier
hitcounter
cookieflag
floaty
image processor
image proxy
supervisor

I was only able to build with I think half of these though.

4

Not necessarily, because you didn’t post your whole config. I can’t validate that. The error might have been in the part you omitted, or in combination with it.

Then that’s your problem, you can’t have a block of config like that floating at the top level. Please review the structure of a Caddyfile. Caddyfile Concepts — Caddy Documentation You can only have one block without a key, i.e. global options.

If you want to apply headers, you need to use the header directive and it must go within a site block.

Do you really need all those? Only include plugins you actually need.

1 Like
5 
Error: adapting config using caddyfile: /etc/caddy/Caddyfile:16: unrecognized global option: header

Like I said, I only moved the headers into a separate file because I couldn’t get them to work in the top block. I didn’t go out of my way to do something so unintuitive for the fun of it. And I’m not sure what the number of plugins I’m using has to do with anything.

6

Yeah header is a directive, not a global option. Directives go in site blocks. As I said, please review the structure of the Caddyfile.

I’m just saying take things one step at a time. Start simple and build up from there. Don’t try to throw the whole kitchen sink from the start.

1 Like
7

So there’s no way to set global headers? I have to repeat the same ones for every single website in addition to the site-specific ones?

Believe it or not I have read it countless times over the last few days and I still ended up here. Telling me to read it again is extremely unhelpful when it clearly hasn’t helped all those times. At some point you have to acknowledge your documentation is confusing and that repeatedly referring people to it instead of answering their direct specific questions is a form of gaslighting.

Since my very first post here I’m talking about global directives. Hell it’s in the title. The first post makes it clear that I’m struggling with global directives, including headers. And yet you still sat there asking for more information being unhelpful and blaming me for it when you’ve refused to be of any help whatsoever at every step of the way.

Telling someone they don’t understand something and directing them to the public documentation when the entire purpose of this forum is allegedly “Help” and they had to go out of their way and create an account and give you their data and wait to be approved just to post here is pretty funny. Your perspective on unnecessary friction doesn’t at all seem aligned with the apparent goal of Caddy of simplicity and straightforwardness.

8

Use snippets to deduplicate site config.

Caddy does not allow globally configured HTTP routes because it we do that, inevitably someone will immediately ask “okay but how do I use global routes but then opt out of it for just one site” and then the level of complexity balloons. Much simpler for every site to always be a blank slate, and you can pull in one or more snippets as needed which may or may not apply to every site. You have much more control that way.

You’re being needlessly rude. I’m just trying to help. I think the structure is quite clear, it doesn’t lie. It’s exactly as it is shown.

In this case, I think you made an assumption about something that wasn’t said in the docs, then encountered an error. That error is pretty self explanatory IMO, it’s telling you quite clearly that you have a block (i.e. a section with { }) with no key (i.e. no name or address in front of the {). That is all terminology that is defined on the concepts page, so I think it’s quite clear if you read through it what the error means.

There’s no such thing as “global directives”. There’s “global options” (which go within the “global options block”) and “site directives” (which go within “site blocks”, or snippets). The structure is clear about that as well.

2 Likes
9

For the record, the approval step was only recent to battle an extreme spam attack from 2 Indian ISPs who don’t acknowledge reports at all for the last 3 weeks. It’s not like we have fun assessing a queue of registrations by checking each one’s IP address, which ASN is it part of, does the email look legit, and all other kinds of factors. It consumes time and energy.

2 Likes
10

Oh no I’m sure you guys have a very good reason for doing it. I actually attempted to make a throwaway account with a fake email address but when I saw the approval step I figured it’ll never get approved so I used my real information. My point was that it’s so much easier to just read the publicly available documentation which is available instantly without having to wait to be approved and give up personal information and then expose even more personal information in the public forum and wait hoping someone will respond and help. I wouldn’t have gone through all that trouble if I could get my answers in the documentation. Which is why it’s frustrating to be referred back to the documentation that clearly didn’t help.

Anyway I have rebuilt Caddy (v2.8.4) with the following command:
~/go/bin/xcaddy build --with github.com/dunglas/mercure --with github.com/dunglas/vulcain --with github.com/caddy-dns/cloudflare --with github.com/caddyserver/replace-response --with github.com/greenpau/caddy-security --with github.com/hslatman/caddy-crowdsec-bouncer --with github.com/ggicci/caddy-jwt --with github.com/WingLim/caddy-webhook --with github.com/greenpau/caddy-git --with github.com/shift72/caddy-geo-ip --with github.com/ueffel/caddy-basic-auth-filter --with github.com/pberkel/caddy-storage-redis --with github.com/mholt/caddy-hitcounter --with github.com/teodorescuserban/caddy-cookieflag --with github.com/ltgcgo/floaty --with github.com/baldinof/caddy-supervisor

This combination of plugins is working through the official downloader as well. In fact imageproxy also worked there even though it’s not in this command.

I have attempted to transform my headers import file into a snippet as Francis suggested:

{
        debug
        storage file_system /var/www/html
        log default {
                output file /var/log/caddy/caddy.log {
                        roll_keep_for 168h
                }
                include
                level debug
                format json
        }
        http_port 8888
        https_port 4444
        email [redacted]
        acme_dns cloudflare [redacted]
        (headers) {
                header {
                        X-Frame-Options "DENY"
                        X-Content-Type-Options "nosniff"
                        Referrer-Policy "strict-origin-when-cross-origin"
                        Content-Security-Policy "default-src 'self'; style-src 'self' 'unsafe-inline'; object-src: none;"
}
        }

}

Error: adapting config using caddyfile: /etc/caddy/Caddyfile:16: unrecognized global option: (headers)

I’ve tried placing that inside a site block too and that also didn’t work.

Error: adapting config using caddyfile: /etc/caddy/Caddyfile:29: unrecognized directive: (headers)

11

Snippets don’t go within the global options block. It goes outside of it (after it). The structure diagram shows this.

Then later, you use import to copy the directives from the snippet into a site.

This is wrong – you’re basically telling Caddy “store your TLS certs and private keys in /var/www/html”. That’s not what you want. That’s extremely dangerous if you’re also possibly serving sites from there. Let Caddy store TLS certs in its default location unless you have a good reason to change it (e.g. to make it easier to sync if you run Caddy in a cluster – I assume you’re not).

You’re probably looking for the root directive (directives go in site blocks) to define your webroot for file_server etc.

2 Likes
12

I accidentally figured out that my storage directive was wrong when I set up the redis plugin. I had completely misunderstood the situation when it was complaining about not having write permissions to /usr/share/httpd.

I’ve made a lot of progress since that state. I got all the global directives working. I also realized that being behind Cloudflare, I can’t actually use custom ports because Cloudflare only proxies ports 80 and 443 unless you’re on the enterprise plan.

First, a quick question. Does Caddy support CSP nonces? And if it does, does it accept the same format as Apache: 'nonce-%{UNIQUE_ID}e' or do I need to modify this in the headers and throughout my code?

My main issue I’m having now is with php-pfm/fast-cgi (since I’ve been unable to install FrankenPHP). I’m not sure if it’s Caddy itself or a security header, but the mime type is getting stripped from my resources, making browsers block them. And it’s only happening with Caddy.
Refused to apply style from 'https://mydomain.com/assets/css/style.css' because its MIME type ('') is not a supported stylesheet MIME type, and strict MIME checking is enabled.
Refused to execute script from 'https://sub.mydomain.com/js/vendor/jquery/jquery.min.js?v=5.2.1-1.el9' because its MIME type ('') is not executable, and strict MIME type checking is enabled.
Or, the strangest one of all:
Refused to apply style from 'https://sub2.mydomain.com/assets/styles.css?61e062690f' because its MIME type ('text/html') is not a supported stylesheet MIME type, and strict MIME checking is enabled.

I already have /etc/mime.types on my system.

I’ve attempted to bypass php-fpm for static content and that has worked partially, like for images, but it’s not working for CSS and JS. It’s also very inefficient to set every directory of every resource type for every site block, especially for websites that use npm modules that have their own CSS and JS in deeply nested /vendor/ directories.

Wouldn’t it be better to do the opposite? Only route *.php through php-fpm? Especially since these are usually in the document root or closeby. Much closer to set the paths to than assets.

Anyway here’s my Caddyfile:

{
        debug
        storage redis {
                address 127.0.0.1:6379
                compression true
        }
        log default {
                output file /var/log/caddy/caddy.log {
                        roll_keep_for 168h
                }
                include
                level debug
                format json
        }
        email [redacted]
        acme_dns cloudflare [redacted]
        order floaty before header
}
(headers) {
        header {
                X-Frame-Options "DENY"
                X-Content-Type-Options "nosniff"
                Referrer-Policy "strict-origin-when-cross-origin"
                Content-Security-Policy "default-src 'self'; style-src 'self' 'unsafe-inline'; object-src 'none>
                Permissions-Policy "interest-cohort=(),gyroscope=(self),fullscreen=(self)"
                Access-Control-Allow-Origin "*"
                Referrer-Policy "no-referrer"
                X-Permitted-Cross-Domain-Policies "none"
                server "mydomain"
        }
}
(cloudflare) {
        tls {
                protocols tls1.3
                key_type p256
                dns cloudflare [redacted]
                resolvers 1.1.1.1
                #strict_sni_host on
        }
}

sub1.mydomain.com {
        import cloudflare
        import headers

        root * /var/www/html/sub1
        file_server
        handle /assets/* {
                file_server
        }
        php_fastcgi unix//run/php-fpm/www.sock
        header {
                Access-Control-Allow-Methods "GET, POST, OPTIONS"
                Access-Control-Max-Age "1000"
                Access-Control-Allow-Headers "x-requested-with, Content-Type, origin, authorization, accept, client-security-token"
        }
        }
        rewrite ^/(?!index\.html)(.*)$ /index.html {
                if {file} !~ -f {
                        if {file} !~ -d {
                                rewrite /index.html
                        }
                }
        }
}

sub2.mydomain.com {
        import cloudflare
        import headers

        root * /var/www/html/sub2
        file_server
        handle /assets/* {
                file_server
        }
        php_fastcgi unix//run/php-fpm/www.sock
}

sub3.mydomain.com {
        import cloudflare
        import headers

        root * /var/www/html/sub3
        file_server
}

sub4.mydomain.com {
        import cloudflare
        import headers

        root * /var/www/html/sub4
        file_server
        handle /assets/* {
                file_server
        }
        php_fastcgi unix//run/php-fpm/www.sock

}

sub5.mydomain.com {
        import cloudflare
        import headers
        @static path /js/vendor/*/* /img/* /svg/* /themes/alfred/css/*.css /*.png
        handle @static {
                root * /usr/share/sub5
                file_server
        }
        root * /usr/share/sub5
        php_fastcgi unix//run/php-fpm/www.sock
        header {
                Content-Security-Policy "default-src 'self'; style-src 'self' 'unsafe-inline'; object-src 'none';
        }

sub6.mydomain.com {
        import cloudflare
        import headers

        reverse_proxy localhost:1234
}
sub7.mydomain.com {
        import cloudflare
        import headers
        @static path /themes/editorial/assets/js/* /themes/editorial/assets/img/* /themes/editorial/assets/svg/* /themes/editorial/assets/*.css /user/themes/editorial/>
        handle @static {
                root * /var/www/html/sub7
                file_server
        }
        root * /var/www/html/sub7
        file_server
        php_fastcgi unix//run/php-fpm/www.sock
        header {
                Content-Security-Policy "default-src 'self'; font-src 'self' data: https:; img-src 'self' data:  blob:;
        }
#.htaccess
        rewrite / {
                if {path} =~ ({|}}|{%|%}) {
                        return 403
                }
                if {query} =~ ({|}}|{%25|%25}) {
                        return 403
                }
                if {query} =~ base64_encode[^(]*\([^)]*\) {
                        return 403
                }
                if {query} =~ (<|%3C)([^s]*s)+cript.*(>|%3E) {
                        return 403
                }
                if {query} =~ GLOBALS(=|\[|\%[0-9A-Z]{0,2}) {
                        return 403
                }
                if {query} =~ _REQUEST(=|\[|\%[0-9A-Z]{0,2}) {
                        return 403
                }
        }

        # Block direct access to specific folders and files
        rewrite ^(\.git|cache|bin|logs|backup|webserver-configs|tests)/(.*) error {
                return 403
        }
        rewrite ^(system|vendor)/(.*)\.(txt|xml|md|html|json|yaml|yml|php|pl|py|cgi|twig|sh|bat)$ error {
                return 403
        }
        rewrite ^(user)/(.*)\.(txt|md|json|yaml|yml|php|pl|py|cgi|twig|sh|bat)$ error {
                return 403
        }
        rewrite \.md$ error {
                return 403
        }
        rewrite (^|/)\.(?!well-known) - {
                return 403
        }
        rewrite ^(LICENSE\.txt|composer\.lock|composer\.json|\.htaccess)$ error {
                return 403
        }
}

#root domain
mydomain {
        import cloudflare
        import headers
        @static path /assets/js/* /assets/img/* /assets/svg/* /assets/*.css /assets/fonts/*
        handle @static {
                root * /var/www/html/root
                file_server
        }
        root * /var/www/html/root
        php_fastcgi unix//run/php-fpm/www.sock
        header {
                Content-Security-Policy "default-src 'self'; font-src 'self' data: https:; img-src 'self';
        }
}

I haven’t fully tested this out, especially the rewrite rules, but all the websites are loading correctly now with HTTPS. It’s hard to test without CSS and JS.

13

I just want to clarify again that it’s not a directive, storage is a global option. There’s no such thing as “global directives”.

We use different terminology for global options vs directives, because they specifically go in different places in the config, and cannot be mixed.

I think it’s useful to use the same terminology as the docs when talking about it so that we know we’re on the same page.

Good that you figured out a solution here :+1:

CSP is an application-layer concern. Caddy itself doesn’t have anything specific for this. This is something a plugin could probably do though. Since you’re running a PHP app it seems, you’d be better off handling the CSP headers in your PHP app itself. If you’re using a modern framework like Laravel or Symfony, they make it easy to configure with middlewares.

The recommendation is to use Docker if you want to use FrankenPHP, it makes it significantly easier to build a working image that way. See FrankenPHP: the modern PHP app server. Remember that FrankenPHP essentially just a custom build of Caddy, but it uses CGO to compile against PHP which is written in C, so it’s more complicated than other plugins which are pure-Go so they don’t require any extra system setup.

Are you sure that Caddy is actually serving the file? Try that path with curl -v, what do you get as output? Do you get the actual CSS file’s contents, or is it an empty response?

If it’s empty then it’s a config problem where you’re not routing that request correctly to serve the file. You might have an overzealous rewrite which changes too many requests to the wrong path, or you might be missing a rewrite to get the request to point to the file on disk.

Where is the file located on disk exactly? You need to figure out that path and how Caddy will assemble the path by starting with the root and appending the request path.

That is what Caddy does automatically, actually. The php_fastcgi directive does that. See the docs, there’s an in-depth explanation of how it works (including the rewrites etc that it performs). That directive is a shortcut (syntax sugar) for a longer actual config.

This doesn’t seem right – did you copy this from nano or vi? I see a > at the end there, implying the line continues. Try to avoid that, you can use cat to print out your config to your terminal without any lines being cut off.

I recommend removing these. You don’t need it, Caddy’s defaults are modern and secure. You should only configure these if you have a very specific need. The issue with overriding defaults like these is that if Caddy later changes the defaults, then your config would become incompatible and it’s an extra thing you’d need to fix to upgrade to the latest version.

I don’t think this handle does anything useful. All it’s doing is "if the path starts with /assets/* then run the file_server. But you already have a file_server in the top-level, so it has no use cause it’ll hit file_server anyway (for non-PHP files because you have php_fastcgi).

This is totally invalid syntax. How did you come up with this? There’s no if in Caddy at all, you use request matchers for applying conditions to directives.

Also, regexp matchers cannot be inlined, you need to define a named matcher with the path_regexp matcher, which you can then apply to a directive.

This also isn’t doing anything useful because you already set root and file_server in the top-level of your site.

Similarly, all this stuff is invalid syntax as well. That’s not now Caddy’s rewrite works. There’s no such thing as return in Caddy. Regexp matchers can’t be inlined. It looks kinda like nginx config to me, not sure where you got that but it’s definitely wrong and not applicable to Caddy.

1 Like
14

I’ll remove the TLS options and try to remember the nomenclature.

I’m not running a PHP app on the website that uses nonces. It’s a collection of hand-written PHP pages, which is the only reason I implemented nonces for it since I have complete control over it. But Apache always handled the nonces for me. I know nginx also handles it. I take it that’s not possible with Caddy?

I’m really not a fan of docker. I find it bulky, complex, and difficult to configure. I prefer to install things manually. I’ll look more into that possibility.

Yes, the asset files are being served, otherwise they wouldn’t be getting blocked by the browser. You were right. They’re actually not being served at all. Curl returns 404.

The rewrite rules are only on two site blocks, but this issue is happening across all PHP site blocks including the ones that don’t have any rewrite rules.

The only reason I added all of these custom configs (handle, etc) is because php_fastcgi did not handle this properly and lead to the blocking of these resources. Without them, even my images don’t load. So they’re clearly doing something. But I agree, this shouldn’t be required and doesn’t make sense.

The regex is my attempt to convert .htaccess rules into a syntax Caddy would understand. I guess I’ll have to look further into that as well. It’s only for security anyway so I’ve removed it for now.

15

On Linux, it’s not bulky at all. It’s very light-weight. It does have a bit of a learning curve, but it’s a huge advantage to be able to encapsulate everything quirky about building a program or shipping a program. Better for automation and portability, easier to move to a different server etc.

I’ve never heard of that functionality in those servers, and it’s the first I hear anyone ask about it for Caddy since I got involved with the project (~7 years ago). But like I said, that’s typically an application-level concern. You can probably do what you want in one PHP file, then require_once it into each of your PHP scripts to set the necessary headers.

Didn’t handle what properly? I’m not sure I follow.

I doubt it.

Since those are invalid syntax, I’d be surprised if Caddy even started with that config. Are you sure Caddy accepted your config? What’s in your Caddy logs? Watch your logs after each config change, it would tell you when there’s a problem with the syntax. You might still be running an old config that was valid earlier but your running config hasn’t changed due to syntax errors, therefore making it seem like certain config changes “fixed” or “didn’t fix” something.

1 Like
16

Caddy fails to start when I make a wrong change, so it’s definitely accepting my config lol I’m editing the same Caddyfile directly and restarting, to ensure new changes apply on every systemd restart. But that’s neither here nor there as I removed the rewrite rules and will deal with them later.

Like I said, php_fastcgi is having trouble with my assets. It’s loading the PHP, but it’s not loading CSS, JS, and images. Adding the handle directives above made images work, but not CSS or JS. Removing the handle directives breaks images again. CSP and JS remain broken.

And I found a few posts discussing CSP nonces on Caddy. Some on an old forum, this issue on github with an allegedly working implementation that was criminally closed due to inactivity, and a couple posts on this current forum.

17

php_fastcgi does not have file_server built in, you need to also have that in your site. See Common Caddyfile Patterns — Caddy Documentation

It would help if you show your current config at this point so that we’re on the same page.

My understanding according to CSP: script-src - HTTP | MDN is that it’s to deal with inline <script>? Why not just move it into .js files instead to avoid that issue? That seems like an issue that only exists if you use inline scripts.

1 Like
18

You’re absolutely right. That is in fact a better way to handle JavaScript.

The relevant parts of my Caddyfile haven’t changed, I only removed the handle directives (as per your suggestion) that were file_serving CSS, JS, and images and that resulted in the breaking of images and the continued breaking of CSS and JS.

This is my main website, for example. The root domain which has no rewrite rules or a backend PHP app/framework or anything fancy:

mydomain.com {
        import cloudflare
        import headers
#       @static path /assets/js/* /assets/img/* /assets/svg/* /assets/*.css /assets/fonts/*
#       handle @static {
#               root * /var/www/html/root
#               file_server
#       }
        root * /var/www/html/root
        php_fastcgi unix//run/php-fpm/www.sock
        header {
                Content-Security-Policy
        }
}

When I uncomment those lines, images load correctly. But other assets still don’t. According to what you said, php_fastcgi should only load .php files, right? Well that is what’s happening. Except the assets are not being served by plain file_serve. They aren’t being served at all.

19

Good :+1: that simplifies things.

Just add a file_server right below php_fastcgi. That’s all you need. Don’t need the handle, and don’t need to repeat root. See the Caddyfile Patterns docs link in my previous comment.

To further explain if you’re confused – Caddy’s HTTP handlers are a middleware pipeline. Handlers may terminate (i.e. produce a response then go back up the middleware chain) or they may pass through the request to the next handler.

In this case, php_fastcgi will test the request to see if the path is to a file that exists on disk, and if it is, and it’s not a .php file, then it’ll pass the request through to the next handler, which should usually be a file_server, which will serve your JS, CSS, images, etc, as normal.

1 Like
20

Thanks for all the help. I’ve been able to resolve all my problems with Caddy except for two.

  1. I can’t get FrankenPHP to work. I’ve successfully compiled it using their own base, and even tried their official binary from the releases on their github repo. In every case it fails to start saying it doesn’t recognize the frankenphp global option, which their documentation and even their homepage declares as a requirement to get it working.

  2. I can’t get monitoring alerts to work on a per-hostname basis. That’s necessary to know which website is throwing the alert.

I have it set as a global option since some of my websites don’t allow custom URIs.

{
       servers {
               metrics
       }
#       frankenphp
#       order php_server before file_server
        debug
        storage redis {
                address 127.0.0.1:6379
                compression true
        }
        log default {
                output file /var/log/caddy/caddy.log {
                        roll_keep_for 168h
                }
                include
                level debug
                format json
        }
        email [email]
        acme_dns cloudflare [token]
        order floaty before header
}

prometheus.yml

---
global:
  scrape_interval: 15s
  query_log_file: prometh.log
scrape_configs:
  - job_name: 'caddy-direct'
    static_configs:
    - targets: ['localhost:2019']
      labels:
        job: 'caddy'
        __param_hostnames: '*'
    relabel_configs:
    - source_labels: ['__param_hostnames']
      target_label: '__param_hostnames'
      regex: '(.*)'
      replacement: '$1'
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - localhost:9093
rule_files:
  - prometheus_rules.yml

prometheus_rules.yml

groups:
  - name: caddy_rules 
    rules:
      - record: caddy_http_requests_total_with_hostname
        expr: label_replace(caddy_http_requests_total, "extracted_hostname", "$1", "X-Hostname", "(.*)")

      - record: caddy_http_request_duration_seconds_sum_with_hostname
        expr: label_replace(caddy_http_request_duration_seconds_sum, "extracted_hostname", "$1", "X-Hostname", "(.*)")
  - name: caddy_alerts
    rules:
      - alert: PageVisit (Test)
        expr: sum(increase(caddy_http_requests_total_with_hostname[1s])) by (extracted_hostname) > 0
        for: 1s
        labels:
          severity: info
        annotations:
          summary: Page visit on {{ $labels.extracted_hostname }}
          description: A page visit was detected on {{ $labels.extracted_hostname }}.
      - alert: HighPageLoadTime
        expr: caddy_http_request_duration_seconds_sum_with_hostname > 0.1
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: High page load time on {{ $labels.extracted_hostname }}
          description: Page load time exceeded 1s on {{ $labels.extracted_hostname }}.
      - alert: HighCPUUsage
        expr: (sum by (instance) (rate(node_cpu_seconds_total{mode!="idle"}[1m]))) / sum by (instance) (node_cpu_seconds_total{mode="system"}) > 0.8
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: High CPU usage on {{ $labels.instance }}
          description: CPU usage is above 80% for 5 minutes on {{ $labels.instance }}.

alertmanager.yml

route:
  receiver: telegram
receivers:
  - name: telegram
    telegram_configs:
      - bot_token: [redacted]
        chat_id: [redacted]
        message: '{{ template "telegram.message" . }}'
templates:
  - default.tmpl

I’ve tried many variations of prometheus.yml trying to pick up the hostnames in many different ways with no luck.