Directing bots to a reverse proxy

[DaveAppleton]

1. Caddy version (caddy version):

2.4.1

2. How I run Caddy:

a. System environment:

Debian 10.0

b. Command:

As per install docs

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo apt-key add -
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

c. Service/unit/compose file:

Paste full file contents here.
Make sure backticks stay on their own lines,
and the post looks nice in the preview pane.

d. My complete Caddyfile or JSON config:

preview.minty.art {
   header User-Agent *bot* {
      reverse_proxy * localhost:9009
   }

   header User-Agent *Bot* {
      reverse_proxy * localhost:9009
   }

   header User-Agent *Python* {
      reverse_proxy * localhost:9009
   }

   root * /var/www
   file_server

   log {
      output file /var/log/preview.log
   }
}

3. The problem I’m having:

The App is a react app, hence og tags cannot be created,

I am trying to figure out how to redirect bots to use a reverse proxy so that I can generate previews

I need to check the “User-Agent” header and given anything with “bot” or “Bot” or “Python” send it to a reverse proxy instead…

(this is on a demo server without the react app - I have figured out how to get the react app working
using try_files {path} /index.html

4. Error messages and/or full log output:

run: adapting config using caddyfile: parsing caddyfile tokens for ‘header’: /etc/caddy/Caddyfile:12 - Error during parsing: cannot specify headers in both arguments and block

5. What I already tried:

All the examples about redirecting bots seem to be old (Caddy 1.x) so it isn’t really clear where to start here…

6. Links to relevant resources:

[Mohammed90 (Mohammed Al Sahaf)]

The issue is due to a misconception about the relationship between matchers and directives. Directives that support matchers take matchers, but not the other way around. I recommend you go over the Caddyfile Concepts — Caddy Documentation page. You will need to use named matchers.

Secondly, the header matcher is case sensitive. To go around this aspect of the header matcher, you need to use the alternative header_regexp matcher. The regular expression engine is Go’s regexp engine whose syntax is detailed here: syntax package - regexp/syntax - pkg.go.dev. I believe this is the matcher you need:

@bot {
    header_regexp User-Agent (?i)(bot|python)
}

Since you want either one of the routing decisions to take place, then you need to wrap them in handle directive. Thus your final Caddyfile is:

preview.minty.art {
	@bot header_regexp User-Agent (?i)(bot|python)

	handle @bot {
		reverse_proxy http://192.168.1.1
	}

	handle {
		root * /Users/mohammed/projects
		file_server browse
	}
	log {
		output file /var/log/preview.log
	}
}

[DaveAppleton]

@Mohammed90

handle @bot {

Thank you so very much. By doing this I believe that you have solved a big issue for galleries written in react