Redirecting 50,000 URLs with redir

Help
1

I’m just set up Caddy and am planning to use it as a lean solution to 301 redirect 50,000 subdomains to directories on my server.

I have a few questions about my Caddyfile:

  1. How do I make redir a 301 instead of a 302-- is it simply “redir 301”?
  2. Do I really need to add separate www and non-www versions for each redirect, or is there a way to do this with a rewrite?
  3. Are there any best practices for doing this I should know about?
  4. Is there a way to redirect any other subdomains (wildcard) not covered in this file to my homepage?
  5. Is there a way to redirect all 404 errors to the homepage?

Note that I am forcing http since I have so many subdomains and am redirecting them to a https address anyway.

d. My complete Caddyfile or JSON config:

http://macfans.xx.yy {
	redir https://xx.yy/apple
}

http://www.macfans.xx.yy {
	redir https://xx.yy/apple
}

http://election2020.xx.yy {
	redir https://xx.yy/politics
}

http://www.election2020.xx.yy {
	redir https://xx.yy/politics
}

http://iphonerumors.xx.yy {
	redir https://xx.yy/apple
}

http://www.iphonerumors.xx.yy {
	redir https://xx.yy/iphone
}

I am grateful for any insight the community has in this. I came across Caddy after another sysadmin recommended it as a lean solution for processing a high volume of 301s.

2

Yep:

redir [<matcher>] <to> 301

You can have site blocks with more than one domain:

http://macfans.xx.yy, http://www.macfans.xx.yy {
	redir https://xx.yy/apple
}

Yeah:

http:// {
	redir https://example.com
}

The Caddyfile will sort them all based on the hostname, so that one should always end up checked last.

Hmm, depends on the source of the 404. Error handling is a bit tricky. If you’re proxying to another server, then right now, it’s not really viable to do from the Caddyfile, there’s a WIP pull request to implement Caddyfile support for it. But if you’re just serving a file_server, you can use handle_errors:

handle_errors {
	@404 expression {http.error.status_code} == 404
	redir @404 https://example.com
}

Another approach that can be taken for the redirects is to use the map directive, which is unfortunately not yet documented on the website (but should be sooner rather than later!)

It can let you define arbitrary mappings between a source variable and one or more destination variables. I haven’t played around with it much yet, so @matt might need to clarify if I explained it wrong, but I think it would look something like this for you:

# Define all the mappings you want (you could even load them from separate files with `import`)
map {host} {new-path} {
	macfans.xx.yy /apple
	www.macfans.xx.yy /apple
	election2020.xx.yy /politics
	www.election2020.xx.yy /politics
	
	# ... a bunch more

	# this one to catch the rest
	~\*.xx.yy /
}

# Do the redirect if there's a value for `new-path`, otherwise nothing happens
@shouldRedir vars_regexp new-path .*
redir @shouldRedir https://xx.yy{new-path}

# Do something here with the non-redirect requests
root * /srv
file_server

You can see how it works in the code, for now, until it’s properly documented:

1 Like
3

Yep, something like that. Here’s the docs, hot off the presses:

https://github.com/caddyserver/website/commit/9dd9309a9bc2321375495825b95cb8fb6eaac418

2 Likes
4

This is immensely helpful. It is great that you can use blocks to handle multiple addresses at once. Despite including http:// I am for some reason running into an SSL error.

I tried moving the root domain to the bottom of the file thinking this was forcing the https redirect. But no luck.

Here is my Caddy file:

http://jrsmotorsport.pt.vu, http://www.jrsmotorsport.pt.vu,http://moreno.pt.vu, http://www.moreno.pt.vu { 
	redir https://pt.vu/carros 301 
    }
    pt.vu {
    root * /var/www/pt.vu/html/wordpress
    php_fastcgi unix//run/php/php7.4-fpm.sock
    file_server
    encode gzip zstd
    }

I updated my Caddy file and ran the following:

sudo caddy reload --config /etc/caddy/Caddyfile"
sudo systemctl restart caddy

The root domain works flawlessly. However I am still being force to https on the redirected subdomains.

5

Please use ``` on the lines before and after your config to use code formatting on the forums!

I think you need a space after the comma in your list of domains.

Also it’s not necessary to restart Caddy, reloading is enough if Caddy was already running without error. Reloads are graceful, restarts involve downtime.

Please post your full Caddy logs, it’s unclear what’s wrong without more specific detail. Also make the request with curl -v to show the behaviour you’re seeing.

2 Likes
6

I edited my post to properly inline the code and will be sure to post a log next time. I figured out the solution-- here’s what was causing the problem. Spaces are needed after each comma as you mention. But I still encountered the SSL protocol error.

Appending :80 to each domain fixed the problem. ie:

http://macfans.xx.yy:80, http://www.macfans.xx.yy:80 {
	redir https://xx.yy/apple 301
}

For whatever reason Caddy would not force http unless both “http://” and “:80” were present.

Thanks for noticing the spaces and all of your other help as well.

I now have all my URLs happily redirected :slight_smile:

7

Can you compare the output of caddy adapt --pretty before and after that change? I don’t think that makes sense. http:// implies port 80, unless you changed the http_port in the global options.

We need more detail here to show that there’s an actual problem, if any.

8

Sure! I tried the change on another server and did not have the same problem, so it must be tied to my local config.

I tested a subdomain without “:80” and the redirect still works. Perhaps it was a caching issue.

I ran caddy adapt --pretty and in both cases this was the output:

	"automatic_https": {
						"skip": [
test.xx.yy

I will report back if I run into the issue again and try to better reproduce what happened as well as my server configuration in case someone else has this issue in the future.

9

This topic was automatically closed after 30 days. New replies are no longer allowed.