Google search console cant crawl my website: 301 Moved Permanently

Help
1

When I user google search console to crawl and render my website i get this response,

HTTP/1.1 301 Moved Permanently
Connection: close
Location: https://mywebsite.com/
Server: Caddy
Date: Thu, 12 Jan 2017 14:51:25 GMT
Content-Length: 57
Content-Type: text/html; charset=utf-8

Moved Permanently.

When not using google search console caddy responds as expected.

my Caddy file is
paperbank.net {
proxy / localhost:3000 {
header_upstream Host {host}
header_upstream X-Forwaded-Proto {scheme}
}
}

2

It’s redirecting to HTTPS. If you want Google to crawl the HTTP version of your site (strongly NOT recommended), disable automatic HTTPS.

3

@matt Thank you so much for replying me. So quick question is this related to caddy or google just can’t crawl https websites?.

4

Of course Google crawls HTTPS sites. I don’t know how to help with Google console, sorry. It’s not a Caddy issue, that’s for sure.

1 Like
5

@upatricck you will need to add the HTTPS version of your site to your list of properties at your search console, as Google will not crawl HTTPS for a HTTP property.

It’s common practice to add both the HTTP and HTTPS version of a property to the webmaster tools anyway.

2 Likes
6

What fixed my issue was updating my caddy file to include all tls version allowed.

mywebsite.net, www.mywebsite.net {

proxy / localhost:3000 {
    header_upstream Host {host} 
    header_upstream X-Forwaded-Proto {scheme}
    
}

tls {
    protocols tls1.0 tls1.2
}

}

1 Like
7

This… shouldn’t logically have been the answer to your above problem. Google’s crawlers used for webmaster tools support TLS1.0, TLS1.1, and TLS 1.2, so expanding Caddy’s support to vulnerable protocols should have had no effect. Not to mention that if it was a TLS negotiation issue, you wouldn’t be looking at a 301 response.

If you were getting redirected to the HTTPS version of the site, it’s almost certainly because Caddy issued a redirect from the HTTP version of the site, or some other redirect you’ve specified in your Caddyfile. This heavily indicates that a property should be added for the literal https://mywebsite.com/, “https” included, rather than the HTTP version of the same.