In this guide, we will learn what crawl rate, crawl budget is and how we can optimise them for better SEO performance. Let’s begin by discussing what the crawl rate and budget are.
What Is Crawl Rate & Budget
Google defines crawl rate as the number of requests per second Googlebot makes to your website.
Crawl budget, on the other hand, is defined as the number of pages Googlebot crawls in a given timeframe.
As you already know that crawling is the entry to Google’s index and eventually ranking.
More crawling should convert to more visibility on search engines. But this does not happen always. That’s why we learn to optimise the crawl budget.
Google has only limited resources as there are billions of web pages on the internet. Therefore, Google assigns some crawl budget (the number of URLs it wants to or is willing to crawl) to your website. Given this, it becomes important that utilize your crawl budget to the maximum benefit.
Should You Worry About Crawl Budget?
If you have a website that is less than a few thousand pages then you don’t need to worry about the crawl budget as Google is able to handle that efficiently.
But if you are maintaining a website that has URLs of more than ten or twenty thousand (as is the case with most of the e-commerce websites) then you need to worry about the crawl budget.
So, what determines your crawl budget?
The first thing which impacts your crawl budget is the crawl health of your website. Bad crawl health means that Googlebot is receiving errors, server errors, large load times when requesting URLs from your website. Poor crawl health decreases your crawl budget.
The crawl limit setting in Google Search Console also impacts your crawl budget. In the search console, Google gives you the ability to set limits on the number of URLs Googlebot can crawl.
In most cases, you should not touch that setting and let Google decide your crawl budget based on historical data about your website.
The other thing which directly impacts your crawl budget is crawl demand. Crawl demand includes the popularity of the content and the freshness of the content.
If some content is more popular on your website then Google will crawl it more often and if there is content that is very old then it won’t be crawled more often as it has lost its freshness.
So, given these facts, what can you do to improve and optimise your crawl budget.
Crawl Budget Optimisation
1. Remove Server Errors (5XX)
To improve the crawl health, you need to watch for the server errors Googlebot might be facing while crawling your website. Logfile analysis will be very helpful to you here as it will show you all the response codes for all the URLs requested by Googlebot. Server errors are indicated by 5XX response codes.
2. Remove Broken Links & Redirects
From the log file, you can check the response codes of all the URLs requested by Googlebot. Remove all the broken URLs from the website. Also, remove all the redirects from your website. Redirects impact the crawl health negatively.
3. Interlink Pages
Keep consistent interlinking in your website. It helps bots crawl and index URLs efficiently. Also, it gives importance to the pages which are interlinked. Try interlinking in the footer as well as the menu.
We have already talked about the website information architecture and you can learn more about it there.
3. Canonicalise Duplicated Pages
If there is only a slight change or no change of content among some pages, then you should canonicalise them to the primary one. This helps Googlebot spend less time crawling duplicate and unimportant pages and focus more on primary and important ones.
4. Canoniclise Faceted Navigation Pages
This happens when you are on some product category page and you apply some filters to narrow down on what you are trying to look for.
Now you might notice that URLs also change with the filter although the information on the page is still the same which was before but slightly modified. This becomes a case of duplication and you should canonicalise such pages also as we did above.
5. Update Old Content
As already said, posts with outdated information are not crawled much frequently after a certain period. Updating old content with new information is also a good practice to improve your crawl budget.
6. Make Important Closer To Homepage
If you want to improve the crawl rate for an important page and try to bring that page closer to the homepage. This means that the page should not be more than three links away from the homepage. We suggest keeping important pages only one click away from the homepage.
These were some of the ways you can optimise your crawl budget and grow your search visibility. Remember again if you have less than a few thousand pages then you don’t need to worry about the crawl budget as Google should be able to crawl all your pages efficiently.
Crawl budget is specifically an important matter for an E-commerce website. Next, we will talk about HTML DOM and why you need to know that for efficient SEO progress.
Search Engine Code Team is comprised of SEO experts and strategists having more than 20 years of combined experience. We keep testing and delivering knowledge of SEO for the community of SEO.
Leave a Reply