Crawl stats report is one of the most powerful tools available to SEO professionals. It holds important information and learning how to correctly interpret it can mean the difference between the success and failure of any website.
The basic rule is — you need to rank well to get organic traffic.
Your website could be the best in the world and maybe even hide a cure for cancer or a solution to end wars and hunger, but it is useless if no one can find it. That’s where Google crawls kick in.
What is Google Crawl?
In short, Google uses it to determine if your website and pages should rank and where they should be in the SERP. It goes through your website and collects information — crawl stats — that will be later considered when filed in Google’s “index.”
What is Google Index?
This is Google’s entire inventory of websites, the place where it drives information from whenever people hit the search button. So, if you have that cure for cancer or world hunger, you better make it easy to find.
What Is a Google Crawl Rate?
The term describes how many requests per second Googlebot makes to a website site when crawling it. Google takes care not to overload the crawled website – it will determine the appropriate speed, making sure it doesn’t clutter the bandwidth.
However, if you notice that the crawl rate is too high, you can limit it. In case you were wondering — you can’t increase the crawl rate.
What is the Crawl Stats Report?
After examining your website, Google will create a report. To access it, log in to your Google Search Console account and go to Settings > Crawl stats.
It will show the following information:
- Total crawl requests
- Total download size
- Average response time
- Host status
- Crawl responses
- File type
- Crawl purpose
- Googlebot type
From there, you can further examine issues and find a way to solve them.
Interesting Google Crawl Stats
Google devotes a lot of resources to probing the internet. And it made us believe that it is all-mighty and all-knowing. However, there are many misconceptions about its knowledge and reach.
For instance, it doesn’t index every page it finds. Why does it do that? Well, some pages are simply not worth the effort of analyzing them.
The following statistics will give you an insight into how Google works and how the crawl stats report is created.
1. Google Search Index contains hundreds of billions of web pages.
In terms of size, it amounts to more than 100,000,000 Gb. However, there is an enormous difference between the number of pages Google knows about and the number of pages it crawls and indexes.
That’s because a massive chunk of the internet is spam, and Google won’t waste resources on these pages. Namely, more than 25 Billion pages Google encounters each day are spammy.
2. Google uses more than 200 factors to rank websites in SERP.
Google’s algorithm is a complex thing. To satisfy the searcher’s intent in the best possible way, Google considers more than 200 factors.
How to Check Website Crawling
You can check if your website has been crawled by:
- Checking it manually
This is the easiest way. Go to Google Search bar, type site: website name. If the website is crawled and indexed, you will see all the indexed pages in the “About XY results” section.
If you want to check a specific URL, use the site: URL.
- Using the URL Inspection tool
Go to Google Search Console and use the URL Inspection Tool. Enter the desired URL, and Google will give a detailed report.
3. There are several types of Googlebots that create the crawl stats report, but the two most common are – desktop and mobile.
Depending on the type of your website and the content you are featuring, it may be crawled by imagebot, videobot, adbot, storebot, etc. However, most crawls will be done by desktop and mobile bots. They simulate and review the user experience on these devices. The Google Search Console crawl report shows you which robots have visited your website.
4. Google has two different web crawling strategies—for new and existing content.
Google has adaptable crawl strategies for new content and refreshing already published content. The process is automatic and even adjusts to the website type. For instance, Googlebot will crawl news websites more often than others.
5. There are two types of Google crawl errors: Site and URL errors.
Crawl errors appear when Googlebot can’t reach the desired location. Site error means it can’t find your website, while URL error signifies that one of your pages is unavailable. Even though the first one is much worse, you don’t want either of them appearing in the crawl report.
When Did Google Last Crawl My Site?
You can check when Google crawled your website in Google Search Console – by using the URL Inspection Tool. After logging in, enter the desired URL. Google will show the last date it was indexed, the status, and indexing or crawling errors it found during the crawl.
6. The average crawl frequency can be anywhere from 3 days to 4 weeks.
This is a really wide margin, and it doesn’t say much. The crawls are automatic and depend on many factors, it is very hard to say when the next one will hit a particular website.
However, you can speed up things by checking for indexing status, errors, and usability problems, regularly updating the website, getting backlinks, and working on your technical SEO. They are all potential crawl triggers.
Crawl Stats Report: Conclusion
The internet is a pretty big place, and it’s not getting any smaller. This means it’s easy to get lost and never get noticed. Since traffic equals earnings, you can’t afford not to be noticed. The best way to improve your Google rankings is to take this report seriously and correct errors you spot.
You might get an occasional visitor, maybe your mom and a couple of your friends, just there to show support. But that’s not really a viable business option. Unless your mom and your friends are loaded, in which case, can we be your friends?