Analysing a log file can help us discover some of the deep hidden insights as well as some critical issues with our website that might be hampering its SEO performance.
So, in this guide, we will see how we can analyse log files for optimal SEO growth. Let’s begin by talking about what is a log file.
What Is a Log File
A log file is like a ledger keeping track of all the visitors who have requested your website. This includes information about end clients and crawlers.
A log file is generated by the servers recording all the activity on the server. Every time a user requests your website, an entry is recorded in the log file with details about the performed request. A log file looks something like this:
This might be something very overwhelming to look at. But you need to know that each line in the file is an single entry and all the entries follow the same pattern.
So, if you know how to read a single entry then you know how to read a log file. Let’s look at a single entry and check what information is available to us.
How To Read A Log File
As already said, you need to know how to read a single entry only since all the entries follow same pattern. So an entry in the log file looks something like this:
188.8.131.52 - - [08/Dec/2017:04:54:20 -0400] "GET /contact/ HTTP/1.1" 200 11179 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
- 184.108.40.206 is the IP address (who)
- [08/Dec/2017:04:54:20 -0400] is the Timestamp (when)
- GET is the Method
- /contact/ is the Requested URL (what)
- 200 is the Status Code (result)
- 11179 is the Bytes Transferred (size)
- “-” is the Referrer URL (source) — it’s empty because this request was made by a crawler
- Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) is the User Agent (signature) — this is user agent of Googlebot (Desktop)
That’s it. This is how simple it is to read entries in a log file. Now you can export log file to excel and start analysing it.
Analysing The Log File
Filtering Google Bot Visits
Since the log file is composed of all entries, you need to filter out the entries made by the Google bot since that is important to us when it comes to SEO.
You can check which URLs are most visited by Google, what status codes is Google Bot getting and are the URLs being requested properly by the crawler.
Look For URLs Most Requested By Google
A high frequency of visits means high priority. The URLs receiving the most number of visits means that the URL is pretty important to Google.
Here you can check whether the high priority URL is the one that you want or is it the one that is unimportant to you for SEO. This analysis can help you figure out whether high priority URLs for you are indeed a high priority for Google or not.
Look For Status Codes
Imagine if there is a URL that Google bot is visiting most frequently but that URL gives 404 (broken) status to the crawler then you might want to do something about it.
Because as said earlier, high visits mean high priority and if a broken URL is still a high priority for Google then you don’t want to waste a crawl budget on that. Maybe you will decide to redirect that URL to some active URL which is important for SEO. This kind of analysis is also helpful.
Monitor Search Bot Behaviour
Regularly monitoring log files can also be helpful. It might give important insights regarding if the behaviour of the bot changed or not. This might help you answer some important questions like is the website still being visited frequently, has to frequently drop or increased, etc.
Categorise By File Types
You can also check what kind of files are frequently being visited like PDF, images, videos etc. This will help you manage your site architecture properly.
Some of the other things that your log file might help you with are:
- Checking the unnecessarily large size of pages being requested.
- Irrelevant visits to irrelevant pages.
- Crawler bot activity over a period of time.
- Checking if your crawl budget is being wasted on wrong pages.
How To Get The Log File
A log file resides on your server. You can ask your developer or IT team for the log file.
Tools To Analyse Log File With
Some of the tools that will make your log file analysis easier are Microsoft’s Excel, Power BI, Tableau, and Semrush’s log file analyser. You need to be well acquainted with these tools but we assume you already know Excel so your job should be easier. Besides this, Semrush’s log file analyser is also easy to use.
These were basics about how you can analyse your server’s log file for optimal SEO growth. It’s time to move to another chapter where we will learn how you can further optimise and manage your crawl rate and budget.
Search Engine Code Team is comprised of SEO experts and strategists having more than 20 years of combined experience. We keep testing and delivering knowledge of SEO for the community of SEO.