I recently began research on Search Engine Optimization to further understand what we can learn from Performance Monitoring solutions. As an IT nerd, I started the investigation on the technical side of the house, speaking with operators and CTO’s about robots. Interestingly, all of them expressed the same concern: “They are an issue! We run more than half of our data center for them!”

At first I found this to be a bit difficult to believe, but then they shared real statistics to support their concerns.

BVA Auctions Real vs Search Engine Traffic

Alexander Pluim CTO of BVA Auctions

BVA Auctions - Search Engine 67% vs. Real User Traffic 33%
BVA Auctions – Search Engine 67% vs. Real User Traffic 33%

PrepSportswear Real vs Search Engine Traffic

Mike Jones Director of Technology and at Prep Sportswear

Prep Sportswear - Search Engine 60% vs Real User Traffic 40%
Prep Sportswear – Search Engine 60% vs Real User Traffic 40%

Have you ever checked the amount of robot traffic on your website? After I heard about it I checked out our own blog and, although the blog content doesn’t change that often, there is quite a bit of bot traffic — around 17%.

Dynatrace APMBlog – JavaScript executing Search Engine 17% vs Real User Traffic 83%
Dynatrace APMBlog – JavaScript executing Search Engine 17% vs Real User Traffic 83%

Are you experiencing the same surge in your data centers? If you want to share your “Search Engine vs Real User Traffic” chart feel free to reach out to me via Twitter, LinkedIn, and I will add it to this blog. To generate this data/chart sign up for Dynatrace Personal Free.

In my next blog I will illustrate how BVA Auctions addressed this challenge and reduced the traffic by 75% without harming its search rank. Also, here’s my Velocity Conference presentation about the topic on youtube.com and the corresponding slide deck.