Nobody wants to be in the news with headlines like this, but it happens. I’d like to share an interesting experience I had when meeting on site with an anonymous customer. I’ve since noticed this same problem a few times, concerning Email Marketing, Conversion impact, nervous operators, CDNs and what happens if there is no communication between the Marketing teams and IT Operations.
It was back in January when I arrived at the operations center of the customer and in state of the usual warm welcome the head of operations – let’s call him Hugo- just said:”I am glad you are here! I think we have a bot/denial of service attack from Asia currently going on! Look at this dashboard!”.
Is it a bot/denial of service attack or a “successful” marketing effort that cause this increase in traffic and degradation in Response Time?
Easy enough to spot there is 4-times more traffic from Asia coming in than usual. But is it really a bot attack?
Me: “Are you really charting here Real User Monitoring visits?”
Hugo: ”Yes it is. Why are you asking?”
Hugo: ”No I am not! But let me ask our CMO”
Hugo on the phone: ”Hi this is Hugo. We are seeing some unusal high traffic from Asia. Is there something going on from your side that could explain this?”
Silence for a couple of seconds
Hugo: ”What the @#!*^” (according to the Netiquettes I am not allowed to repeat that part)
After hanging up, Hugo explained to me what the CMO told him on the phone and he unvealed that they have a massive email campaign scheduled for this week with 3 000 000 emails being sent each day for 3 days in a row. Marketing did not tell anyone in IT about their plans and had no idea such a massive traffic peak could bring down data centers around the world.
For me 2 interesting questions popped up:
- How did the data center survive this unexpected load?
- Was this email blast successful?
How did the data center survive this unexpected load?
This is a simple answer than expected. Their CDN (Akamai) took care of ~90% of all the requests and the number of requests hitting the datacenter was gladely not enough to bring it down the datacenter but had after reaching a certain limit an impact on the response time for all users around the world.
Was this email blast successful?
Looking at the below dashboard showing the number of visitors, bounce rate and conversion rate. We have to clearly say “NO” unless more traffic was the criterion. Bounce rate went up. Conversion rate went down and revenue (was not allowed to take a screenshot of this with me) did not change at all compared to the previous week.
After the email campaign was over there was retrospective of the “campaign” and during this meeting Hugo also presented the bill from the CDN showing the extra costs they had to pay for the additional traffic because of not booking a larger package ahead of the promotion which came on top of the normal marketing campaign costs.
If you are planning your next marketing event, newsletter promotion, etc. do not forget to include your IT operators. If you miss the chance to sync with them ahead of time best case they will only swear at you. Worst case scenario, you could bring down your datacenter and anger prospects with a broken promotional effort. On top of that, IT might bill you for the extra CDN costs!