Cloud analytics works for me

When you have an ad hoc analytical project, internal IT dependency usually comes your way. Until a few years back, we had to accommodate and account for it in our delivery timelines. In today’s world, we can avoid this altogether by opting for public cloud analytics.

The Problem

Let’s assume that you are asked to analyze a significantly large data set and provide insights. Based on the data size, you determine that you’d require a clustered and distributed environment to be able to run the analysis within the pre-decided timeline. You ask your manager for help and he says to raise a request with IT. You raise the request in the ticketing system and check with IT. They answer that they don’t have it ready and will need to procure the system first. You comply and ask how long will the process take? The IT personnel responds, “Four weeks provided you get the budget approval from your department.” Your timelines have just run amok!

If the above scenario is true in your case, then the situation is similar to what I have heard from several other analysts in different organizations. While the IT department centrally manages all infrastructure, they mostly focus on normal operational efficiency goals. Project specific ad hoc requirements usually do not fall under the preview of IT supply and demand hence is looked upon as ad-hoc. This is the norm in every service industry today. The 30,000-foot question is: What do we do?

Cloud Analytics to the Rescue

The obvious answer comes in the form of using public cloud infrastructure for your analytical needs. Using public cloud practically eliminates the time lapse on infrastructure procurement. In addition, analytics on public cloud comes in three different flavors:

1. The black-box approach allows you to consume already trained models as services;
   
   
2. The white box approach allows you to use the public cloud infrastructure for data storage, deploy analytical tools for performing descriptive or predictive needs and a visualization platform for reporting insights; and
   
   
3. The grey box approach allows you to create an ensemble of models using statistical techniques such as decision trees and infer underlying rules for relationships.

In this case, while the actual implementation of statistical technique is black boxed, you still have control in determining the right path and relationships by varying the factors required for rules determination. All you need is to make a choice of determining which approach to take based on the problem definition.

Custom Analytics on Public Cloud

As part of an R&D engagement for one of our clients, we needed to perform sentiment analysis on tweets in order to determine the problem context, keywords, sentiments and topics. The main objective of this exercise was to determine service-related issues and resulting sentiment of customers. This being a social media analytics exercise, the solution was to incorporate aspects of real-time sentiment complex event processing along with data aggregation and historical descriptive analytics.

We determined that using a Twitter streaming service would be the best option. This would require constantly listening to the streaming service and would result in collecting and analyzing up to 2,000 tweets a minute.

Based on the infrastructure requirement, we looked into public cloud providers options and decided to go with Amazon Web Services. It took us just a couple of days (includes internal approval process) to get three large capacity nodes with Linux pre-installed. We wanted a clean system as we were building a custom natural language processing system to process the tweets.

It took us one week to deploy Apache Hadoop, Druid, R statistical tool, Zero MQ, Python and Java and two weeks to run through performance benchmark testing and an optimization exercise. In parallel, we also developed and deployed a custom HTML5 based visualization tool for insights.

All in all, the entire exercise took a month and a half from start to finish for the initial proof of concept. The NLP algorithm development journey is still underway and being constantly optimized for accuracy and performance.

What could have easily been a three months exercise if we had waited for internal infrastructure procurement, took only half the time to deliver. We also have both data ingestion and insights visualization on the public cloud along with backend processing components – not to mention, we did not make any depreciating investments, hence keeping the bottom line intact.

Conclusion

Cloud-based analytics or cloud analytics is an evolution in the making. We are seeing traditional BI vendors providing cloud-based SAAS-BI services on one hand, while the others are providing specific niche services in social media analytics using third-party sourced data. In between, we have big cloud infrastructure providers, providing both infrastructure and analytical platform. On the most crucial aspects of data security, most of them have ironclad service-level agreements and adhere to them. Based on our own experience and listening from other engagements, I can confidently say, “Cloud analytics works for me,” provided you know how to make it work for you.