The future of data and analytics

From well-connected computers, social networks, mobile shoppers and the digitization of our lives to the ability to put sensors into and onto just about anything, we have a tidal wave of data. Mobile connectivity itself is driving a significant change in how we think about data. When a customer “showrooms” – uses their phone in a retail store to compare prices online or uses their PC to purchase an item for after-work pickup at a local retailer – the lines between online and offline start to blur. And therein lie both challenges and opportunities.

As we saw in the July/August “Executive Edge” column from Piyanka Jain on predictive analytics, predicting the future relies on understanding the past. But there are times to break from it. Albert Einstein said, “We can’t solve problems by using the same kind of thinking we used when we created them,” and taming the data deluge requires some creative thinking. For inspiration we turn to computer scientist Alan Kay, who said, “The best way to predict the future is to invent it.”

eBay’s Marketplaces team (think eBay.com) has many tens of petabytes under management, with a constant stream of queries, tools and people that process and derive insights from the data. While job one is to keep up with the volume, variety and velocity (the “3 Vs”) of incoming data, treading water doesn’t make for a very competitive business. In order to provide as much consumable intelligence to as many as possible, we’ve set upon a path to build and operate a data and analytics ecosystem that addresses these futures:

1. The future will happen in parallel. This isn’t surprising, or even futuristic. Intel stopped making 80×86 processors in 2006, and even today’s smartphones have multiple processing cores. Yet for too many years, and even in some organizations today, when you ran out of memory, storage, processing or I/O capacity, you bought a bigger machine and moved your workload. Several advances mean we’ll start to scale out instead of always having to scale up. A small amount of capacity is cheap. Tie a lot of small boxes together and you have an enormous amount of capacity.

The open source Hadoop environment makes it possible to harness that capacity in a coordinated way. Rather than move the data to the processor, we avoid moving data, which avoids the biggest expense. Instead systems like Hadoop move the processing to the data. With hundreds or even thousands of processors working simultaneously, we’re now able to do things we weren’t able to do before.

2. The future will be multi-platform. The Enterprise Data Warehouse (EDW) is no longer the canonical source of truth for everything, because we’re analyzing data that doesn’t belong in a structured, relational system. We have semi-structured data, like clickstreams, and data that appears to be unstructured, such as text, images or competitor’s Web pages. Different data sets and different access patterns demand different systems.

At eBay we have three major data systems: our EDW for transactional data, an enhanced SQL-like system for transactional + behavioral data, and Hadoop for everything. While this provides us with ultimate flexibility, it’s not without its costs – not only for data replication but governance as well. Now metadata, data lineage and high-speed data transfer between the systems becomes much more important. Is it worth it? Absolutely, because today, no one system can handle the entire set of analytical needs.

3. The future will have machines that learn. We’re seeing more machine learning lately. This is due in part to the sophistication of the toolkits that are emerging, but it’s also an acknowledgement that we have larger haystacks and smaller needles. We need machines to help sift through the data and help decide what’s actually important. In a widely cited study, Microsoft researchers Michele Banko and Eric Brill (now head of eBay Research Labs) showed how five different natural language algorithms improved dramatically as they were fed more data. The lesson? In some cases, more data can trump new algorithms. That doesn’t spell the end of the researcher, but it does show that improvements can come from multiple lines of inquiry.

4. The future will be self-serve. As data becomes integral to even the most mundane decisions, it’s important to allow the people with the questions to get at the answers themselves. While this has practical considerations – lower costs, faster time to insights – it has an unexpected benefit: innovation. These new tools allow direct manipulation of data without having to translate between the person with he question and an analyst.

Anyone at eBay can create his or her own analytics sandbox with one click. These “virtual data marts” allow rapid prototyping outside the main production environment, without requiring copies of data or rogue machines running under a desk. Another popular tool is Tableau Desktop, which makes interactive visualization within reach of many who would shy away from Microsoft Excel.

5. The future will be collaborative. Good analysis is a terrible thing to waste. What if you could surf the shoulder of a master analyst, or even your own analysis from a year ago, and leverage that for a current task? What if you could find others who had worked with similar data or business problems? At eBay we built what we call the DataHub – a one-stop shop for all things data. It is a portal for publishing and finding data and analytics, but also has social-networking features such as friends/followers, groups and discussions. I can publish my Tableau or Microstrategy-based visualization or dashboard, and others can copy, modify and comment on it. No more Excel/Powerpoint in e-mail!

6. The future will be live. Can you imagine trading stock solely by reading yesterday’s closing price in the morning paper? Many business decisions are based on data even older than next-day, but organizations must be more agile. For a business like eBay, up-to-date means before the next click. Yes, some important data isn’t updated in real-time, but the business shouldn’t be held back by the slowest data feed.

Yesterday’s world ran on monthly or weekly sales reports, or even daily flash reports. But these analyses are a lens on the past. Running the business by examining the past is like driving a car by looking in the rear-view mirror. New tools such as memory-based data systems, Complex Event Processing and streaming query systems are getting mature enough to change business.

So that’s a quick half-dozen of the “futures” we’re working on today. All are in various states of evolution, but it’s clear that they will all play critical roles as we evolve the state of data and analytics. It’s an exciting time.