Singing the Silicon Valley Blues

A few weeks ago, I was out at the ballpark with my friend “Gus” when he told me about his work at a local technology start-up. “We have a platform that can quickly analyze huge datasets and locate hidden patterns,” he explained.

Gus had my attention almost at once, because finding patterns in huge datasets is an increasingly large part of the business analytics landscape [1]. Part of Gus’ job with this firm (let’s call them Beta Company, or “BC” for short) was to find applications for their technology, and as it happened I had just started a consulting project that featured a large-data set. Since I am an inveterate slacker, I’m always looking for ways to make my life as a data analyst easier.

A few days later I found myself in conference room with a BC product manager, a laptop and an overhead projector – and that’s when my head began to hurt.

My immediate reaction: the BC technology was really cool. I had to sign a non-disclosure agreement before the meeting, so I can’t say too much. But there was interesting statistical logic underneath the innovative data structures, which enabled huge datasets to be stored and presented in novel ways. The core technology was fast, scalable and protected by patents. My Inner Geek was drooling over the powerful potential of BC’s technology.

At the very same time, my intuition told me to be very skeptical about BCs prospects for commercial success, and the Resident Cynic silently began with the usual questions:

• What industry is BC targeting? What size does a company need to be in order to justify buying and using it? What are the expectations/requirements for this size company in this industry? What entrenched vertical competitors is BC going to be taking on? How strong are they? How will they respond?
• Who in the customer’s organization will use this software? How much of what these prospective users “need” is possible in BC’s application as it exists today? What are the gotta-haves? How feasible and how expensive will it be for BC to add them? How much time, money and patience will it take?
• Who buys it? How much will it be worth to them, and how is this value demonstrated and realized? What does it cost them financially? How much internal capital do they need to spend to get others to adopt a new solution like this one? Where is corporate technology spending going in this difficult economy?

These questions are not easy ones and virtually impossible to determine from within the test tube of a technology start-up’s office. This stuff usually evolves through a frantic marathon of sales meetings and design documents, pleasant and not-so-pleasant demands from customer/partners, late nights in product feature meetings, heated arguments, exhausted software engineers, moments where the company hangs in the balance, days where the founders wonder why they ever started this thing in the first place, euphoric celebrations of victories, and sad late nights at a hotel bar far from home lamenting defeats and delays.

Watching the BC demo, I began to feel an immense sense of fatigue just thinking about all this. The quick and cheerful answers to my questions – confident responses like “yes, we’re know we’re going to need to build that” and “sure, we could definitely do that” – all seemed to confirm the length and steepness of the road that my friend Gus and his BC friends had embarked on.

In Silicon Valley, we celebrate the winners. Almost everyone who knows me has heard the (true) story of my grad school softball team: “Eleven guys on the team, and 10 of us managed to finish school...the other guy dropped out...(pregnant pause)...and started Yahoo!” Similarly, in O.R. we see our work as powerful and heroic, and our successes as inevitable. We rightly honor our Edelman Award finalists, but we virtually never share stories about O.R. methods and projects that have failed, or merely meandered to an obscure ending.

 The reality is that most analytic projects’ and companies’ experiences are almost always less linear, more textured, harder to describe – and infinitely less predictable. The non-mathematical factors that help determine success are rarely discussed by our journals, our textbooks and our courses. And that is both a shame and a disservice.

Anyway, from the offices of BC I headed down the road to visit another company – let’s call them Already Disillusioned, or “AD” for short. I have served as a consultant and advisor to AD for several years. They too have a very powerful analytic engine that can store and process massive amounts of data with powerful statistical tool, and at this point they sell a business solution carefully crafted and targeted for large-scale customer service call centers. They are growing and have some marquee customers, including a few notable success stories – but along the way, AD has also burned through tens of millions of dollars of capital and narrowly escaped death more than once.

And still a long way to The Promised Land... 

Reference

1. In fact, the very same month I had received an e-mail about a conference dedicated to “Algorithms for Massive Modern Datasets” see www.stanford.edu/group/mmds/ for more details.