Classic mistakes doom promising startup

About 10 days ago, I got word that a good friend (I’ll call him “Hiram”) had been laid off for the first time in his life. I was shocked.

I had spoken with him a few months earlier, just before he was hired at a startup that I’ll call Fast Company (“FC”). The company, he explained to me, had some powerful patent-pending algorithms to help identify which online offer should be presented on a given Web page to a given Web surfer. FC was targeting large national retail clients, and, thanks to a strong marketing and sales organization, it had won contracts with several impressive customers. “This is a great chance for me,” Hiram said at the time. “The company just raised $12 million in VC money, they’re growing really quickly, and they want me to come in as director of analytics. I think there’s a really interesting market opportunity here, a lot of upside.”

Hiram had been an undergraduate mathematics major and later completed an MBA (both degrees are from terrific schools). He also arrived at FC with years of experience in enabling companies to successfully deploy such as FC that sought to create a group to communicate the real-world needs from its client services team to the scientific research team and to ensure that the core technology was successfully deployed with the company’s customers. I had been pleased that he had found a job that fit him so well, and soon thereafter he accepted their offer to join the company.

A scant four months later, he had been laid off.

I saw Hiram a few hours after the layoffs had been announced (roughly 25 of the company’s 100 employees had been given the pink slip, including Hiram’s entire group), and surprisingly he seemed more relieved than distressed. “Dude, it was totally crazy,” he confided. “You cannot believe the problems that I lived through while I was in there …”

FC’s targeting technology promised significant increases in online sales volumes and revenues for its customers. Not surprisingly, most companies chose to examine this claim very carefully, typically by conducting a straightforward controlled experiment, known in the online marketing world as an “A/B Test,” in which some percentage of Web traffic received offers based on recommendations from FC’s analytic engine while the remaining visitors were treated as a control group. Whenever FC’s technology failed to one involved to find a way to make the numbers look good for the client. In the absence of either diagnostic tools to systematically analyze system performance or an earnest desire to understand the truth, this short-
term focus led to many hastily conceived and undocumented software changes. As Hiram ruefully recalled, “A lot of half-baked stuff got dropped into production.”

The story gets worse. Like many machine-learning platforms, FC’s recommendation engine had a number of configuration parameters, and these “knobs” had a big impact on the system’s performance. Yet, despite the fact that the core technology had been deployed for more than two years at several customer sites, there was no documentation of these parameters, nor were there any internal guidelines on how to set them in practice. None.

Granted, this lack of knowledge management is a common problem with startup companies, especially those built around analytics-based technologies. However, at FC there were a wide variety of confounding factors that had created a far worse situation than usual. First, the scientific development team was located in a separate office a thousand miles away from company headquarters, and the distance was psychological as well as physical. Secondly, getting a solid understanding of the system’s logic was made more difficult as a result of the constant code modifications that were implemented on the fly, which also produced a lot of acronyms and buzzwords that only added to the internal confusion. Finally, and perhaps worst of all, the members of the company’s client services team had been hired for their relationship management skills, without any real regard to their analytic background or aptitude. Hiram was shocked to discover that there was not a single client services person who understood the significance of the configuration settings and that many did not even know that these settings even existed.

During the last several weeks of Hiram’s time working at FC, he was at the center of a pivotal A/B test with an important client. When the initial test produced only bad news for FC, a member of the executive team jumped in to personally lead the company’s efforts with this client’s Web site. “He went into military mode,” Hiram recounted. “We made endless lists of tasks, assigned owners and were ordered to report back in to him every four hours, seven days a week. Meanwhile, there were a lot of random voices chiming in from all sides, and a ton of wasted energy …”

When Hiram pushed back hard on the idea of tracking test results every four hours – “the sample sizes were definitely too small to understand what was really happening in that short a timeframe” – nobody wanted to hear it.

There are some important lessons to learn from FC, a promising analytics startup that woke up one day to find itself in serious financial trouble. There is no reliable textbook on how to make the treacherous journey from creative analytics concept to successful analytics company. Though most of them ultimately fail, a great many such analytics companies are launched every year, each with its own hopes of using smart algorithms encoded in software to channel the brainpower of their scientists into fame and fortune for their founders and funders. For these aspiring analytics entrepreneurs, Hiram’s sad story about his experiences at FC clearly points out many classic mistakes to avoid along the way.