Hybrid Agile Methods in Big Data and Analytics Development

In a typical industry setting, data scientists develop predictive models in collaboration with software engineers, product managers and other domain experts, all while attempting to work within the scope of stakeholder needs. For the average research scientist transitioning into an industry data science role, however, this codependent way of working can be alien, a far cry from the independence more typical in the academic research world.

It is no real surprise then that many new analytics projects end in failure [1]. One reason for this is that data scientists and software engineers often fail to truly work in tandem, instead retreating to “waterfall”-like ways of working, where pieces of a project are “handed-off” in linear fashion from one group to another [2]. The result is often a breakdown in team communication and less successful collaboration overall.

Interestingly, new ideas are emerging on how best to integrate typical research data science methods with traditional software engineering concepts. In traditional software development, the sole focus is on iteration, iteration, iteration. In a hybrid setting, where research or data science principles are also at play, it is equally important to incorporate a parallel focus on experimentation, experimentation, experimentation. In other words, analytics developers should iterate and experiment, ensuring that they deliver regular, real analytics improvements along the way.

Similar suggestions have also recently been made in Russell Jurney’s manifesto for agile data science [2, 3]. By recognizing that data science is “part science” and “part engineering,” teams can more easily move the focus away from performing elemental engineering “tasks” and toward software development that is truly underpinned by research science principles. This type of approach can presumably help mitigate many of the causes of analytics project failure, e.g., as cataloged by Demirkan and Dal. [1].

As analytics developers in industries as diverse as fintech [4] and healthcare [5] develop these new hybrid ways of working, no doubt the software and analytics industries as a whole will also evolve.

References and Notes

1. http://analytics-magazine.org/the-data-economy-why-do-so-many-analytics-projects-fail/
2. https://www.oreilly.com/library/view/agile-data-science/9781491960103/
3. https://www.oreilly.com/ideas/a-manifesto-for-agile-data-science
4. https://www.forbes.com/sites/ciocentral/2018/07/10/how-fintech-initiatives-are-driving-financial-services-innovation/#7b036beb54fa
5. https://www.frontiersin.org/articles/10.3389/fdata.2018.00007/full