Shedding light on a new business asset

Deep within the astonishing volumes of raw information generated by business transactions, social media, search engines, IoT and countless other sources, valuable intelligence about customers, markets and organizations, lies waiting to be discovered.

Leveraging advanced technologies to explore this expansive universe of unstructured and “dark” data reveals hidden insights to inform decision-making and chart new paths to the future. Machine learning, robotics process automation, visualization, natural language processing, and image and video analysis allow questions to be answered and opportunities brought to light that were unimaginable only a few years ago.

We’re only beginning to explore the digital universe, yet significant business value lies within. In fact, International Digital Corporation predicts that organizations that can analyze all relevant data and deliver actionable information could achieve an extra $430 billion in productivity gains over their peers by 2020 [1].

The influence of dark data is inescapable, increasingly driving innovation and value to customers and stakeholders. From financial services to healthcare, we’re starting to see the powerful impact of evolving technologies, new types of data and its potential to drive competitive advantage and shape entire industries. In the automotive industry, connected cars could help make usage-based insurance possible and accurately predict when vehicles need maintenance. Some automakers are beginning to experiment with an augmented reality dashboard.

Defining Dark Data

With a digital universe expected to reach 44 zettabytes by 2020, and 90 percent of it being unstructured data from the IoT and non-traditional sources [2], how is dark data defined? Today’s dark analytics efforts usually focus on three dimensions:

1. Untapped data already in your possession. Most organizations have large collections of structured and unstructured data sitting idle. On the structured side, it’s often due to the difficulty of making connections between disparate data sets. Insights lie waiting to be discovered: One company mapped the addresses of employees against workplace satisfaction ratings and retention data, finding that one of the biggest factors fueling voluntary turnover was commute time.

Unstructured data is often text-based, and until recently, tools and techniques needed to leverage them efficiently did not exist. Today, scanned patient records could hold the key to better understanding disease and prescribing effective treatments; executive emails and communications could unearth wisdom needed to pass along to a younger generation of workers.

2. Nontraditional unstructured data. Another dark analytics dimension focuses on data such as audio and video files and still images that could not be explored until now. Now, computer vision, advanced pattern recognition, and video and sound analytics allow companies to mine this data to better understand customers, employees, operations and markets.

This insight can be illuminating; retailers have a window into customer sentiment by analyzing in-store posture and facial expressions or online browsing patterns. Oil and gas companies can use acoustic sensors to monitor pipelines and algorithms to provide visibility into flow rates and fluid composition.

3. Data in the deep web. The deep web may offer the largest of body of untapped information – data curated by academics, government agencies, communities and other third-party domains – that is often hidden behind firewalls. Organizations may soon be able to curate competitive intelligence using search tools designed to help target specific data types. Analysis of this “deep web” data is especially promising in its potential to allow better prevention, detection and response to cyber threats.

Light Up the Dark Side

Developing a strategy for discovering value in unstructured data can help your organization generate insights today and prepare for even greater opportunities in the years ahead. How can your organization get the most value out of the mountains of data that it has created, owns or has access to? To help optimize the value of this business asset, consider these practical steps:

Ask the right questions. Work with business teams to identify specific questions that need to be answered. Then identify the sources of data that make the most sense for your analytics efforts. To boost sales of sports equipment, analytics teams can focus on sales transactions, inventory and product pricing in a specific geographic area. Supplementing the data with unstructured data, such as instore analysis of foot traffic or social media trends, generates more nuanced insights.

Look outside of your organization. Augmenting your data with publicly available demographic, location and statistical information helps put insights in context. For example, when a physician makes recommendations to an asthma patient about how to manage symptoms, he can also provide short-term solutions to help her deal with flare-ups during pollen season.

Expand analytics talent for impact. Optimizing dark data analytics depends on assembling the right teams to meet specific needs and identify emerging opportunities. Build a cohesive team that encompasses organizational, business and technical knowledge.

View analytics as a business-driven effort. Plan your project with your organization’s business goals in mind to determine the value that must be delivered, define the questions to ask, and decide how to harness data to generate the right answers. Data analytics then becomes an insight-driven advantage that wins support throughout the organization and fuels future endeavors.

Think broadly. As new strategies and capabilities for using dark analytics are developed, consider how they can be extended across the organization, as well as to customers, vendors and business partners.

By tapping into dark data, organizations have an opportunity to turn previously hidden or unknown patterns and correlations into powerful insights. Uncovering this business value leads to new opportunities, reduced risk and increased analytics ROI.

The potential is exciting, but staying grounded with specific business questions that have a defined scope and measurable value is critical. Focus your dark analytics efforts on areas that matter to your business – and avoid getting lost in the increasingly vast unknown.

References

1. “IDC FutureScape: Worldwide big data, business analytics, and cognitive software predictions,” International Data Corporation, 2016.
2. EMC Digital Universe with research and analysis by IDC, “The digital universe of opportunities: Rich data and the increasing value of the Internet of Things,” April 2014, https://www.emc.com/leadership/digital-universe/2014iview/index.htm