What is Dark Data? Why Does it Matter?

The stakes have never been higher for leaders, and each year presents a consistent upward trend in intelligence related challenges. Like any good leader our key focus is to consistently find the most optimal method of leveraging information to transform those challenges into opportunities. Data-driven decision-making, has been one of the most useful tools for leaders, however a significant portion of data remains shrouded in mystery, often overlooked and quite frankly oblivious to most leaders. This enigmatic entity is known as “dark data.” As organizations delve deeper into the intricacies of their data reservoirs, understanding the essence of dark data becomes increasingly paramount.

Gartner defines “dark data” as “the informational assets organizations, collect, process and store during regular business activities, but generally fail to use for other purposes (for example, analytics, business relationships and direct monetizing)¹. This definition refers to the vast amount of information that organizations collect, process, and store but fail to analyze or leverage for decision-making. Unlike structured and readily accessible data, dark data exists in unstructured formats, such as text documents, images, videos, and other types of data that are often overlooked or remain untouched.

The problem with this definition is that it doesn’t provide us with guidance on how to interact and use this data in a meaningful way. Simply put, who cares? If and organization has been doing fine without using dark data why should they expend the effort to start using it now.

Gartner was the first official source to provide a definition for dark data. Since then, there have been variations of the definition of it, as it is relational by context. For our exploration I will focus on the business need for value.

Value Focused Definition of Dark Data:

I’d like to introduce a value-focused definition for dark data pulled from the book, “Foundations for Value Creation: Data Products and Dark Data.” This definition will provide us with purpose and act as a driving force behind the need for dark data powered data products.

Dark data are data in any form that has or has not been attained and are needed but are not creating or providing value or impact. It is also, data that is unknown due to poor positioning, inability, illiteracy, obliviousness, or any impacting factor that is unintentional.

Foundations for Value Creation: Data Products and Dark Data
– Curtis Thompson

Essentially, dark data are the unused or underutilized data that could be valuable to an organization if properly identified and leveraged.

Understanding the impact of dark data on a company’s profitability is crucial as it can reveal untapped opportunities, improve decision-making, and increase overall efficiency. If left unaddressed, dark data can lead to lost revenue, increased costs, and missed opportunities for growth and innovation.

Tableau defines “data products” as “an application or tool that uses data to help businesses improve their decisions and processes².”

Value-focused Definition of Data Product

The value focused definition of data products is centered around how we can use the consistent evolution of data to enact value.

Data products are tools or assets that define their end user utility by creating value or having actionable impact based on the application of its constantly changing and evolving data; transforming data into useful information, that does something for an end user or enables an end user to do something better.

Foundations for Value Creation
– Curtis Thompson

Data products serve as vehicles for transforming dark data into useful intelligence that can be consumed by people and can be used to drive business decisions and improve operational efficiencies. This process involves several layers of transitions and transformations that convert raw data into information and then into intelligence. People typically access intelligence through user interfaces, which present data in a completely different format than it was initially collected. Examples of data products include business intelligence dashboards, predictive analytics models, recommendation engines, and data visualization tools.

Data products are typically designed to serve specific business needs, with the goal of making data accessible and actionable to stakeholders across the organization. They require a combination of technical expertise, data analytics skills, and business knowledge to develop and maintain.

Overall, data products can help organizations leverage their data assets to drive innovation, improve decision-making, and ultimately achieve better business outcomes.

Dark data contains a learning element that can provide valuable insights and help organizations gain a competitive advantage. In today’s fast-paced world, industries are becoming increasingly hyper-competitive, and extracting value from dark data can help organizations move from being competitive to top-tier or even industry leaders. Talent, technology, and positioning are all subject to change over time, but the unknowns that dark data brings to the table can provide a competitive edge. To capitalize on this advantage, organizations must proactively learn from the unknowns in dark data to improve their ability to create and deliver value.