Dark Data Tiers

Value-focused Definition of Captive Data:

Data in your ecosystem that is known or unknown and unintentionally ne being used.

Captive Dark Data

Trying to classify dark data is difficult because it is very amorphous, here I attempt to describe the concept more plainly. I simplify the tiers of dark data so as we see how we can best work with the unknown.

Dark data is literally being created, captured and uncaptured almost every second of the day. It is flying around us at light speed, but most time we are oblivious of the opportunities it presents. To a novice this might sound a bit confusing, while to the best data engineers or data scientists it might sound obvious or overdramatic. Both reactions are correct in that we are aiming to develop a clear picture of what’s in front of us but is unknown

Interestingly enough, 65 percent of all data in an organization is dark, while 23 percent is redundant, obsolete, or trivial (what is called “rot data”) and 12 percent is critical to business operations. Dark data is hidden in network systems, email, documents, random hard drives, and in the minds of people.[i] This presents great opportunities for growth. Developing an understanding of what missing or being improperly used can improve enterprise performance greatly. Currently, 85 percent of companies lack the capability to unlock dark data, and 25% can only access structured data. The volume of data that exceeds the capacity to make use of it is 39 percent, while 66 percent of data is missing or incomplete. However, many enterprises already have ways to capitalize on dark data via protocols, capture mechanisms, automation, and machine learning.

Dark data can be classified into five tiers that fit into two classes. Class A is inside data—data that belongs to the organization and class B is outside data—data that does not. Furthermore, data and data sets can change tiers based on what is done with them and how that data is interacted with. For example, uncaptured data can be tier 1 or tier 2 depending on its nature. Typically, the data that sits outside of an organization occupies tiers 3, 4, and 5—this data can provide value, but a barrier in the form of people, culture, skills, priorities, needs, or capacity prevents its capture.