The Four Core Principles of Extracting Value from Dark Data
Data is generally an engineering disciple however, Extracting value from Dark Data is more of an art that is founded on engineering and scientific principles. Being able to obtain value from Dark Data can be very difficult to do effectively. The purpose and impact of the principles outlined here are: reduce unknowns, increase transparency, operate in control, create value optimally, and enhance overall collective intelligence.
The Four core principles for extracting value from Dark Data are:
- Commitment to data granularity
- Design for change
- Standardization of data enrichment
- Dynamic intelligence management
Data Granularity
Data granularity is a measure of the level of detail in a data structure. In time-series data.
Example: In time-series data, the granularity of measurement might be based on intervals of years, months, weeks, days, or hours.
Design for Change
Designing for change builds off the foundation principle of Granularity. Change is inevitable, whether we realize it or not everything is constantly changing daily, along with our teams, organization, and our data.
- Are we prepared for change when it happens?
- Can we navigate the change to Growth? Or are we driving change?
- And are we better prepared to attain the most value from the change we initiate?
Standardization of Data Enrichment
Standardizing newly created Intelligence, Insights, Information, & Data that was derived from the source data, relational data and environmental knowns. Data Enrichment can be strategically used in many data assets, however the data assets that find the most nuanced use for data enrichment are Dynamic intelligence layers (DIL). These small data layers are typically developed for highly specialized use cases and contain large array’s of calculations and derived insights strategically integrated to create a picture for the end user that drills into a specific opportunity. These DIL’s also provide the capability for scale and repeatability, allowing for users to have up to the minute intelligence on demand making easier for people to make confident and well advised decisions.
Intelligence Management
Intelligence management is focused around how we use newly created data elements, and insights collectively to create a new picture of the intelligence gathered. This picture should be unique in nature and present the information in a manner that allows people to understand nuances and opportunities that they could not before, because the new information generated was hidden in Dark Data.
With intelligence management we are focusing on the Dynamic nature of intelligence from the perspective of Dark Data and Data Products. We will use the tools listed below to generate the value that we are seeking.
- Intelligence layer
- ODL (Operational Data Layer)
- ODS Operational Data Store
- Data Mart
- RDBMS
- Data Warehouse
- Data Lake
