Dark Data: How to Find It and What to Do with It

Like the proverbial man looking for his keys under the streetlight, when it comes to enterprise data, if you only look at where the light is already shining, you can end up missing a lot.

In a previous blog, I explored the value of dark data and how it can reveal insights that can streamline processes, improve customer experiences, generate more revenue – and maybe even help make the world a better place.

Looking where the light doesn’t shine

Remember that dark data is the data you have but don’t understand. It might be stored away in a forgotten silo – a byproduct of another process. Or it might lie in plain sight but needs refining and filtering to detect the diamonds that reveal insight.

So how do you find your dark data? Here are some ideas:

  • Look at your IT budget. Review your IT budget and figure out where you’re spending the most on storage. The data you’ve collected and saved over the years isn’t free. If storage costs are escalating in a particular area, you may have found a good source of dark data.
  • Analyze your metadata. If you’ve been properly managing your metadata as part of a broader data governance policy, you can use metadata management explorers to reveal silos of dark data in your landscape. If you’ve yet to implement data governance, this is another great reason to get moving quickly.
  • Create a catalog. Build a data catalog that systematizes data exploration and makes dark data stand out. Define and enforce cataloging standards (a top-down approach) – but also empower business users to create categories and add to the catalog (a bottom-up approach).
  • Aggregate and pool. Modern technologies allow the creation of data orchestration pipelines that help pool and aggregate dark data silos. With more dark data visible in a unified view, you’re in a better position to reveal hidden insights.
  • Use people. Going back to a core theme from my last blog, the best detectors of valuable data are people. If you’re trying to become a data-driven organization, it makes sense to hire someone whose job it is to care about data – like a chief data officer. Without such a role, your dark data may be much more likely to stay dark.

Tools for moving forward

The point of finding your dark data is to generate insight from it. To this end, SAP offers a wide range of tools that support the following capabilities:

  • Data orchestration. Information landscapes are complex. The SAP Data Intelligence Cloud solution helps you simplify your landscape with tools for creating data pipelines that integrate data and data streams on the fly for any type of use – from data warehousing to complex data science projects to real-time embedded analytics in business applications. It also helps you fix data quality problems so that you can separate the signal from the noise.
  • Real-time, cloud-based data ingestion and storage. With the SAP HANA Cloud database, you can ingest live data in real time where it lives. Data is delivered in context, enabling you to make better business decisions faster.
  • Data sense-making. Storing data isn’t enough. With the SAP Data Warehouse Cloud solution, you can connect data across multi-cloud and on-premise repositories while preserving business context – combining top-down data modeling with tools that enable business units to augment these models with their own particular data sources and needs.
  • Data analysis and exploration. Data is useless unless it can yield insights. With the SAP Analytics Cloud solution, you can put the right data into the hands of the right people who can then turn it into actionable insight for improving business performance.
  • Business process intelligence. Data that spans business processes can be a valuable source for improving efficiency and delivering better experience. With business process intelligence solutions from SAP, you can shine a light on end-to-end business processes, identify bottlenecks, and reveal best practices that should be copied more widely.

SAP and Intel

Almost all of these scenarios call for demanding processing capabilities – which is why SAP works closely with Intel. With Intel® Xeon® Scalable processors and Intel® Optane™ persistent memory technology, for example, Intel supports the highest levels of performance for memory-dependent large-scale datacenter applications, like AI, in-memory databases, content delivery networks (CDNs), and more.

Together, Intel and SAP help intelligent enterprises unlock the potential of data with real-time analytics and improve performance with modern infrastructure. The result is a suite of hardware foundations and software solutions that can handle the increased scale, capacity, and processing that “big dark data” demands – while addressing critical challenges around data privacy and regulations.

To sharpen your data strategy skills, download this outcome-driven data strategy masterclass. Here, you’ll get a 12-part video series and supporting workbook with templates.

Or to learn more about how to get greater value from your dark data, click here for information on unified data and analytics from SAP.

 

 

 


Posted

in

,

by