From manual to fully automated: data discovery for a digitalised world

This gap exists because they are using solutions that are either completely manual, or hybrids of manual and partially-automated tools. With data growth exponential and regulatory pressures ramping up, this approach is no longer sustainable.

Why Data Discovery Matters More Than Ever
Data discovery—which involves the collection and analysis of data across the organisation to enable firms to understand their data, gain insights from it and then act on those insights—is a growing corporate priority. It is a trend driven, in large part, by just how much volume today’s enterprises amass.

Yet comprehensive data discovery is extremely challenging to achieve. Where information has been created and managed is hard to keep track of, amid acquisition, myriad legacy technologies and a business environment where employees rarely stay in one position or company for several years.

Taken all together, this means that many organisations do not have a handle on what data they have at their disposal—and what they do not. They may think they do, but do not know for fact, which creates a serious issue, starting at the very top of the organisation. Most senior decision-makers today are focused on transforming their businesses and adapting to changing market environments. Moreover, they are also often asked to produce certain information through regulatory need and shareholder request. Regardless of the source of the need, these leaders must ensure that they have accurate data at hand. 

The Risks of Manual Data Discovery

The reality is that most organisations today are still using manual methods of data discovery. While they might have the capability to generate an automated report from the CRM system, for example, automation remains largely in pockets. Few businesses have automated the entire data discovery process end-to-end.

As a result, they typically do not have an end-to-end view of how their data is moving across the enterprise, nor if and how it is transforming along the way. Moreover, most manual data discovery processes are cumbersome, requiring a great deal of man-hours to pull relevant information, which often results in it being out-of-date as soon as it is created. Businesses may put a lot of work into data discovery, but when done manually, its value starts to diminish as soon as it is finished.  Another risk of a manual approach is that businesses can never be certain the data they are reporting to regulators is 100 percent accurate due to human error.

Beyond core compliance risks, businesses must also consider the risks that incomplete data poses to their agility. Without complete information, business may be unable to react quickly enough to opportunities. For example, if a special event cropped up quickly and the business wanted to re-price some merchandise, it would have to manually search for relevant data and trace it through all its systems to change all the pricing and coding. This is likely to be time-consuming and expensive and by the time the manual process is complete, the opportunity may have passed.   

The understanding of these risk factors is, in itself, a powerful driver for organisations to migrate from a manual to an automated approach.

Making the Move to Automated Data Discovery

To kick-start the move, proponents of automated data discovery need to first prove the case for it internally. The move to automation will obviously entail investment, so organisations need to start thinking about what business case they can make for it. With digital transformation at the forefront of many organisation’s priorities, this case cannot centre on the avoidance of regulatory fines and reducing risk. It must focus on how the investment is going to be leveraged to help the business evolve. That is, how can automated data discovery help the business become more profitable and achieve its end goals?

In line with that, it is key to determine what data the business needs to understand to help them achieve those goals. They must ask what systems hold that data and what areas of the data are likely to bring them most value, if automated?  As more projects and initiatives are automated, businesses can start filling in the gaps by identifying where the information is and what systems and technology it is connected to in order to determine what efforts should be prioritised next.

Technology plays a critical role in helping with this transition. Organisations need to span the metadata regardless of the technology it is in to see what transformations are taking place, even down to the code level. Then they need to bring that back into a metadata hub where they can visually display the end-to-end flow of physical data through underlying applications, services and data stores.  That flow of data then informs the application layer and it is at this stage that technology vendors can start to get the business more involved in the discussion about change management, transformation and compliance. It is here where business leaders’ insights are key, as they understand their systems and applications much better than the deep technical data flows involved.

With that foundation in place, the business can then overlay a data governance approach to better manage change to the evolving data environment. That involves identifying data stewards and data practitioners, while allowing data consumers to access information that is relevant to them. Being able to dynamically map all the data flows and dependencies is also key here as the key first step to getting stakeholders together to discuss the likely impact of any changes to the data environment and make informed decisions as to the best route forward.

The Benefits of Automated Data Discovery

With execution underway, the business benefits begin to come to fruition.

One of the most immediate benefits organisations see is the reduction of man days of effort made in capturing information. Once the system is automated, it runs continuously. It also gives the business the ability to look at and analyse the past—particularly what changes have been made to the environment and what their effects have been. Another benefit is that automated data discovery enables organisations to improve the way that data is visualised. Putting the end-to-end lineage on display allows stakeholders to clearly see how data is moving and transforming. When taken together, these benefits allow the organisation to see not only whether changes have, in fact, improved the environment or made it worse, but also make more informed decisions about future improvements to the environment.

Organisations are also able to more clearly understand if they are possibly missing out on key market opportunities. An automated approach allows them to do this much more efficiently, giving them the understanding of and insight into data they can use to create competitive advantage.

Only when organisations implement new automated systems and consign manual data discovery to the past can they have a fact-based understanding of their landscape and fully thrive in the new age of digital now upon us.