Invisible data may sound like an IT ghost story, but it’s a term that’s being applied to a chronic data problem afflicting many modern organizations. When we think about data within large companies, our minds invariably conjure up pictures of massive warehouses, populated with rows upon rows of servers, reminiscent of the famous scene from The Raiders of The Lost Ark. The more technically inclined among us add additional detail, thinking about Enterprise Data Warehouses, SharePoint servers and the like. While it is true that businesses use these technologies to manage the data, it is only a small part of the picture. How can an organization possibly have data that isn’t managed by the massive IT infrastructure? The remaining data, often the most important pieces of the puzzle, is invisible data.
As you’ve likely surmised at this point, invisible data is critical to the operation of the organization but is not managed by any of the plethora of existing enterprise IT systems. How can this happen? How can a company with a significant IT program fail to capture critical data? The answer is surprisingly simple. The gap between what enterprise IT is able to provide and what people need to run the everyday needs of the business is filled by the efforts of individual employees. The results of these efforts is invisible data.
Invisible data is the unending waterfall of Excel files, PDFs, Access Databases, CSV files, word documents, etc. that are produced by employees in order to bridge the gap between existing enterprise IT services and business needs. This data goes untracked, unmanaged and unrecognized. Only the resultant reports, documents and board room answers are ever acknowledged, but even these remain untracked by enterprise IT and become additional pieces in the invisible data puzzle.
This is a problem for numerous reasons:
1. Invisible data is a leading indicator that there is a problem with the existing enterprise solutions. People have created alternate solutions in order to get business done, and these solutions often take extensive amounts of time, as they involve numerous manual steps.
2. The processes for creating these invisible data artifacts are typically undocumented. When people change jobs or leave the company, their knowledge of these processes and the data created by them is often lost.
3. When an unexpected event occurs, such as a recall due to a part failure, or the contamination of an ingredient in the supply chain, the organization often cannot respond quickly. The information necessary to understand the full impact of this type of event is often held in invisible data, and it is either not accessible to the people who need to respond, or, again, cannot be generated quickly enough.
So now that we know what invisible data is and how it can affect businesses, what can we do about it? The good news is that there are several things that can be done to minimize the impact of invisible data and to eventually eliminate it altogether.
1. Promote a culture of data honesty. Recognize that your existing IT systems have gaps and people are likely filling those gaps with their own processes. Don’t punish the IT staff or the people working on their own processes.
2. Survey your existing invisible data. You will need to send a type of internal questionnaire or survey to find places where people have developed their own processes that are creating invisible data. Remember, they may be nervous admitting that they have circumvented existing IT processes, so you may need to incentivize people to come forward.
3. Start looking at new agile processes and technologies. There are new technologies that can help you to more easily, quickly and efficiently connect your data, eliminating the invisible data.
Remember, you are not alone. This is a common problem, especially with larger organizations. With a little diligence, you can reveal and eventually remove your invisible data.