Introduction
Data is an asset as well as a liability. In a complex enterprise environment, we have large IT systems involving multiple enhancements and change requests all the time. It is critical to test software changes on production-like data to mitigate any issues which may otherwise arise in production. For this purpose, it is a common practice for enterprises to have a copy of production data in non-production environments — usually in UAT, Pre-Prod and SIT environments.
This provides the development team with the ability to test on real data, however, it introduces significant data compliance risks. Often there are developers and quality assurance professionals from different external vendors engaged in these projects, sometimes offsite. Hence the risk of personally identifiable and sensitive data being mishandled is quite high. There have been instances of communications being triggered to actual customers from test environments, which is a serious issue and can lead to reputational risk and significant damage control costs.
We need to find a balance between data compliance needs and the ability of the QA team to provide similar quality assurance as if the test data were real production data. Data masking is a process through which we can convert datasets into structurally similar but inauthentic (non-identifiable) versions which can be used for testing and other activities such as user training.
DX1 Approach to Enterprise Data Masking
DX1's approach to enterprise data masking is based on executing an overarching process which involves key subprocesses such as identifying PII data (also known as data profiling), technical setup, tools selection, development of masking scripts, QA and handover.
PII Data Identification
Identifying Personally Identifiable Information (PII) is the critical first step in enterprise data masking. DX1 has developed frameworks to engage relevant stakeholders to define and identify PII data in the business context. Typical stakeholders include application owners, SMEs, the test team and InfoSec/CyberSecurity teams. DX1 recommends first developing an overarching definition of PII data and applying it consistently across applications to avoid confusion.
Tool Selection and Technical Design
Although there are multitudes of similar-looking toolsets for data masking, we cannot provide the best solution without understanding the infrastructure and environment landscape of the business. In our experience, a significantly large amount of effort may be consumed in arriving at the correct architecture for a data masking appliance. A proof of concept with the appropriate tool is highly recommended to test assumptions and resolve technical challenges early on.
Quality Assurance
QA is an important step of the masking process. DX1 recommends QA activities be carefully planned rather than an afterthought. There are three layers of QA activities proposed as part of masking process:
- Technical Tests: Mostly performed by DBAs and Data Engineers, these involve back-end verification and comparing records against production data.
- UI/Regression Tests: Usually performed by system testers, involving screenshots of key PII fields before and after masking to verify coverage.
- End-to-End Tests: The most critical and time-consuming part, verifying referential integrity across interconnected systems using critical end-to-end scenarios.
Governance
DX1 has developed its own governance model for managing all types of Quality Engineering projects including Data Masking. Our governance model is based on proactive management of risks and issues, real-time traceability and effective management of stakeholder expectations. We can provide frameworks to manage the data masking project as part of Agile, Waterfall or any other delivery model as applicable.
Conclusion
Data masking, for a standalone database, is straightforward and relatively simple. However, when multiple interconnected applications are involved, this becomes a complex engineering task requiring careful planning and experienced execution. DX1's approach and experience in this space can provide significant value on both the process and technology sides. Please reach out to us to understand challenges in data compliance and our approach to solving them.
