Manual data testing is impossible because humans cannot deal with more than a few hundred records. Some data warehouse testing tools test their data by first storing data in a temporary database, such tools cannot scale and do not work for large volumes of data.

By its very nature a data warehouse has multiple data sources, different database vendors, and different data formats. Since the source database is different from the data warehouse it is very difficult to compare data across two systems. For each source the datatypes are different and so are the flavors of their SQL dialects.

If the data is received in files, then it adds another layer of complexities. Since it is not possible to analyze the data in a file with SQL, advanced scripting is required.

In a typical data warehouse project there are thousands of data processes. It is impossible to create manual test scripts for each of these data processes.

There is scheduling and orchestration of various interdependent data jobs that define the final data in a data warehouse. This requires equivalent testing effort to deal with the process sequencing and scheduling complexity.

A data warehouse typically integrates multiple records or updates one record from multiple data sources. It becomes very complex to test such conditions.

Even if the data is in production, changes in business often result in changes of the data processing patterns. These changes require regression testing of the data warehouse.

Unlike application screens, data warehouse testing is not static. The testing totally depends on the input data. Every time the data warehouse is loaded the input data changes and hence, the output. This makes testing very difficult.

Testing incremental load is difficult as the dataset needs dynamic conditions to choose only a specific part of the data at runtime of each test. Usually this is based on transactions and entry dates.

Just passing and failing a data test does not help. The test must be able to provide the exact row and column location of the data issue as it is needed for defect resolution.

Midrand, Gauteng
Republic of South Africa

Hyattsville, Maryland
United States of America

Agbado Ijaiye, Lagos State
Federal Republic of Nigeria