Delta Validation

Sometimes it is required to validate the data between two different data segments. This can be achieved by using the Delta validation. The Delta validation compares the data between two different data segments and provides the difference between them.

Configuration

The Delta validation requires the following configuration:

• on: The type of delta validation to be performed. The statement should start with delta followed by the type of supported validation function
• ref: The reference asset to compare with. The reference asset should follow the format <data_source_name>.<dataset_name> or <data_source_name>.<dataset_name>.<field_name>

Supported Delta Validations

• count_rows: This validation compares the count of rows between two different data segments.

Example

validations for iris_pgsql_source.dcs_iris:
  - reference count difference count rows:
      on: delta count_rows
      threshold: < 1
      ref: iris_pgsql_target.dcs_iris

In the above example we are comparing the count of rows between iris_pgsql_source.dcs_iris and iris_pgsql_target.dcs_iris. If the difference between the count of rows is less than 1, then the validation will pass.