Should is_not_empty exist & what happens when coverage (pct) is zero?

[marrov]

Hello,

I was just checking the is_empty check and bumped into the following. My understanding is that it sums the number of NULL values in a column. Therefore, considering that the typical thing that one would want is to check that a column does not contain null values, the usage of this rule for the described check should be:

check = Check().add_rule(method="is_empty", column="id", pct=0)

However, this fail because:

ValueError: Coverage should be between 0 and 1

This is not a very informative error because the type of bounds (> or >= ?) are not defined and they are asymmetrical (i.e. pct can be 1 but cannot be 0).

So considering this, my suggestion is to add an is_not_empty check to be able to write the desired usecase as:

check = Check().add_rule(method="is_not_empty", column="id", pct=1)

And a more general question, why is zero not allowed as a pct? Should it be?

Thanks!

[marrov]

Oh wow... I feel silly now: is_not_empty does exist but it is called is_complete 🤦

Nevertheless, my question on informative errors in terms of valid bounds still stands.

[canimus]

Hi @marrov no worries, I also sometime overlook the checks available, and choosing for a representative name is always tricky.
The Data Management Body of Knowledge or DAMA-DMBOK has some nice references when it comes to data quality standards. In table 2, first row, the quality attribute of completeness is cited. From there the is_complete check
Reference: https://www.dama-nl.org/wp-content/uploads/2020/09/DDQ-Dimensions-of-Data-Quality-Research-Paper-version-1.2-d.d.-3-Sept-2020.pdf
With regards to the zero pct is a nice one, I am thinking in area of logical constructs, that like mathematically there is an inverse function, cuallee could inverse the function based on the percentage. Meanwhile I think simply allowing 0 as part of the lower bound could fix the remark you made.