upgrade to 3.4 and fix interface changes

[anqini]

Note: Can we create a separate branch for this change to merge into? The changes are only compatible with scala-deequ version >=2.0.4 and not backward compatible. It will look fractional if we make it work universally since for the same class both new and old interfaces need to be invoked.

Issue #, if available:

#148

Forward compatibility issue for scala deequ version >=2.0.4.

Description of changes:

Update the invocation code according to the change of the Scala "deequ-2.0.6-spark-3.4" version interfaces

The interface changes include

• Analyzer class
  • Compliance
  • Histogram
  • MaxLength
  • MinLength
  • Mean
• Check methods
  • hasMaxLength
  • hasMinLength
  • satisfy

Logic change

• prioritize StringColumnProfile over StandardColumnProfile

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[chenliu0831]

Note: Can we create a separate branch for this change to merge into? The changes are only compatible with scala-deequ version >=2.0.4 and not backward compatible. It will look fractional if we make it work universally since for the same class both new and old interfaces need to be invoked.

Currently we don't want to maintain separate branch for different version combinations. I agree we shouldn't make the code fractional to achieve the goal and we are working with a long-term plan with changes in Deequ core ideally. I'll share more details around early Jan.

Let's keep this PR open for more ideas.

[anqini]

Thanks for your update. That sounds good.

…

________________________________ 发件人: Chenyang Liu ***@***.***> 发送时间: Thursday, December 14, 2023 11:18:31 PM 收件人: awslabs/python-deequ ***@***.***> 抄送: Angelo Ni ***@***.***>; Author ***@***.***> 主题: Re: [awslabs/python-deequ] upgrade to 3.4 and fix interface changes (PR #178) Note: Can we create a separate branch for this change to merge into? The changes are only compatible with scala-deequ version >=2.0.4 and not backward compatible. It will look fractional if we make it work universally since for the same class both new and old interfaces need to be invoked. Currently we don't want to maintain separate branch for different version combinations. I agree we shouldn't make the code fractional to achieve the goal and we are working with a long-term plan with changes in Deequ core ideally. I'll share more details around early Jan. Let's keep this PR open for more ideas. ― Reply to this email directly, view it on GitHub<#178 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AK7M4FCIMNLP76WRBDXMIJDYJMKEPAVCNFSM6AAAAABAUDCBQOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJWGA2DCMZWG4>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[LeandroLTM]

Hey @anqini @chenliu0831, is there a potential date to merge this PR? Looks like Deequ is already adding support to Spark 3.5

[chenliu0831]

Closing this PR for now - we do not plan to maintain multiple branches and the changes mostly likely have to go to Deequ

[theopilbeam]

@chenliu0831 could you explain more the backwards compatibility requirements for pydeequ? Is it strictly required that all future pydeequ versions continue to support all of the currently supported spark & deequ versions, or is there scope for dropping support?