Profiling job fix

[SofiaSazonova]

Feature or Bugfix

• Bugfix

Detail

• ColumnProfilerRunner must be imported from pydeequ.profiles
• Workaround about missing SPARK_VERSION

Relates

• Profiling jobs failing with ImportError: cannot import name 'ColumnProfilerRunner' from 'awsglue.transforms' (/opt/amazon/lib/python3.6/site-packages/awsglue/transforms/__init__.py) #1216

Security

Please answer the questions below briefly where applicable, or write N/A. Based on
OWASP 10.

• Does this PR introduce or modify any input fields or queries - this includes
  fetching data from storage outside the application (e.g. a database, an S3 bucket)?
  • Is the input sanitized?
  • What precautions are you taking before deserializing the data you consume?
  • Is injection prevented by parametrizing queries?
  • Have you ensured no eval or similar functions are used?
• Does this PR introduce any functionality or component that requires authorization?
  • How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  • Are you logging failed auth attempts?
• Are you using or adding any cryptographic features?
  • Do you use a standard proven implementations?
  • Are the used keys controlled by the customer? Where are they stored?
• Are you introducing any new policies/roles/users?
  • Have you used the least-privilege principle? How?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[dlpzx]

I confirm that the changes fix the issue! In a separate PR we could upgrade the Glue version we are using to version 4, wdyt?

[SofiaSazonova]

I checked this script with newer versions of Glue, Spark and PyDeequ. It works without adjustments. At least for my test tables)