[SPARK-40770][PYTHON] Improved error messages for applyInPandas for schema mismatch#38223

Closed

EnricoMi wants to merge 21 commits intoapache:masterfrom

G-Research:branch-pyspark-apply-in-pandas-schema-mismatch

Commits on Feb 8, 2023

Add test on different column names
EnricoMi
committed
Improve error on schema mismatch, add more tests
EnricoMi
committed
Reuse _test_merge in test_pandas_cogrouped_map.py as much as possible
EnricoMi
committed
Make _test_apply_in_pandas* static as in test_pandas_cogrouped_map.py
EnricoMi
committed
Simplify error assertion in python/pyspark/sql/tests/test_pandas_cogrouped_map.py
EnricoMi
committed
Verify pandas result in wrap_cogrouped_map_pandas_udf
EnricoMi
committed
Remove classmethods in test, add comments
EnricoMi
committed
Fix python lint on lambda
EnricoMi
committed
Remove redundant test for empty DataFrame
EnricoMi
committed
Also provide expected schema in error message
EnricoMi
committed
Separate Missing, Unexpected, Expected and Schema by two spaces
EnricoMi
committed
Add field name to ValueError and TypeError message
EnricoMi
committed
Fix test_arrow.py asserting inner exception
EnricoMi
committed
Limit missing / unexpected columns to 5, schema string to 1024 characters
EnricoMi
committed
Remove argument n from create_array, use s.name instead
EnricoMi
committed
Simplify backslash use in expected message regexp
EnricoMi
committed
Call assign_cols_by_name once per dataframe, not once per group
EnricoMi
committed
Remove limiting number of columns and schema in error messages
EnricoMi
committed
Rework spacing of expected, missing, actual in error message
EnricoMi
committed
Rename test class names to proper name
EnricoMi
committed
Assert full expected result, not only subset
EnricoMi
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-40770][PYTHON] Improved error messages for applyInPandas for schema mismatch#38223

[SPARK-40770][PYTHON] Improved error messages for applyInPandas for schema mismatch#38223
EnricoMi wants to merge 21 commits intoapache:masterfrom
G-Research:branch-pyspark-apply-in-pandas-schema-mismatch

Commits on Feb 8, 2023

Add test on different column names

Improve error on schema mismatch, add more tests

Reuse _test_merge in test_pandas_cogrouped_map.py as much as possible

Make _test_apply_in_pandas* static as in test_pandas_cogrouped_map.py

Simplify error assertion in python/pyspark/sql/tests/test_pandas_cogrouped_map.py

Verify pandas result in wrap_cogrouped_map_pandas_udf

Remove classmethods in test, add comments

Fix python lint on lambda

Remove redundant test for empty DataFrame

Also provide expected schema in error message

Separate Missing, Unexpected, Expected and Schema by two spaces

Add field name to ValueError and TypeError message

Fix test_arrow.py asserting inner exception

Limit missing / unexpected columns to 5, schema string to 1024 characters

Remove argument n from create_array, use s.name instead

Simplify backslash use in expected message regexp

Call assign_cols_by_name once per dataframe, not once per group

Remove limiting number of columns and schema in error messages

Rework spacing of expected, missing, actual in error message

Rename test class names to proper name

Assert full expected result, not only subset