Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Qualification tool: Report nested complex types in Potential Problems and improve write csv identification. #3521

Merged
merged 4 commits into from
Sep 18, 2021

Conversation

nartal1
Copy link
Collaborator

@nartal1 nartal1 commented Sep 16, 2021

This partially addresses NVIDIA/spark-rapids-tools#814 . Out of 3 tasks in the issue, this PR addresses 2.

  1. Report nested complex types in potential problems. Updated existing tests.
  2. Removed the hardcode of csv write format. Instead we are checking if the format != TypeSig.none. Running mvn verify did not update supportedDataSource.csv. Hence it does the same functionality without hardcoding. And later if we support csv write then we don't have to update the code here.
  3. Write format score is not reported as we might need further discussion on scoring in general.

@nartal1 nartal1 added the tools label Sep 16, 2021
@nartal1 nartal1 added this to the Sep 13 - Sep 24 milestone Sep 16, 2021
@nartal1 nartal1 self-assigned this Sep 16, 2021
@nartal1
Copy link
Collaborator Author

nartal1 commented Sep 17, 2021

build

Copy link
Collaborator

@tgravescs tgravescs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should also update the docs to explain this and perhaps say if you see this go look at what type it is in other field.

Signed-off-by: Niranjan Artal <nartal@nvidia.com>
@tgravescs
Copy link
Collaborator

build

@nartal1 nartal1 merged commit 52300bb into NVIDIA:branch-21.10 Sep 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants