Profiling tool support for collection and analysis #2590
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Thomas Graves <tgraves@apache.org>
This reverts commit 6f5271c.
Signed-off-by: Thomas Graves <tgraves@apache.org>
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
|
I'll upmerge to #2588 once its merged |
|
build |
17 tasks
andygrove
reviewed
Jun 4, 2021
...4-spark-tools/src/main/scala/com/nvidia/spark/rapids/tool/profiling/CollectInformation.scala
Show resolved
Hide resolved
|
build |
andygrove
approved these changes
Jun 4, 2021
nartal1
approved these changes
Jun 4, 2021
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
* Qualification tool Signed-off-by: Thomas Graves <tgraves@apache.org> * remove unused func * Add missing files * Add checks for format option * cast columsn to string to write to text * Revert "Add checks for format option" This reverts commit 6f5271c. * cleanup Signed-off-by: Thomas Graves <tgraves@nvidia.com> * update output dir * formating * Update help messages * update app name * cleanup * put test functions back * fix typo * add printSQLPlanMetrics and printRapidsJar * use opt * Add Analysis * format output * more tests Signed-off-by: Thomas Graves <tgraves@apache.org> * tests working * test rearrange utils * move test file * move test file right location * add Analysis Suite * update test analysis * add rapids-4-spark-tools/src/test/resources/ProfilingExpectations/rapids_join_eventlog_jobandstagemetrics_expectation.csv * add more tests * more tests Signed-off-by: Thomas Graves <tgraves@nvidia.com> * remove unneeded expectation file * Add more analysis tests * comment * cleanup * fix logging include
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
* Qualification tool Signed-off-by: Thomas Graves <tgraves@apache.org> * remove unused func * Add missing files * Add checks for format option * cast columsn to string to write to text * Revert "Add checks for format option" This reverts commit 6f5271c. * cleanup Signed-off-by: Thomas Graves <tgraves@nvidia.com> * update output dir * formating * Update help messages * update app name * cleanup * put test functions back * fix typo * add printSQLPlanMetrics and printRapidsJar * use opt * Add Analysis * format output * more tests Signed-off-by: Thomas Graves <tgraves@apache.org> * tests working * test rearrange utils * move test file * move test file right location * add Analysis Suite * update test analysis * add rapids-4-spark-tools/src/test/resources/ProfilingExpectations/rapids_join_eventlog_jobandstagemetrics_expectation.csv * add more tests * more tests Signed-off-by: Thomas Graves <tgraves@nvidia.com> * remove unneeded expectation file * Add more analysis tests * comment * cleanup * fix logging include
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds in support for more profiling. printing rapids jar, printing sql plane metrics, then everything under Analysis - job and stage metrics and sql metrics aggregated and a check for shuffle skew.
Most of these were already in the code so it was just modifying a bit and then adding tests.