Work around crash when profiling multi-process/multi-GPU application #376

benrichard-amd · 2024-06-06T21:50:36Z

Omniperf invokes rocprof to collect profiling data. rocprof has a -o option to specify an output CSV file. In the multi-GPU case, multiple processes will try to write to this file, corrupting it.

This change removes the -o option when invoking rocprof. Instead, Omniperf scans the output directory and combines the multiple CSV files into one CSV file.

Note: this only avoids the crash that occurs when using rocprofv2.

coleramos425 · 2024-06-07T14:46:17Z

@benrichard-amd in the multi-gpu case, can you confirm that we don't see any irregularities in csv output?

Handling Indexes: By default, pd. concat preserves the original indexes of the DataFrames or Series being concatenated. This can lead to duplicate index values, which might cause issues in subsequent data operations. You can use the ignore_index=True parameter to reset the index in the resulting DataFrame.

I'm not convinced that in the multi-gpu case we'll output a different set of counters in each run, thus I'm curious how this implementation handles merging overlapping output files.

coleramos425 · 2024-06-07T23:02:17Z

Additionally, could you add a sign-off on these commits? Information on how to do so can be found under the "DCO" action attached to this PR:

benrichard-amd · 2024-06-10T20:48:43Z

@benrichard-amd in the multi-gpu case, can you confirm that we don't see any irregularities in csv output?

Handling Indexes: By default, pd. concat preserves the original indexes of the DataFrames or Series being concatenated. This can lead to duplicate index values, which might cause issues in subsequent data operations. You can use the ignore_index=True parameter to reset the index in the resulting DataFrame.

I'm not convinced that in the multi-gpu case we'll output a different set of counters in each run, thus I'm curious how this implementation handles merging overlapping output files.

Good catch. Looking more closely we were seeing the unnamed index column in the output. This has been fixed.

The output CSVs look good and have correct indexing.

Exclude -o option when invoking rocprof so that each rocprof process writes to a different .csv file. Concatenate into a single .csv file when finished. Signed-off-by: benrichard-amd <ben.richard@amd.com>

rocprofv1 does not have separate csv files Signed-off-by: benrichard-amd <ben.richard@amd.com>

Use ignore_index flag to ensure there are no duplicate indices. Signed-off-by: benrichard-amd <ben.richard@amd.com>

-Pandas was inserting an unnamed column (index column) -Overwrite the Dispatch_ID column so that every row is unique, starting at 0 -Remove fixup_rocprofv2_dispatch_ids as no longer needed Signed-off-by: benrichard-amd <ben.richard@amd.com>

Signed-off-by: benrichard-amd <ben.richard@amd.com>

coleramos425

LGTM

…OCm#376) * Fix crash in multi-GPU scenario Exclude -o option when invoking rocprof so that each rocprof process writes to a different .csv file. Concatenate into a single .csv file when finished. Signed-off-by: benrichard-amd <ben.richard@amd.com> * Only combine csv files when using rocprofv2 rocprofv1 does not have separate csv files Signed-off-by: benrichard-amd <ben.richard@amd.com> * Fix indices in combined CSV file Use ignore_index flag to ensure there are no duplicate indices. Signed-off-by: benrichard-amd <ben.richard@amd.com> * Fix Dispatch_ID column and remove unnamed column -Pandas was inserting an unnamed column (index column) -Overwrite the Dispatch_ID column so that every row is unique, starting at 0 -Remove fixup_rocprofv2_dispatch_ids as no longer needed Signed-off-by: benrichard-amd <ben.richard@amd.com> * Fix code formatting Signed-off-by: benrichard-amd <ben.richard@amd.com> * Fix code formatting (for real this time) Signed-off-by: benrichard-amd <ben.richard@amd.com> --------- Signed-off-by: benrichard-amd <ben.richard@amd.com>

…376) * Fix crash in multi-GPU scenario Exclude -o option when invoking rocprof so that each rocprof process writes to a different .csv file. Concatenate into a single .csv file when finished. Signed-off-by: benrichard-amd <ben.richard@amd.com> * Only combine csv files when using rocprofv2 rocprofv1 does not have separate csv files Signed-off-by: benrichard-amd <ben.richard@amd.com> * Fix indices in combined CSV file Use ignore_index flag to ensure there are no duplicate indices. Signed-off-by: benrichard-amd <ben.richard@amd.com> * Fix Dispatch_ID column and remove unnamed column -Pandas was inserting an unnamed column (index column) -Overwrite the Dispatch_ID column so that every row is unique, starting at 0 -Remove fixup_rocprofv2_dispatch_ids as no longer needed Signed-off-by: benrichard-amd <ben.richard@amd.com> * Fix code formatting Signed-off-by: benrichard-amd <ben.richard@amd.com> * Fix code formatting (for real this time) Signed-off-by: benrichard-amd <ben.richard@amd.com> --------- Signed-off-by: benrichard-amd <ben.richard@amd.com>

…OCm#376) * Fix crash in multi-GPU scenario Exclude -o option when invoking rocprof so that each rocprof process writes to a different .csv file. Concatenate into a single .csv file when finished. Signed-off-by: benrichard-amd <ben.richard@amd.com> * Only combine csv files when using rocprofv2 rocprofv1 does not have separate csv files Signed-off-by: benrichard-amd <ben.richard@amd.com> * Fix indices in combined CSV file Use ignore_index flag to ensure there are no duplicate indices. Signed-off-by: benrichard-amd <ben.richard@amd.com> * Fix Dispatch_ID column and remove unnamed column -Pandas was inserting an unnamed column (index column) -Overwrite the Dispatch_ID column so that every row is unique, starting at 0 -Remove fixup_rocprofv2_dispatch_ids as no longer needed Signed-off-by: benrichard-amd <ben.richard@amd.com> * Fix code formatting Signed-off-by: benrichard-amd <ben.richard@amd.com> * Fix code formatting (for real this time) Signed-off-by: benrichard-amd <ben.richard@amd.com> --------- Signed-off-by: benrichard-amd <ben.richard@amd.com> Signed-off-by: xuchen-amd <xuchen@amd.com>

benrichard-amd requested review from koomie and coleramos425 as code owners June 6, 2024 21:50

benrichard-amd added 6 commits June 11, 2024 08:14

Fix crash in multi-GPU scenario

3e559ae

Exclude -o option when invoking rocprof so that each rocprof process writes to a different .csv file. Concatenate into a single .csv file when finished. Signed-off-by: benrichard-amd <ben.richard@amd.com>

Only combine csv files when using rocprofv2

f3bf9b5

rocprofv1 does not have separate csv files Signed-off-by: benrichard-amd <ben.richard@amd.com>

Fix indices in combined CSV file

d3ffa6a

Use ignore_index flag to ensure there are no duplicate indices. Signed-off-by: benrichard-amd <ben.richard@amd.com>

Fix code formatting

bc48b21

Signed-off-by: benrichard-amd <ben.richard@amd.com>

Fix code formatting (for real this time)

786d527

Signed-off-by: benrichard-amd <ben.richard@amd.com>

benrichard-amd force-pushed the multi-gpu branch from 285d05c to 786d527 Compare June 11, 2024 13:14

coleramos425 approved these changes Jun 12, 2024

View reviewed changes

coleramos425 merged commit 4d7d624 into ROCm:dev Jun 12, 2024
9 of 10 checks passed

benrichard-amd deleted the multi-gpu branch July 17, 2024 20:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Work around crash when profiling multi-process/multi-GPU application #376

Work around crash when profiling multi-process/multi-GPU application #376

benrichard-amd commented Jun 6, 2024

coleramos425 commented Jun 7, 2024

coleramos425 commented Jun 7, 2024

benrichard-amd commented Jun 10, 2024

coleramos425 left a comment

Work around crash when profiling multi-process/multi-GPU application #376

Work around crash when profiling multi-process/multi-GPU application #376

Conversation

benrichard-amd commented Jun 6, 2024

coleramos425 commented Jun 7, 2024

coleramos425 commented Jun 7, 2024

benrichard-amd commented Jun 10, 2024

coleramos425 left a comment

Choose a reason for hiding this comment