Work around crash when profiling multi-process/multi-GPU application (R…

…OCm#376)

* Fix crash in multi-GPU scenario

Exclude -o option when invoking rocprof so that each rocprof process
writes to a different .csv file. Concatenate into a single .csv file
when finished.

Signed-off-by: benrichard-amd <ben.richard@amd.com>

* Only combine csv files when using rocprofv2

rocprofv1 does not have separate csv files

Signed-off-by: benrichard-amd <ben.richard@amd.com>

* Fix indices in combined CSV file

Use ignore_index flag to ensure there are no duplicate indices.

Signed-off-by: benrichard-amd <ben.richard@amd.com>

* Fix Dispatch_ID column and remove unnamed column

-Pandas was inserting an unnamed column (index column)
-Overwrite the Dispatch_ID column so that every row is unique, starting at 0
-Remove fixup_rocprofv2_dispatch_ids as no longer needed

Signed-off-by: benrichard-amd <ben.richard@amd.com>

* Fix code formatting

Signed-off-by: benrichard-amd <ben.richard@amd.com>

* Fix code formatting (for real this time)

Signed-off-by: benrichard-amd <ben.richard@amd.com>

---------

Signed-off-by: benrichard-amd <ben.richard@amd.com>
Signed-off-by: xuchen-amd <xuchen@amd.com>

src/omniperf_profile/profiler_rocprof_v2.py

@@ -48,6 +48,8 @@ def get_profiler_options(self, fname):
             # v2 requires output directory argument
             "-d",
             self.get_args().path + "/" + "out",
+            # v2 does not require quotes on cmd
+            app_cmd,
         ]
         args.extend(app_cmd)
         return args