rocmPackages.rocprofiler: fix rocprof and rocprofv2

[Madouura]

Description of changes

Tracking: #197885
Fixes: #262615

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.11 Release Notes (or backporting 23.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[Madouura]

Result of nixpkgs-review pr 262798 run on x86_64-linux 1

1 package built:

• rocmPackages.rocprofiler (rocmPackages_5.rocprofiler)

[pbsds]

these seem to be marked executable:

results/rocmPackages.rocprofiler/libexec/rocprofiler/counters/basic_counters.xml
results/rocmPackages.rocprofiler/libexec/rocprofiler/counters/derived_counters.xml

[Madouura]

So many weird things going on with this program.
Anyway thank you for pointing that out!

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/2816

[pbsds]

I'd like to learn more about the rocm tooling, how may I test this? I have a 580 if it helps

[Madouura]

A basic test for rocprofiler is pretty easy.
See: #262615 (comment)

1. Compile it with hipcc (rocmPackages.clr)
2. Make sure it runs ./a.out
3. Run it with rocprofiler rocprof --hip-trace a.out & rocprofv2 a.out
rocprof needs a -- option for any useful output. (--stats --roctx-trace --hip-trace --hsa-trace --sys-trace)
   I suggest just using rocprofv2 instead anyway.

[Madouura]

That said, I'm not entirely sure anything you do with rocprofiler will succeed on a 580.
I don't think polaris is supported in ROCm?

[pbsds]

Been at it for a while now, but it seems polaris support is truly dropped. Diff LGTM

[Flakebi]

You should be able to run rocm (or at least part of it) on polaris when you set the environment variable ROC_ENABLE_PRE_VEGA=1 (at runtime).

[pbsds]

Been trying it out, rebuilding my system with rocm stuff enabled, and no luck... My repro for those interested:

NIXPKGS_ALLOW_UNFREE=1 ROC_ENABLE_PRE_VEGA=1 nix shell --impure github:Madouura/nixpkgs/pr/rocprofiler#rocmPackages.{clr,rocprofiler} -c bash -x <(cat <<'EOF'
    cd "$(mktemp -d)"

    curl -L https://gist.githubusercontent.com/pbsds/aab148188bdcf308f70053d3e53cefb4/raw/935bfe11709e9e3b69b89435bcd487c69a742e5b/test.hip > test.hip
    hipcc test.hip
    ./a.out
    rocprof --hip-trace ./a.out
    rocprofv2 ./a.out

    rm -rf "$(pwd)"
EOF
)

output:

+ hipcc test.hip
+ ./a.out
DEVICE: 
Max thread per block: 0
/dev/fd/63: line 5: 21052 Floating point exception(core dumped) ./a.out
+ rocprof --hip-trace ./a.out
RPL: on '231026_160503' from '/nix/store/2sml94akzdgpm9vvdbpxf096d2gcnnhp-rocprofiler-5.7.1' in '/tmp/tmp.bZ4F6eKy5e'
RPL: profiling '"./a.out"'
RPL: input file ''
RPL: output dir '/tmp/rpl_data_231026_160503_21061'
RPL: result dir '/tmp/rpl_data_231026_160503_21061/input_results_231026_160503'
ROCtracer (21083):
    HIP-trace(*)
/nix/store/2sml94akzdgpm9vvdbpxf096d2gcnnhp-rocprofiler-5.7.1/bin/rocprof: line 301: 21083 Floating point exception(core dumped) "./a.out"
+ rocprofv2 ./a.out
DEVICE: hg���
Max thread per block: 0
/nix/store/2sml94akzdgpm9vvdbpxf096d2gcnnhp-rocprofiler-5.7.1/bin/rocprofv2: line 247: 21138 Floating point exception(core dumped) LD_PRELOAD=$LD_PRELOAD:$ROCPROFILER_DIR/lib/rocprofiler/librocprofiler_tool.so $*

Could you post your result @Flakebi ? I'm sorry if I'm being overly cautious.

[Flakebi]

I get this on a gfx10 / navi10 card (RX 5700 XT):

++ mktemp -d
+ cd /run/user/1000/tmp.p4asAXbx5a
+ curl -L https://gist.githubusercontent.com/pbsds/aab148188bdcf308f70053d3e53cefb4/raw/935bfe11709e9e3b69b89435bcd487c69a742e5b/test.hip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   977  100   977    0     0   3523      0 --:--:-- --:--:-- --:--:--  3527
+ hipcc test.hip
+ ./a.out
DEVICE: AMD Radeon RX 5700 XT
Max thread per block: 1024
0
1
2
3
4
5
6
7
8
9
+ rocprof --hip-trace ./a.out
RPL: on '231026_163017' from '/nix/store/h7piknhdjnv4z4y880dlg2xbd3c9x665-rocprofiler-5.7.1' in '/run/user/1000/tmp.p4asAXbx5a'
RPL: profiling '"./a.out"'
RPL: input file ''
RPL: output dir '/tmp/rpl_data_231026_163017_3895426'
/nix/store/h7piknhdjnv4z4y880dlg2xbd3c9x665-rocprofiler-5.7.1/bin/rocprof: line 560: /bin/ls: No such file or directory
Traceback (most recent call last):
  File "/nix/store/h7piknhdjnv4z4y880dlg2xbd3c9x665-rocprofiler-5.7.1/libexec/rocprofiler/tblextr.py", line 797, in <module>
    indir = re.sub(r'\/[^\/]*$', r'', infiles[0])
                                      ~~~~~~~^^^
IndexError: list index out of range
Profiling data corrupted: ''
+ rocprofv2 ./a.out
aqlprofile API table load failed: HSA_STATUS_ERROR: A generic error has occurred.
/nix/store/h7piknhdjnv4z4y880dlg2xbd3c9x665-rocprofiler-5.7.1/bin/rocprofv2: line 246: 3895453 Aborted                 (core dumped) LD_PRELOAD=$LD_PRELOAD:$ROCM_DIR/lib/rocprofiler/librocprofiler_tool.so $*
++ pwd
+ rm -rf /run/user/1000/tmp.p4asAXbx5a

[Madouura]

Only things that don't work for me is rocprof with --hsa-trace, and the default --stats.
I can't get rocprofv2 to fail at all.
(RX 6900 XT, gfx1030)

[Madouura]

I get this on a gfx10 / navi10 card (RX 5700 XT):

++ mktemp -d
+ cd /run/user/1000/tmp.p4asAXbx5a
+ curl -L https://gist.githubusercontent.com/pbsds/aab148188bdcf308f70053d3e53cefb4/raw/935bfe11709e9e3b69b89435bcd487c69a742e5b/test.hip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   977  100   977    0     0   3523      0 --:--:-- --:--:-- --:--:--  3527
+ hipcc test.hip
+ ./a.out
DEVICE: AMD Radeon RX 5700 XT
Max thread per block: 1024
0
1
2
3
4
5
6
7
8
9
+ rocprof --hip-trace ./a.out
RPL: on '231026_163017' from '/nix/store/h7piknhdjnv4z4y880dlg2xbd3c9x665-rocprofiler-5.7.1' in '/run/user/1000/tmp.p4asAXbx5a'
RPL: profiling '"./a.out"'
RPL: input file ''
RPL: output dir '/tmp/rpl_data_231026_163017_3895426'
/nix/store/h7piknhdjnv4z4y880dlg2xbd3c9x665-rocprofiler-5.7.1/bin/rocprof: line 560: /bin/ls: No such file or directory
Traceback (most recent call last):
  File "/nix/store/h7piknhdjnv4z4y880dlg2xbd3c9x665-rocprofiler-5.7.1/libexec/rocprofiler/tblextr.py", line 797, in <module>
    indir = re.sub(r'\/[^\/]*$', r'', infiles[0])
                                      ~~~~~~~^^^
IndexError: list index out of range
Profiling data corrupted: ''
+ rocprofv2 ./a.out
aqlprofile API table load failed: HSA_STATUS_ERROR: A generic error has occurred.
/nix/store/h7piknhdjnv4z4y880dlg2xbd3c9x665-rocprofiler-5.7.1/bin/rocprofv2: line 246: 3895453 Aborted                 (core dumped) LD_PRELOAD=$LD_PRELOAD:$ROCM_DIR/lib/rocprofiler/librocprofiler_tool.so $*
++ pwd
+ rm -rf /run/user/1000/tmp.p4asAXbx5a

Noticed the /bin/ls error.
I don't think you're using this PR's patch, since that shouldn't be coming up.
https://github.com/NixOS/nixpkgs/blob/977784c08e7afed994d349e2bfd243f06b5a0d94/pkgs/development/rocm-modules/5/rocprofiler/0001-fix-shell-scripts.patch#L107C16-L107C16

[Flakebi]

I don't think you're using this PR's patch, since that shouldn't be coming up.

Yeah, I merged this branch into some other branch, I would have expected it to work. Anyway, I tried again without merging and it seems to work fine.

Output

++ mktemp -d
+ cd /tmp/tmp.ulzeX6HqGV
+ curl -L https://gist.githubusercontent.com/pbsds/aab148188bdcf308f70053d3e53cefb4/raw/935bfe11709e9e3b69b89435bcd487c69a742e5b/test.hip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   977  100   977    0     0   2612      0 --:--:-- --:--:-- --:--:--  2612
+ hipcc test.hip
+ ./a.out
DEVICE: AMD Radeon RX 5700 XT
Max thread per block: 1024
0
1
2
3
4
5
6
7
8
9
+ rocprof --hip-trace ./a.out
RPL: on '231027_123732' from '/nix/store/2sml94akzdgpm9vvdbpxf096d2gcnnhp-rocprofiler-5.7.1' in '/tmp/tmp.ulzeX6HqGV'
RPL: profiling '"./a.out"'
RPL: input file ''
RPL: output dir '/tmp/rpl_data_231027_123732_4079854'
RPL: result dir '/tmp/rpl_data_231027_123732_4079854/input_results_231027_123732'
ROCtracer (4079874):
    HIP-trace(*)
DEVICE: AMD Radeon RX 5700 XT
Max thread per block: 1024
0
1
2
3
4
5
6
7
8
9
hsa_copy_deps: 0
scan ops data 1:2                                                                                                    File '/tmp/tmp.ulzeX6HqGV/results.copy_stats.csv' is generating
dump json 0:1
File '/tmp/tmp.ulzeX6HqGV/results.json' is generating
File '/tmp/tmp.ulzeX6HqGV/results.hip_stats.csv' is generating
dump json 11:12
File '/tmp/tmp.ulzeX6HqGV/results.json' is generating
File '/tmp/tmp.ulzeX6HqGV/results.stats.csv' is generating
dump json 0:1
File '/tmp/tmp.ulzeX6HqGV/results.json' is generating
+ rocprofv2 ./a.out
DEVICE: AMD Radeon RX 5700 XT
Max thread per block: 1024
0
1
2
3
4
5
6
7
8
9
Dispatch_ID(1), GPU_ID(1), Queue_ID(1), Queue_Index(0), Process_ID(4079928), Thread_ID(4079928), Grid_Size(1024), Workgroup_Size(1024), LDS(0), Scratch_Size(0), Arch_VGPR(8), Accumulative_VGPR(0), SGPR(128), Wave_Size(32), Kernel_Name("gene"), Begin_Timestamp(1465566548148299), End_Timestamp(1465566548153059), Correlation_ID(0)
++ pwd
+ rm -rf /tmp/tmp.ulzeX6HqGV

[pbsds]

Seems like the linked issue is fixed