Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates for ROCm 6 support #12681

Merged

Conversation

seanofthemillers
Copy link
Contributor

@trilinos/ShyLu

Motivation

These changes are meant to support building Trilinos with ROCm 6.0.

The KokkosKernels changes have already been upstreamed in the KokkosKernels repository.

The ShyLu changes are there to fix linking, and to handle some directory changes in ROCm 6.

FYI @jjellio

Stakeholder Feedback

Testing

@seanofthemillers seanofthemillers requested review from a team as code owners January 19, 2024 19:19
@trilinos-autotester
Copy link
Contributor

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

etphipp
etphipp previously approved these changes Jan 22, 2024
Copy link
Contributor

@etphipp etphipp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stokhos changes look good.

@seanofthemillers
Copy link
Contributor Author

Thanks guys, is there anyone from the ShyLu team that can double check the rocsolver/blas changes?

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

@ndellingwood
Copy link
Contributor

@iyamazaki can you review the changes to Tacho?

@ndellingwood
Copy link
Contributor

For the kokkos-kernels changes in this PR, I cherry-picked kokkos/kokkos-kernels#2050 to the release-candidate-4.2.01 branch so that changes in this PR are not clobbered by the patch release

@@ -65,6 +65,7 @@ TRIBITS_REPOSITORY_DEFINE_TPLS(
Cusp "cmake/TPLs/" ST
ROCBLAS "cmake/TPLs/" PT
ROCSPARSE "cmake/TPLs/" PT
ROCSOLVER "cmake/TPLs/" PT
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 for adding cmake/TPLs/FindTPLROCSOLVER.cmake , this will allow a follow-up revert of kokkos/kokkos-kernels#2037

@seanofthemillers
Copy link
Contributor Author

@ndellingwood Actually, I was curious how you guys were managing the snap-shooting of kokkos/kokkoskernels into Trilinos without submodules. How does one cherry-pick kokkos/kk changes into Trilinos?

@ndellingwood
Copy link
Contributor

How does one cherry-pick kokkos/kk changes into Trilinos?

Hi @seanofthemillers , I think what you did hear is just fine - you ensured the changes made to Trilinos were already submitted, reviewed, and approved by devs for the kokkos-kernels stand-alone repo. That is the necessary step to ensure that changes between the repos stay consistent.

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

@ndellingwood
Copy link
Contributor

Hi @seanofthemillers , the kokkos-kernels snapshot in PR #12707 is merged which includes the kokkos-kernels updates in this PR, would you be able to rebase this PR on top of develop? I'll approve and can add the AUTOMERGE label once that is done. Thanks for your patience waiting on that PR to complete

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

@seanofthemillers
Copy link
Contributor Author

Thanks @ndellingwood, I've rebased and squashed it down.

Copy link
Contributor

@ndellingwood ndellingwood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ndellingwood ndellingwood added AT: AUTOMERGE Causes the PR autotester to automatically merge the PR branch once approvals are completed AT: PRE-TEST INSPECTED Required to test outside contributions. This label alone will not allow a PR to merge. labels Feb 7, 2024
@seanofthemillers
Copy link
Contributor Author

@ndellingwood Actually I have a question - If I come across issues in KokkosKernels (and fix it there), is it better for me to patch them into Trilinos manually, or should I submit an issue to for you guys to pull in a fully updated KokkosKernels? I'm not sure what the standard approach is.

@trilinos-autotester trilinos-autotester removed the AT: PRE-TEST INSPECTED Required to test outside contributions. This label alone will not allow a PR to merge. label Feb 7, 2024
@trilinos-autotester
Copy link
Contributor

Status Flag 'Pre-Test Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED by label AT: PRE-TEST INSPECTED! Autotester is Removing Label; this inspection will remain valid until a new commit to source branch is performed.

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: Trilinos_PR_gcc-8.3.0

  • Build Num: 3635
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-8.3.0-openmpi-1.10.1-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS AT: AUTOMERGE
PULLREQUESTNUM 12681
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_REPO https://github.com/seanofthemillers/Trilinos
TRILINOS_SOURCE_SHA 7fd5b26
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85ae0b5

Build Information

Test Name: Trilinos_PR_gcc-8.3.0-serial

  • Build Num: 2134
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-v2-gnu-8.3.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS AT: AUTOMERGE
PULLREQUESTNUM 12681
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_REPO https://github.com/seanofthemillers/Trilinos
TRILINOS_SOURCE_SHA 7fd5b26
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85ae0b5

Build Information

Test Name: Trilinos_PR_gcc-8.3.0-debug

  • Build Num: 2124
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-8.3.0-openmpi-1.10.1-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS AT: AUTOMERGE
PULLREQUESTNUM 12681
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_REPO https://github.com/seanofthemillers/Trilinos
TRILINOS_SOURCE_SHA 7fd5b26
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85ae0b5

Build Information

Test Name: Trilinos_PR_clang-11.0.1

  • Build Num: 2123
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-clang-11.0.1-openmpi-1.10.1-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS AT: AUTOMERGE
PULLREQUESTNUM 12681
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_REPO https://github.com/seanofthemillers/Trilinos
TRILINOS_SOURCE_SHA 7fd5b26
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85ae0b5

Build Information

Test Name: Trilinos_PR_python3

  • Build Num: 3302
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-7.2.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS AT: AUTOMERGE
PULLREQUESTNUM 12681
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL ascic
TRILINOS_SOURCE_REPO https://github.com/seanofthemillers/Trilinos
TRILINOS_SOURCE_SHA 7fd5b26
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85ae0b5

Build Information

Test Name: Trilinos_PR_cuda-11.4.2-uvm-off

  • Build Num: 3126
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS AT: AUTOMERGE
PULLREQUESTNUM 12681
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL GPU
TRILINOS_SOURCE_REPO https://github.com/seanofthemillers/Trilinos
TRILINOS_SOURCE_SHA 7fd5b26
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85ae0b5

Build Information

Test Name: Trilinos_PR_intel-2021.3

  • Build Num: 1765
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-intel-2021.3-sems-openmpi-4.0.5_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-off_no-package-enables
PR_LABELS AT: AUTOMERGE
PULLREQUESTNUM 12681
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_REPO https://github.com/seanofthemillers/Trilinos
TRILINOS_SOURCE_SHA 7fd5b26
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85ae0b5

Using Repos:

Repo: TRILINOS (seanofthemillers/Trilinos)
  • Branch: rocm_6_fixes
  • SHA: 7fd5b26
  • Mode: TEST_REPO

Pull Request Author: seanofthemillers

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: Trilinos_PR_gcc-8.3.0

  • Build Num: 3635
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-8.3.0-openmpi-1.10.1-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS AT: AUTOMERGE
PULLREQUESTNUM 12681
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_REPO https://github.com/seanofthemillers/Trilinos
TRILINOS_SOURCE_SHA 7fd5b26
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85ae0b5

Build Information

Test Name: Trilinos_PR_gcc-8.3.0-serial

  • Build Num: 2134
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-v2-gnu-8.3.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS AT: AUTOMERGE
PULLREQUESTNUM 12681
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_REPO https://github.com/seanofthemillers/Trilinos
TRILINOS_SOURCE_SHA 7fd5b26
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85ae0b5

Build Information

Test Name: Trilinos_PR_gcc-8.3.0-debug

  • Build Num: 2124
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-8.3.0-openmpi-1.10.1-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS AT: AUTOMERGE
PULLREQUESTNUM 12681
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_REPO https://github.com/seanofthemillers/Trilinos
TRILINOS_SOURCE_SHA 7fd5b26
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85ae0b5

Build Information

Test Name: Trilinos_PR_clang-11.0.1

  • Build Num: 2123
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-clang-11.0.1-openmpi-1.10.1-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS AT: AUTOMERGE
PULLREQUESTNUM 12681
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_REPO https://github.com/seanofthemillers/Trilinos
TRILINOS_SOURCE_SHA 7fd5b26
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85ae0b5

Build Information

Test Name: Trilinos_PR_python3

  • Build Num: 3302
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-7.2.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS AT: AUTOMERGE
PULLREQUESTNUM 12681
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL ascic
TRILINOS_SOURCE_REPO https://github.com/seanofthemillers/Trilinos
TRILINOS_SOURCE_SHA 7fd5b26
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85ae0b5

Build Information

Test Name: Trilinos_PR_cuda-11.4.2-uvm-off

  • Build Num: 3126
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS AT: AUTOMERGE
PULLREQUESTNUM 12681
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL GPU
TRILINOS_SOURCE_REPO https://github.com/seanofthemillers/Trilinos
TRILINOS_SOURCE_SHA 7fd5b26
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85ae0b5

Build Information

Test Name: Trilinos_PR_intel-2021.3

  • Build Num: 1765
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-intel-2021.3-sems-openmpi-4.0.5_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-off_no-package-enables
PR_LABELS AT: AUTOMERGE
PULLREQUESTNUM 12681
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_REPO https://github.com/seanofthemillers/Trilinos
TRILINOS_SOURCE_SHA 7fd5b26
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85ae0b5


CDash Test Results for PR# 12681.

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ ndellingwood ]!

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pull Request AutoTester' - Pull Request will be Automerged

@trilinos-autotester trilinos-autotester merged commit 93d676f into trilinos:develop Feb 8, 2024
6 checks passed
@trilinos-autotester
Copy link
Contributor

Merge on Pull Request# 12681: IS A SUCCESS - Pull Request successfully merged

@trilinos-autotester trilinos-autotester removed the AT: AUTOMERGE Causes the PR autotester to automatically merge the PR branch once approvals are completed label Feb 8, 2024
@ndellingwood
Copy link
Contributor

ndellingwood commented Feb 8, 2024

@ndellingwood Actually I have a question - If I come across issues in KokkosKernels (and fix it there), is it better for me to patch them into Trilinos manually, or should I submit an issue to for you guys to pull in a fully updated KokkosKernels? I'm not sure what the standard approach is.

@seanofthemillers yeah, it's a good question. Generally we prefer kokkos-kernels changes to come into Trilinos via snapshot during releases, but we know the period between releases can be quite long and important changes (like in this PR) need to be patched directly to Trilinos (which still requires the separate PR to the kokkos-kernels repo and approvals from the devs). The process you followed in this PR was correct, where you submitted the change to kokkos-kernels and then peeled them off to submit to Trilinos (or could have been done in the opposite order). If you run into issues with merge conflicts between the separate branches feel free to ping me and I'll help to get them cleaned up. Thanks again for the work on this PR!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants