Add BLAS to SpMV operations for BsrMatrix #1297

uhetmaniuk · 2022-01-31T16:25:26Z

Rebased version of PR #1296

Add calls to BLAS routines for SpMV of BsrMatrix (for all the operations on GPU)
Add 2 conjugate versions of BLAS routines GEMV and GEMM.

…routines.

uhetmaniuk · 2022-01-31T16:25:44Z

Here is the rebased version.

kokkos-devops-admin · 2022-01-31T16:29:09Z

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

lucbv

@uhetmaniuk this looks pretty good, I only have some minor comments.
Does the change to using more BLAS increase runtime or is it more or less neutral for performance? I know you mentioned the scaling operation might be more costly?

src/blas/impl/KokkosBlas2_team_gemv_spec.hpp

src/sparse/impl/KokkosSparse_spmv_bsrmatrix_impl.hpp

uhetmaniuk · 2022-01-31T20:12:09Z

@lucbv
I think the changes are neutral (at least on the example I ran).
I kept KokkosBlas::Scal when it is outside of the kernel and KokkosBatched::TeamVectorScaleInternal inside a kernel.

lucbv

Thanks for the changes, this looks fine.

kokkos-devops-admin · 2022-02-02T06:28:51Z

Status Flag 'Pre-Test Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED by label AT: PRE-TEST INSPECTED! Autotester is Removing Label; This inspection will remain valid until a new commit to source branch is performed.

kokkos-devops-admin · 2022-02-02T07:35:06Z

Status Flag 'Pull Request AutoTester' - Failure: Timed out waiting for job KokkosKernels_PullRequest_Tpls_CUDA9_Tpls_CUDA10_Tpls_CUDA10_LayoutRight_GCC720_Light_GCC720_GCC740 to start: Total Wait = 3603

Other jobs have been previously started - We must stop them...

kokkos-devops-admin · 2022-02-05T00:32:45Z

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

kokkos-devops-admin · 2022-02-05T00:32:51Z

Status Flag 'Pre-Test Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED by label AT: PRE-TEST INSPECTED! Autotester is Removing Label; This inspection will remain valid until a new commit to source branch is performed.

kokkos-devops-admin · 2022-02-05T00:34:32Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA9_Tpls_CUDA10_Tpls_CUDA10_LayoutRight_GCC720_Light_GCC720_GCC740

Build Num: 45
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	bsr-cuda-blas-spmv
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`14e56f8`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`4b9e29b`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1297
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC720

Build Num: 816
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	bsr-cuda-blas-spmv
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`14e56f8`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`4b9e29b`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1297
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC720_Light_LayoutRight

Build Num: 463
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	bsr-cuda-blas-spmv
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`14e56f8`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`4b9e29b`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1297
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_GCC720

Build Num: 807
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	bsr-cuda-blas-spmv
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`14e56f8`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`4b9e29b`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1297
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_INTEL18

Build Num: 795
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	bsr-cuda-blas-spmv
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`14e56f8`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`4b9e29b`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1297
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_CLANG1001

Build Num: 200
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	bsr-cuda-blas-spmv
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`14e56f8`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`4b9e29b`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1297
TEST_REPO_ALIAS	KOKKOSKERNELS

Using Repos:

Repo: KOKKOSKERNELS (NexGenAnalytics/kokkos-kernels)

Pull Request Author: uhetmaniuk

kokkos-devops-admin · 2022-02-05T03:10:30Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA9_Tpls_CUDA10_Tpls_CUDA10_LayoutRight_GCC720_Light_GCC720_GCC740

Build Num: 45
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	bsr-cuda-blas-spmv
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`14e56f8`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`4b9e29b`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1297
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC720

Build Num: 816
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	bsr-cuda-blas-spmv
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`14e56f8`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`4b9e29b`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1297
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC720_Light_LayoutRight

Build Num: 463
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	bsr-cuda-blas-spmv
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`14e56f8`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`4b9e29b`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1297
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_GCC720

Build Num: 807
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	bsr-cuda-blas-spmv
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`14e56f8`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`4b9e29b`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1297
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_INTEL18

Build Num: 795
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	bsr-cuda-blas-spmv
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`14e56f8`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`4b9e29b`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1297
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_CLANG1001

Build Num: 200
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	bsr-cuda-blas-spmv
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`14e56f8`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`4b9e29b`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1297
TEST_REPO_ALIAS	KOKKOSKERNELS

kokkos-devops-admin · 2022-02-05T03:10:50Z

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
THE LAST COMMIT TO THIS PULL REQUEST HAS NOT BEEN REVIEWED YET!

kokkos-devops-admin · 2022-02-05T03:10:56Z

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

kokkos-devops-admin · 2022-02-06T03:17:21Z

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

kokkos-devops-admin · 2022-02-07T03:21:59Z

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

lucbv

Let us see if it will start testing now...

kokkos-devops-admin · 2022-02-07T17:09:37Z

Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ lucbv ]!

kokkos-devops-admin · 2022-02-07T17:09:44Z

Status Flag 'Pull Request AutoTester' - Pull Request MUST BE MERGED MANUALLY BY Project Team - This Repo does not support Automerge

Update SpMV to use BLAS routines. Add conjugate operations to 2 BLAS …

586259b

…routines.

lucbv self-requested a review January 31, 2022 16:46

Remove unused typedef

de81ffd

lucbv reviewed Jan 31, 2022

View reviewed changes

Add more const characterizations. Use int for block_dim and num_rhs.

ac581a3

lucbv approved these changes Jan 31, 2022

View reviewed changes

lucbv added the AT: PRE-TEST INSPECTED Mark this PR as approved for testing. label Jan 31, 2022

kokkos-devops-admin removed the AT: PRE-TEST INSPECTED Mark this PR as approved for testing. label Feb 2, 2022

lucbv added the AT: RETEST Have this PR retested. label Feb 3, 2022

Merge branch 'develop' into bsr-cuda-blas-spmv

14e56f8

lucbv added the AT: PRE-TEST INSPECTED Mark this PR as approved for testing. label Feb 3, 2022

kokkos-devops-admin removed the AT: PRE-TEST INSPECTED Mark this PR as approved for testing. label Feb 5, 2022

kokkos-devops-admin removed the AT: RETEST Have this PR retested. label Feb 5, 2022

lucbv self-requested a review February 7, 2022 17:02

lucbv approved these changes Feb 7, 2022

View reviewed changes

lucbv merged commit ef10639 into kokkos:develop Feb 7, 2022

Add BLAS to SpMV operations for BsrMatrix #1297

Add BLAS to SpMV operations for BsrMatrix #1297

Conversation

uhetmaniuk commented Jan 31, 2022

uhetmaniuk commented Jan 31, 2022

kokkos-devops-admin commented Jan 31, 2022

lucbv left a comment

Choose a reason for hiding this comment

uhetmaniuk commented Jan 31, 2022

lucbv left a comment

Choose a reason for hiding this comment

kokkos-devops-admin commented Feb 2, 2022

kokkos-devops-admin commented Feb 2, 2022

kokkos-devops-admin commented Feb 5, 2022

kokkos-devops-admin commented Feb 5, 2022

kokkos-devops-admin commented Feb 5, 2022

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA9_Tpls_CUDA10_Tpls_CUDA10_LayoutRight_GCC720_Light_GCC720_GCC740

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_GCC720

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_GCC720_Light_LayoutRight

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_GCC720

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_INTEL18

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_CLANG1001

Jenkins Parameters

Using Repos:

kokkos-devops-admin commented Feb 5, 2022

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA9_Tpls_CUDA10_Tpls_CUDA10_LayoutRight_GCC720_Light_GCC720_GCC740

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_GCC720

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_GCC720_Light_LayoutRight

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_GCC720

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_INTEL18

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_CLANG1001

Jenkins Parameters

kokkos-devops-admin commented Feb 5, 2022

kokkos-devops-admin commented Feb 5, 2022

kokkos-devops-admin commented Feb 6, 2022

kokkos-devops-admin commented Feb 7, 2022

lucbv left a comment

Choose a reason for hiding this comment

kokkos-devops-admin commented Feb 7, 2022

kokkos-devops-admin commented Feb 7, 2022