Interfacing Ginkgo with Eigen and slow performance of gko::read

[marco-butz]

Hi together,

I'm using Ginkgo to solve very large complex sparse non-hermitian linear systems.
I execute a program employing the Ginkgo library from another python script which passes the paths to the matrices "A.mtx", "b.mtx" (dense matrix / vector) and "initial_guess.mtx".

Saving the matrices in the python script is quite fast (~3s for A), but reading them through

auto A = gko::share(gko::read<gko::matrix::Csr<ValueType, GlobalIndexType>>(input_A, gpu));

is insanely slow (~ 70s for A). Also reading b and writing the result are not really fast. Therefore, the majority of the computation time is consumed by I/O operations done by ginkgo. So my general question is: How can I accelerate reading large sparse matrices from disk?

I employ Ginkgo in an iterative optimization setting, where I am aiming to replace the current MAGMA-based implementation. Ginkgo is indeed faster (and offers Multi-GPU for sparse matrices), but the I/O seems to make it infeasible...
For MAGMA I solved the issue by creating a sparse matrix in Eigen and filling it using the library fast_matrix_market: https://github.com/alugowski/fast_matrix_market

Eigen::SparseMatrix<std::complex<double>, Eigen::RowMajor> A;
std::ifstream A_stream(fileNameA);
fast_matrix_market::read_matrix_market_eigen(A_stream, A);

Afterwards I convert the Eigen matrix into a MAGMA matrix:

magma_z_matrix A_mag={Magma_CSR};
magma_zcsrset(A.rows(), A.cols(), A.outerIndexPtr(), A.innerIndexPtr(), reinterpret_cast <magmaDoubleComplex*> (A.valuePtr()), &A_mag, queue);

This process is fast. However, I cannot manage to convert the Eigen matrix into a ginkgo matrix.

Any help on how to accelerate gko::read or how to make Eigen and Ginkgo able to talk to each other would be very much appreciated.

Best,
Marco

[MarcelKoch]

Hi @marco-butz. Thanks for bringing up the subpar IO performance. The project you linked looks interesting. We will investigate, if it makes sense for us to also use it.
Regarding using Eigen matrices in Ginkgo, could you specify the issues you ran into? From the snippet that you showed, it should be possible to transfer this directly to Ginkgo.
Here is how I would do that, assuming that A.outerIndexPtr gives the row-pointers and A.innerIndexPtr the column indices:

auto row_ptrs_view = gko::make_array_view(exec, A.rows(), A.outerIndexPtr());
auto col_idxs = gko::make_array_view(exec, <nnz of A>, A.innerIndexPtr());
auto vals = gko::make_array_view(exec, <nnz of A>, A.valuePtr());

auto mtx = gko::matrix::Csr<ValueType>::create(exec, gko::dim<2>{A.rows(), A.cols()}, row_ptrs, col_idxs, vals);

Please take note that if the data of A is on the CPU, then you need to use exec->get_master() when creating the views. The exec can stay for the mtx as it will handle copying the data to GPU automatically.

[marco-butz]

Amazing, thank you very much for the answer. In fact, I probably mixed up column-major and row-major formats here. However, it works like you suggested.