"problem too large " for normalizeGiotto()

[lijinbio]

Hi,

I have an input matrix of 100,000 cells by 20,000 genes, and normalizeGiotto() exited with the below error.

Error in asMethod(object) : 
  Cholmod error 'problem too large' at file ../Core/cholmod_dense.c, line 102
Calls: source ... normalizeGiotto -> as.matrix -> as.matrix.Matrix -> as -> asMethod
Execution halted

Could you please help suggest how to apply normalizeGiotto() on large data? Thanks.

[RubD]

hi @lijinbio If you can convert your input matrix to a delayedMatrix, then you should be able to process very large datasets. It might take some time. We currently don't have a vignette for this yet, but we could try to make one if that would help.

[lijinbio]

Hi @RubD, thanks for your reply. I am able to bypass the error by turning off scaling when calling normalizeGiotto() (i.e., only normalized expression matrix is generated). I am sorry that I don't quite understand how to convert my input matrix to delayedMatrix and apply it to Giotto functions. My input matrix is a dgCMatrix class though. In fact, this "problem too large" error persists in the downstream analysis. For example, I again got the below error when calling binSpect() using the normalized expression.

Error in asMethod(object) : 
  Cholmod error 'problem too large' at file ../Core/cholmod_dense.c, line 102
Calls: source ... apply -> as.matrix -> as.matrix.Matrix -> as -> asMethod
Execution halted

Could you please help elaborate how could I convert my input matrix to a delayedMatrix? and I am expecting that the normalized or scaled expression matrix will be a delayedMatrix too. I hope this delayedMatrix method could resolve this "problem too large" error in other downstream functions too. Thanks.

[sinanugur]

Hi @RubD

Is there any update on this issue?

I tried using DelayedArray function from DelayedArray package but it did not help.

I used a public dataset from:
https://www.dropbox.com/sh/qmhdbqh56cp0t3n/AAD3CwN1ky5H2F6yPQOVh7Bva/Single%20Cell?dl=0&subfolder_nav_tracking=1

I got the RDS which contains a Seurat object. I pulled out the raw counts in sparse matrix format.

sc<-readRDS("all_compartments.RDS")

da <- DelayedArray::DelayedArray(sc@assays$RNA@counts)

instrs = createGiottoInstructions()

sc_data <- createGiottoObject(
    raw_exprs = da,
    instructions = instrs
)
Consider to install these (optional) packages to run all possible Giotto commands for spatial analyses:  MAST smfishHmrf trendsceek SPARK multinet RTriangle FactoMiner
 Giotto does not automatically install all these packages as they are not absolutely required and this reduces the number of dependenciesError in evaluate_expr_matrix(raw_exprs, cores = cores, sparse = TRUE) : 
  raw_exprs needs to be a path or an object of class 'Matrix', 'data.table', 'data.frame' or 'matrix'

BTW createGiottoObject does not fail for reading with sparse matrix but normalization fails later.

Thanks.