Error in silhouetteRankTest "/par." file not found

[alexanderjwhite]

I'm attempting to produce Giotto results for the Spatial LIBD data using the SEDR Analyses

The issue seems to be that the par files are not getting created via the do_gpd.R script called within the python stack. I'm not sure why this is happening. Thoughts?

Output

`R version 4.0.4 (2021-02-15) -- "Lost Library Book"
Copyright (C) 2021 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

paths <- .libPaths()
paths <- c("/geode2/home/u100/whitealj/BigRed3/r_4_0_4_library/",paths)
.libPaths(paths)

library(Giotto)
library(Seurat)
Registered S3 method overwritten by 'spatstat.geom':
method from
print.boxx cli
Attaching SeuratObject
library(ggplot2)
library(patchwork)
options(bitmapType = 'cairo')

args <- commandArgs(trailingOnly = TRUE)
sample.name <- args[1]
n_cluster <- args[2]

1. Load Data

data_path = file.path('../data/DLPFC/', sample.name)
dir.output = file.path('../output/DLPFC/', sample.name, 'Giotto')

if(!dir.exists(file.path(dir.output))){

• dir.create(file.path(dir.output), recursive = TRUE)
• }

expr_data_path=fs::path(data_path, "filtered_feature_bc_matrix.h5")
raw_matrix=get10Xmatrix_h5(path_to_data=expr_data_path)$Gene Expression
spatial_locations=data.table::fread(fs::path(data_path, "spatial", "tissue_positions_list.csv"))
spatial_locations = spatial_locations[match(colnames(raw_matrix), V1)]
colnames(spatial_locations) = c('barcode', 'in_tissue', 'array_row', 'array_col', 'col_pxl', 'row_pxl')
myinst=createGiottoInstructions(save_plot=T, show_plot=F, save_dir=dir.output)

no external python path was provided, but a giotto python environment was found and will be used
sh: /geode2/home/u100/whitealj/BigRed3/.local/share/r-miniconda/envs/giotto_env/lib/libtinfo.so.6: no version information available (required by /lib64/libreadline.so.7)

2. Create Giotto object & process data.

visium_brain <- createGiottoObject(raw_exprs = raw_matrix,

•                                spatial_locs = spatial_locations[,.(row_pxl,-col_pxl)], 
  

•                                instructions = myinst, 
  

•                                cell_metadata = spatial_locations[,.(in_tissue, array_row, array_col)])
  

Consider to install these (optional) packages to run all possible Giotto commands for spatial analyses: scran MAST smfishHmrf trendsceek SPARK multinet RTriangle FactoMiner
Giotto does not automatically install all these packages as they are not absolutely required and this reduces the number of dependenciessh: /geode2/home/u100/whitealj/BigRed3/.local/share/r-miniconda/envs/giotto_env/lib/libtinfo.so.6: no version information available (required by /lib64/libreadline.so.7)
Warning messages:
1: In createGiottoObject(raw_exprs = raw_matrix, spatial_locs = spatial_locations[, :
module: igraph was not found with python path: /geode2/home/u100/whitealj/BigRed3/.local/share/r-miniconda/envs/giotto_env/bin/python

2: In createGiottoObject(raw_exprs = raw_matrix, spatial_locs = spatial_locations[, :
module: leidenalg was not found with python path: /geode2/home/u100/whitealj/BigRed3/.local/share/r-miniconda/envs/giotto_env/bin/python

3: In createGiottoObject(raw_exprs = raw_matrix, spatial_locs = spatial_locations[, :
module: community was not found with python path: /geode2/home/u100/whitealj/BigRed3/.local/share/r-miniconda/envs/giotto_env/bin/python

4: In createGiottoObject(raw_exprs = raw_matrix, spatial_locs = spatial_locations[, :
module: networkx was not found with python path: /geode2/home/u100/whitealj/BigRed3/.local/share/r-miniconda/envs/giotto_env/bin/python

metadata = pDataDT(visium_brain)
in_tissue_barcodes = metadata[in_tissue == 1]$cell_ID
visium_brain = subsetGiotto(visium_brain, cell_ids = in_tissue_barcodes)

filter genes and cells

visium_brain <- filterGiotto(gobject = visium_brain,

•                          expression_threshold = 1,
  

•                          # gene_det_in_min_cells = 50,
  

•                          # min_det_genes_per_cell = 500,
  

•                          expression_values = c('raw'),
  

•                          verbose = T)
  

Number of cells removed: 2 out of 4226
Number of genes removed: 23711 out of 33538

normalize

visium_brain <- normalizeGiotto(gobject = visium_brain, scalefactor = 6000, verbose = T)

first scale genes and then cells

add gene & cell statistics

visium_brain <- addStatistics(gobject = visium_brain)

3. Dimensional reduction

HVG

visium_brain <- calculateHVG(gobject = visium_brain)
return_plot = TRUE and return_gobject = TRUE

      plot will not be returned to object, but can still be saved with save_plot = TRUE or manually 

PCA

select genes based on HVG and gene statistics, both found in gene metadata

gene_metadata = fDataDT(visium_brain)
featgenes = gene_metadata[hvg == 'yes' & perc_cells > 3 & mean_expr_det > 0.4]$gene_ID

run PCA on expression values (default)

visium_brain <- runPCA(gobject = visium_brain,

•                    genes_to_use = featgenes, 
  

•                    scale_unit = F, 
  

•                    center=T, 
  

•                    method="factominer")
  

a custom vector of genes will be used to subset the matrix

UMAP and tSNE

visium_brain <- runUMAP(visium_brain, dimensions_to_use = 1:20)
visium_brain <- runtSNE(visium_brain, dimensions_to_use = 1:20)

create spatial network

visium_brain <- createSpatialNetwork(gobject=visium_brain,

•                                  method='kNN', 
  

•                                  k=5, 
  

•                                  maximum_distance_knn=400, 
  

•                                  name='spatial_network')
  

silhouette

spatial_genes=silhouetteRankTest(visium_brain,

•                              overwrite_input_bin=F, 
  

•                              output=file.path(dir.output, "sil.result"), 
  

•                              matrix_type="dissim", 
  

•                              num_core=20, 
  

•                              verbose=T, 
  

•                              expression_values="norm", 
  

•                              query_sizes=10)
  

sh: /geode2/home/u100/whitealj/BigRed3/.local/share/r-miniconda/envs/giotto_env/lib/libtinfo.so.6: no version information available (required by /lib64/libreadline.so.7)
Reading gene expression...
Reading cell coordinates...
Calculate all pairwise Euclidean distance between cells using their physical coordinates
For rbp_p 0.95:
Start ranking forward...
Done 0 of 4224
Done 1000 of 4224
Done 2000 of 4224
Done 3000 of 4224
Done 4000 of 4224
Start ranking backward...
Done 0 of 4224
Done 1000 of 4224
Done 2000 of 4224
Done 3000 of 4224
Done 4000 of 4224
Calculate mutual rank...
Calculate exponential transform...
For rbp_p 0.99:
Start ranking forward...
Done 0 of 4224
Done 1000 of 4224
Done 2000 of 4224
Done 3000 of 4224
Done 4000 of 4224
Start ranking backward...
Done 0 of 4224
Done 1000 of 4224
Done 2000 of 4224
Done 3000 of 4224
Done 4000 of 4224
Calculate mutual rank...
Calculate exponential transform...
Start calculating silhouette rank, saving logs to log directory (check progress here)...
sh: /usr/bin/parallel: No such file or directory
cat: write error: Broken pipe
Start randomization, saving logs to log directory (check progress here)...
sh: /usr/bin/parallel: No such file or directory
cat: write error: Broken pipe
Start computing P-values...
Entering silhouette.exact.rbp.0.95.top.0.005.txt...
Setting query size 3 instead of 10
Error in py_call_impl(callable, dots$args, dots$keywords) :
FileNotFoundError: [Errno 2] No such file or directory: '../output/DLPFC//151507/Giotto/sil.result/silhouetteRank_output//result_5000_0.95_0.005/par.43'

Detailed traceback:
File "", line 100, in silhouette_rank
File "/geode2/home/u100/whitealj/BigRed3/.local/share/r-miniconda/envs/giotto_env/lib/python3.6/site-packages/silhouetteRank/use_previous_cluster.py", line 140, in do_one
f = open("%s/par.%d" % (outdir,target))
Calls: silhouetteRankTest -> silhouette_rank -> py_call_impl
Execution halted`

Session Info
R version 4.0.4 (2021-02-15)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: SUSE Linux Enterprise Server 15 SP2

Matrix products: default
BLAS/LAPACK: /geode2/soft/hps/cle6/intel/19.5/compilers_and_libraries_2019.5.281/linux/mkl/lib/intel64_lin/libmkl_rt.so

locale:
[1] C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] eva_0.2.6 patchwork_1.1.1 ggplot2_3.3.5 SeuratObject_4.0.4
[5] Seurat_4.1.0 Giotto_1.1.0

[bernard2012]

Hi,

SilhouetteRankTest requires the silhouetteRank python package to be installed:
https://github.com/bernard2012/silhouetteRank (see Prerequisite and Installation section).

It also requires the parallel unix command (typically located in /usr/bin/parallel, but it could be different if you are using Macs).
Once you have both, specify the parallel_path parameter in silhouetteRankTest function.

Also make sure that when you install SilhouetteRankTest python package, you are installing it to the python used by Giotto. I see you are not specifying the python path in createGiottoInstruction. So to avoid any confusion or errors in not able to find python package, explicitly define python_path.