error in multiqc -- KeyError: 'No metadata except PKG-INFO is available' #51

kelly-sovacool · 2023-11-21T16:55:48Z

Error reported by Krithika & Ying.

snakemake.log

Error executing rule rnaseq_multiqc on cluster (jobid: 324, external: 12899399, jobscript: /vf/users/CCRCCDI/analysis/ccrtegs4/renee_20231120/.snakemake/tmp.39gqdzne/snakejob.rnaseq_multiqc.324.sh). For error details see the cluster log and the log files of the involved rule(s).
Exiting because a job execution failed.

logfiles/slurmfiles/12886741.12899073.rnaseq_multiqc..err:

/// MultiQC 🔍 | v1.12
 
|           multiqc | MultiQC Version v1.17 now available!
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3023, in _dep_map
      return self.__dep_map
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2817, in __getattr__
    raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map
 
During handling of the above exception, another exception occurred:
 
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3014, in _parsed_pkg_info
    return self._pkg_info
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2817, in __getattr__
    raise AttributeError(attr)
AttributeError: _pkg_info
 
During handling of the above exception, another exception occurred:
 
Traceback (most recent call last):
  File "/usr/local/bin/multiqc", line 5, in <module>
    from multiqc.__main__ import multiqc
  File "/usr/local/lib/python3.8/dist-packages/multiqc/__main__.py", line 53, in <module>
    multiqc.run_cli(prog_name="multiqc")
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/multiqc/multiqc.py", line 249, in run_cli
    multiqc_run = run(
  File "/usr/local/lib/python3.8/dist-packages/multiqc/multiqc.py", line 612, in run
    template_mod = config.avail_templates[config.template].load()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2444, in load
    self.require(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2467, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 795, in resolve
    new_requirements = dist.requires(req.extras)[::-1]
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2738, in requires
    dm = self._dep_map
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3025, in _dep_map
    self.__dep_map = self._compute_dependencies()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3034, in _compute_dependencies
    for req in self._parsed_pkg_info.get_all('Requires-Dist') or []:
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3016, in _parsed_pkg_info
    metadata = self.get_metadata(self.PKG_INFO)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1896, in get_metadata
    raise KeyError("No metadata except PKG-INFO is available")
KeyError: 'No metadata except PKG-INFO is available'
[Tue Nov 21 02:01:48 2023]

The text was updated successfully, but these errors were encountered:

kelly-sovacool · 2023-11-21T17:27:54Z

I'm seeing if the error can be reproduced on the test dataset. Update: I didn't get any error with the test data

renee run --input .tests/*.R?.fastq.gz --output /data/sovacoolkl/renee_test2 --genome hg38_30 --mode slurm

I suspect this could be related to the fact that biowulf upgraded the singularity module to v4.0.1, which may have introduced breaking changes.

kelly-sovacool · 2023-12-06T15:41:34Z

Update: Krithika got this same error with the test dataset. It seems to be a software environment issue.

slsevilla · 2023-12-06T17:30:03Z

Second PI running on Biowulf is seeing this error.

Plan to test myself and see if I can replicate since @kelly-sovacool could not.

kelly-sovacool · 2023-12-06T17:39:19Z

Krithika is testing it now with --singularity-cache /data/CCBR_Pipeliner/SIFS, we'll see if that works.

kelly-sovacool · 2023-12-06T18:14:17Z

Krithika is testing it now with --singularity-cache /data/CCBR_Pipeliner/SIFS, we'll see if that works.

Failed:

renee: error:
            [6;37;41mFatal: Failed to provided a valid singularity cache![0m
                The provided --singularity-cache already exists on the filesystem with a different owner.
                Singularity strictly enforces that the cache directory is not shared across users.
                Please run /gpfs/gsfs10/users/CCBR_Pipeliner/Pipelines/RENEE/.v2.5.8/renee again with a different --singularity-cache location.

I'm confused about this because it works for me and I'm able to use this shared cache dir location in my nextflow pipelines.
Do we need to change the permissions for /data/CCBR_Pipeliner/SIFS to make sure users not in the CCBR_Pipeliner group can use it?

kelly-sovacool · 2023-12-06T20:05:58Z

alt. potential solution: use different docker container that has been working in nextflow pipelines 6987849

Edit: this still failed for Krithika

kelly-sovacool · 2023-12-14T15:34:54Z

Solutions that we (Krithika) tried:

use shared singularity cache (--singularity-cache /data/CCBR_Pipeliner/SIFS)
- fails for non-CCBR users because they don't have access to write to that dir.
use singularity v3.10.5 instead of v4.0.1 (module load singularity/v3.10.5 before renee run)
- fails with same multiqc error
use different docker container docker://nciccbr/ccbr_multiqc_1.15:v1
- fails with same multiqc error
use --sif-cache /data/CCBR_Pipeliner/SIFS
- fails with same multiqc error

kopardev · 2023-12-14T17:46:32Z

Look into the differences between SINGULARITY_LOCALCACHEDIR SINGULARITY_CACHEDIR and SINGULARITY_TMPDIR

kelly-sovacool · 2023-12-15T20:26:49Z

multiqc worked for Krithika outside of the snakemake workflow.

script:

#!/usr/bin/env bash
#SBATCH --cpus-per-task=2
#SBATCH --mem=20g
#SBATCH --time=00:10:00
#SBATCH --parsable
#SBATCH -J "test-renee"
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH --output "test_singularity.log"
#SBATCH --error "test_singularity.log"

workdir=/data/CCRCCDI/analysis/ccrtegs4/renee_test/output_test_sifcache/
#workdir=/data/sovacoolkl/renee_test/
cd $workdir
module load singularity
export SINGULARITY_CACHEDIR=${workdir}/.singularity

cmd="
which python;
echo $PATH;
multiqc --ignore '*/.singularity/*' -f -c ${workdir}/resources/multiqc_config.yaml --interactive --outdir ${workdir}/Reports ${workdir};
echo '=========='
"

echo "echo TEST1; $cmd" |\
    singularity exec -B $workdir docker://nciccbr/ccbr_multiqc_1.15:v1 bash
echo "echo TEST2; $cmd" |\
    singularity exec -e -B $workdir docker://nciccbr/ccbr_multiqc_1.15:v1 bash

output:

[+] Loading singularity  4.0.1  on cn4332 
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob sha256:e88f0d2e3e7849afafd4041fd86764a5f53b536aadb461dd3205b95459580653
Copying blob sha256:56e0351b98767487b3c411034be95479ed1710bb6be860db6df0be3a98653027
Copying blob sha256:a51180f2933c21cd973c72aa7fa8188d28c349f7c64670306792b7ca7e742d39
Copying blob sha256:733cf2d680d6f8ed2337737c3697405d0996278069d8b60a1672d5f21278bdcc
Copying blob sha256:25b3077843af896cd92eeadc7d1926f97d3578b224afae4636862519cc3c025b
Copying blob sha256:5f3ff2784635ccf884c461fe5a7ff4fab25389a418726d8bcf63b3f97277f9e6
Copying blob sha256:5421ffbaa795a5726aa4cbbc087320c7a6588bee26a8ca9e1f1636244827f1e8
Copying blob sha256:5de5bc716a2dd322f2e23e25d234d14676e7813dc4e82fc6620f6eb190818600
Copying blob sha256:5c454f3b0b73f2cb9e33421f95afeb397b057b3eb4679bbf94a9968e75b7815b
Copying blob sha256:6e258499a1cb942df31da05b81e1daaef80441d6d4956938b7cc5a16da2b7894
Copying blob sha256:6e258499a1cb942df31da05b81e1daaef80441d6d4956938b7cc5a16da2b7894
Copying blob sha256:835c15da49d683bb52e781fea7230c6d5e8fc9eb47cd8b3e9a96ead937b6fc7f
Copying blob sha256:835c15da49d683bb52e781fea7230c6d5e8fc9eb47cd8b3e9a96ead937b6fc7f
Copying blob sha256:8f3452a0019dc5658d852299ffa6b96f8403ad6d8c058a5e9add57bd004b5781
Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
Copying blob sha256:62e9366d14e7e5a7b04e68897fab7ff7f67694bd9fd358ac11c4fb50b91cf8d2
Copying blob sha256:dd8f4fddbd3eee320c75b2d4606339aaed7ae3e0cf1f014a3957941a8a834afe
Copying blob sha256:dd8f4fddbd3eee320c75b2d4606339aaed7ae3e0cf1f014a3957941a8a834afe
Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
Copying blob sha256:1d0817e0b5db957a9526eceb13733b65485ec36465621103e99ac2faa66cf6ed
Copying config sha256:b7087ffa6887c3a240cac3a33271335f972db5f9a97c1bc11f8f6b88bff9ba49
Writing manifest to image destination
2023/12/15 14:50:11  info unpack layer: sha256:56e0351b98767487b3c411034be95479ed1710bb6be860db6df0be3a98653027
2023/12/15 14:50:11  info unpack layer: sha256:a51180f2933c21cd973c72aa7fa8188d28c349f7c64670306792b7ca7e742d39
2023/12/15 14:50:11  info unpack layer: sha256:5f3ff2784635ccf884c461fe5a7ff4fab25389a418726d8bcf63b3f97277f9e6
2023/12/15 14:50:11  info unpack layer: sha256:733cf2d680d6f8ed2337737c3697405d0996278069d8b60a1672d5f21278bdcc
2023/12/15 14:50:12  info unpack layer: sha256:25b3077843af896cd92eeadc7d1926f97d3578b224afae4636862519cc3c025b
2023/12/15 14:50:14  info unpack layer: sha256:e88f0d2e3e7849afafd4041fd86764a5f53b536aadb461dd3205b95459580653
2023/12/15 14:50:23  info unpack layer: sha256:5421ffbaa795a5726aa4cbbc087320c7a6588bee26a8ca9e1f1636244827f1e8
2023/12/15 14:50:23  info unpack layer: sha256:5de5bc716a2dd322f2e23e25d234d14676e7813dc4e82fc6620f6eb190818600
2023/12/15 14:50:26  info unpack layer: sha256:5c454f3b0b73f2cb9e33421f95afeb397b057b3eb4679bbf94a9968e75b7815b
2023/12/15 14:50:26  info unpack layer: sha256:6e258499a1cb942df31da05b81e1daaef80441d6d4956938b7cc5a16da2b7894
2023/12/15 14:50:26  info unpack layer: sha256:6e258499a1cb942df31da05b81e1daaef80441d6d4956938b7cc5a16da2b7894
2023/12/15 14:50:26  info unpack layer: sha256:835c15da49d683bb52e781fea7230c6d5e8fc9eb47cd8b3e9a96ead937b6fc7f
2023/12/15 14:50:26  info unpack layer: sha256:835c15da49d683bb52e781fea7230c6d5e8fc9eb47cd8b3e9a96ead937b6fc7f
2023/12/15 14:50:26  info unpack layer: sha256:8f3452a0019dc5658d852299ffa6b96f8403ad6d8c058a5e9add57bd004b5781
2023/12/15 14:50:26  info unpack layer: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
2023/12/15 14:50:26  info unpack layer: sha256:62e9366d14e7e5a7b04e68897fab7ff7f67694bd9fd358ac11c4fb50b91cf8d2
2023/12/15 14:50:28  info unpack layer: sha256:dd8f4fddbd3eee320c75b2d4606339aaed7ae3e0cf1f014a3957941a8a834afe
2023/12/15 14:50:28  info unpack layer: sha256:dd8f4fddbd3eee320c75b2d4606339aaed7ae3e0cf1f014a3957941a8a834afe
2023/12/15 14:50:28  info unpack layer: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
2023/12/15 14:50:28  info unpack layer: sha256:1d0817e0b5db957a9526eceb13733b65485ec36465621103e99ac2faa66cf6ed
INFO:    Creating SIF file...
TEST1
Illegal option --
Usage: /usr/bin/which [-a] args
/usr/local/current/singularity/4.0.1/bin:/home/bhuvaneshwark2/.local/bin:/home/bhuvaneshwark2/bin:/usr/local/slurm/bin:/usr/local/bin:/usr/X11R6/bin:/usr/local/jdk/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/mysql/bin

  /// MultiQC 🔍 | v1.15

|           multiqc | MultiQC Version v1.17 now available!
|           multiqc | Search path : /data/CCRCCDI/analysis/ccrtegs4/renee_test/output_test_sifcache
|         searching | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 883/883  
|             bbmap | Found 4 reports
|      fastq_screen | Found 16 reports
|            kraken | Found 4 reports
|            fastqc | Found 16 reports
|          cutadapt | Found 4 reports
|              star | Found 4 reports and 4 gene count files
|            preseq | Found 4 reports
|              rsem | Found 4 reports
|          qualimap | Found 4 BamQC reports
|             rseqc | Found 4 read_distribution reports
|             rseqc | Found 4 inner_distance reports
|             rseqc | Found 4 infer_experiment reports
|             rseqc | Found 4 tin reports
|            picard | Found 4 MarkDuplicates reports
|            picard | Found 4 RnaSeqMetrics reports
|          samtools | Found 4 flagstat reports
|           multiqc | Report      : Reports/multiqc_report.html
|           multiqc | Data        : Reports/multiqc_data
|           multiqc | MultiQC complete
==========
INFO:    Using cached SIF image
TEST2
/usr/bin/python
/usr/local/current/singularity/4.0.1/bin:/home/bhuvaneshwark2/.local/bin:/home/bhuvaneshwark2/bin:/usr/local/slurm/bin:/usr/local/bin:/usr/X11R6/bin:/usr/local/jdk/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/mysql/bin

  /// MultiQC 🔍 | v1.15

|           multiqc | MultiQC Version v1.17 now available!
|           multiqc | Search path : /data/CCRCCDI/analysis/ccrtegs4/renee_test/output_test_sifcache
|         searching | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 884/884  
|             bbmap | Found 4 reports
|      fastq_screen | Found 16 reports
|            kraken | Found 4 reports
|            fastqc | Found 16 reports
|          cutadapt | Found 4 reports
|              star | Found 4 reports and 4 gene count files
|            preseq | Found 4 reports
|              rsem | Found 4 reports
|          qualimap | Found 4 BamQC reports
|             rseqc | Found 4 read_distribution reports
|             rseqc | Found 4 inner_distance reports
|             rseqc | Found 4 infer_experiment reports
|             rseqc | Found 4 tin reports
|            picard | Found 4 MarkDuplicates reports
|            picard | Found 4 RnaSeqMetrics reports
|          samtools | Found 4 flagstat reports
|           multiqc | Report      : Reports/multiqc_report.html   (overwritten)
|           multiqc | Data        : Reports/multiqc_data   (overwritten)
|           multiqc | MultiQC complete
==========

kopardev · 2023-12-20T14:43:35Z

Try:

different version of snakemake
different multiqc docker/sif

kelly-sovacool · 2023-12-20T14:47:37Z

will update on FRCE /mnt/projects/CCBR-Pipelines/pipelines/RENEE and see if we get the error

kelly-sovacool · 2024-01-16T17:17:05Z

Let's release v2.5.11 and ask Krithika to re-run (after #95)

kopardev · 2024-01-17T15:03:21Z

Seems like this is working now ... weird!!

kelly-sovacool · 2024-01-17T15:09:46Z

How bizarre! I'm glad it's working now but wish we knew why.

kopardev · 2024-01-17T15:37:05Z

@kelly-sovacool ... just double check if @Krithika-Bhuvan still has this error!!

slsevilla · 2024-01-17T15:39:00Z

I just completed a run on Biowulf with main branch and it worked for me. I think we should leave this open until we have Krithika test it

kelly-sovacool · 2024-01-17T16:00:27Z

Krithika is re-running it now

kelly-sovacool · 2024-01-17T17:39:38Z

Krithika's test succeeded!

kopardev added the RENEE RepoName label Nov 21, 2023

kelly-sovacool added the bug Something isn't working label Nov 21, 2023

kelly-sovacool self-assigned this Dec 6, 2023

kelly-sovacool mentioned this issue Dec 6, 2023

Use shared singularity cache dir on biowulf and frce #53

Closed

This was referenced Dec 8, 2023

shared singularity cache doesn't work for non-CCBR users CCBR/CHAMPAGNE#161

Open

fix: use container from nciccbr for multiqc #55

Closed

kelly-sovacool added this to the Jan 2024 milestone Jan 10, 2024

kopardev closed this as completed Jan 17, 2024

slsevilla reopened this Jan 17, 2024

kelly-sovacool closed this as completed Jan 17, 2024

kopardev modified the milestones: Jan 2024, Jan 2025 Jan 24, 2024

kelly-sovacool modified the milestones: Jan 2025, Jan 2024 Jan 24, 2024

kopardev modified the milestones: Jan 2024, 2024-01 Jan 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

error in multiqc -- KeyError: 'No metadata except PKG-INFO is available' #51

error in multiqc -- KeyError: 'No metadata except PKG-INFO is available' #51

kelly-sovacool commented Nov 21, 2023

kelly-sovacool commented Nov 21, 2023 •

edited

Loading

kelly-sovacool commented Dec 6, 2023

slsevilla commented Dec 6, 2023

kelly-sovacool commented Dec 6, 2023

kelly-sovacool commented Dec 6, 2023 •

edited

Loading

kelly-sovacool commented Dec 6, 2023 •

edited

Loading

kelly-sovacool commented Dec 14, 2023 •

edited

Loading

kopardev commented Dec 14, 2023 •

edited

Loading

kelly-sovacool commented Dec 15, 2023

kopardev commented Dec 20, 2023

kelly-sovacool commented Dec 20, 2023

kelly-sovacool commented Jan 16, 2024

kopardev commented Jan 17, 2024

kelly-sovacool commented Jan 17, 2024

kopardev commented Jan 17, 2024

slsevilla commented Jan 17, 2024

kelly-sovacool commented Jan 17, 2024

kelly-sovacool commented Jan 17, 2024

error in multiqc -- KeyError: 'No metadata except PKG-INFO is available' #51

error in multiqc -- KeyError: 'No metadata except PKG-INFO is available' #51

Comments

kelly-sovacool commented Nov 21, 2023

kelly-sovacool commented Nov 21, 2023 • edited Loading

kelly-sovacool commented Dec 6, 2023

slsevilla commented Dec 6, 2023

kelly-sovacool commented Dec 6, 2023

kelly-sovacool commented Dec 6, 2023 • edited Loading

kelly-sovacool commented Dec 6, 2023 • edited Loading

kelly-sovacool commented Dec 14, 2023 • edited Loading

kopardev commented Dec 14, 2023 • edited Loading

kelly-sovacool commented Dec 15, 2023

kopardev commented Dec 20, 2023

kelly-sovacool commented Dec 20, 2023

kelly-sovacool commented Jan 16, 2024

kopardev commented Jan 17, 2024

kelly-sovacool commented Jan 17, 2024

kopardev commented Jan 17, 2024

slsevilla commented Jan 17, 2024

kelly-sovacool commented Jan 17, 2024

kelly-sovacool commented Jan 17, 2024

kelly-sovacool commented Nov 21, 2023 •

edited

Loading

kelly-sovacool commented Dec 6, 2023 •

edited

Loading

kelly-sovacool commented Dec 6, 2023 •

edited

Loading

kelly-sovacool commented Dec 14, 2023 •

edited

Loading

kopardev commented Dec 14, 2023 •

edited

Loading