Building DAG of jobs... Creating conda environment envs/pybedtools.yaml... Downloading and installing remote packages. Environment for envs/pybedtools.yaml created (location: .snakemake/conda/565115efc65067de76d2e75e3dfc793c) Creating conda environment envs/biopython_py3.yaml... Downloading and installing remote packages. Environment for envs/biopython_py3.yaml created (location: .snakemake/conda/16694b516323f76886057daf0bcfb69c) Creating conda environment envs/core.yaml... Downloading and installing remote packages. Environment for envs/core.yaml created (location: .snakemake/conda/b2cf3cb0bb8978aae83afeb8d09ff3fa) Creating conda environment envs/R.yaml... Downloading and installing remote packages. Environment for envs/R.yaml created (location: .snakemake/conda/7ec541b42a8286a359c1b5ff6dcc2c15) Creating conda environment envs/core_py3.yaml... Downloading and installing remote packages. Environment for envs/core_py3.yaml created (location: .snakemake/conda/f16266b7c13b6623e8fef23c53c221a6) Using shell: /software/centos7/bin/bash Provided cores: 1 (use --cores to define parallelism) Rules claiming more threads will be scaled down. Job stats: job count min threads max threads ------------------------------- ------- ------------- ------------- GetPWM 1 1 1 Get_Genome 1 1 1 Get_ME_from_annotation 1 1 1 ME_SJ_coverage 2 1 1 ME_reads 2 1 1 Micro_Exon_Tags 1 1 1 Micro_Exon_table 1 1 1 Output 1 1 1 Round1_alingment_pre_processing 2 1 1 Round1_bwa_mem_to_tags 2 1 1 Round1_filter 2 1 1 Round2_alingment_pre_processing 2 1 1 Round2_bowtie_tags_index 1 1 1 Round2_bowtie_to_tags 2 1 1 Round2_filter 2 1 1 Splice_Junction_Library 1 1 1 Total_sample_exon_counts 1 1 1 bowtie_genome_index 1 1 1 bowtie_to_genome 2 1 1 bwa_index 1 1 1 coverage_filter 1 1 1 coverage_to_PSI 1 1 1 download_fastq 2 1 1 generate_fasta_from_bed12 1 1 1 high_confident_filters 1 1 1 hisat2_genome_index 1 1 1 hisat2_to_Genome 2 1 1 merge_ME_centric 1 1 1 merge_tags 1 1 1 quant 1 1 1 row_Micro_Exon_reads 2 1 1 write_ME_matches 1 1 1 total 44 1 1 Select jobs to execute... [Tue Oct 12 12:37:39 2021] rule Get_Genome: input: /home-3/mliu63@jhu.edu/scratch/hsref/hg38.fa output: data/Genome jobid: 32 priority: 100 resources: tmpdir=/tmp [Tue Oct 12 12:37:47 2021] Finished job 32. 1 of 44 steps (2%) done Select jobs to execute... [Tue Oct 12 12:37:47 2021] rule bowtie_genome_index: input: data/Genome output: data/Genome.1.ebwt jobid: 33 priority: 100 resources: tmpdir=/tmp Activating conda environment: /scratch/users/mliu63@jhu.edu/MicroExonator/MicroExonator/.snakemake/conda/b2cf3cb0bb8978aae83afeb8d09ff3fa [Tue Oct 12 15:49:50 2021] Finished job 33. 2 of 44 steps (5%) done Select jobs to execute... [Tue Oct 12 15:49:50 2021] rule hisat2_genome_index: input: /home-3/mliu63@jhu.edu/scratch/hsref/hg38.fa output: data/Genome.1.ht2 jobid: 16 resources: tmpdir=/tmp Activating conda environment: /scratch/users/mliu63@jhu.edu/MicroExonator/MicroExonator/.snakemake/conda/b2cf3cb0bb8978aae83afeb8d09ff3fa [Tue Oct 12 17:34:20 2021] Finished job 16. 3 of 44 steps (7%) done Select jobs to execute... [Tue Oct 12 17:34:20 2021] rule GetPWM: input: /home-3/mliu63@jhu.edu/scratch/hsref/hg38.fa, /home-3/mliu63@jhu.edu/scratch/hsref/gencode.v36.bed12 output: data/GT_AG_U2_5.pwm, data/GT_AG_U2_3.pwm jobid: 17 resources: tmpdir=/tmp Activating conda environment: /scratch/users/mliu63@jhu.edu/MicroExonator/MicroExonator/.snakemake/conda/16694b516323f76886057daf0bcfb69c [Tue Oct 12 17:35:47 2021] Finished job 17. 4 of 44 steps (9%) done Select jobs to execute... [Tue Oct 12 17:35:47 2021] rule generate_fasta_from_bed12: input: /home-3/mliu63@jhu.edu/scratch/hsref/hg38.fa, /home-3/mliu63@jhu.edu/scratch/hsref/gencode.v36.bed12 output: data/transcripts.fa jobid: 7 resources: tmpdir=/tmp Activating conda environment: /scratch/users/mliu63@jhu.edu/MicroExonator/MicroExonator/.snakemake/conda/565115efc65067de76d2e75e3dfc793c [Tue Oct 12 17:37:22 2021] Finished job 7. 5 of 44 steps (11%) done Select jobs to execute... [Tue Oct 12 17:37:22 2021] rule Splice_Junction_Library: input: /home-3/mliu63@jhu.edu/scratch/hsref/hg38.fa, data/transcripts.fa, /home-3/mliu63@jhu.edu/scratch/hsref/gencode.v36.bed12 output: Round1/ME_TAGs.fa jobid: 6 resources: tmpdir=/tmp Activating conda environment: /scratch/users/mliu63@jhu.edu/MicroExonator/MicroExonator/.snakemake/conda/b2cf3cb0bb8978aae83afeb8d09ff3fa [Tue Oct 12 17:39:38 2021] Finished job 6. 6 of 44 steps (14%) done Select jobs to execute... [Tue Oct 12 17:39:38 2021] rule bwa_index: input: Round1/ME_TAGs.fa output: Round1/ME_TAGs.fa.amb jobid: 14 resources: tmpdir=/tmp Activating conda environment: /scratch/users/mliu63@jhu.edu/MicroExonator/MicroExonator/.snakemake/conda/b2cf3cb0bb8978aae83afeb8d09ff3fa [Tue Oct 12 17:41:13 2021] Finished job 14. 7 of 44 steps (16%) done Select jobs to execute... [Tue Oct 12 17:41:13 2021] rule download_fastq: input: download/SRR10316370_1_val_1.download.sh output: FASTQ/SRR10316370_1_val_1.fastq.gz jobid: 22 wildcards: sample=SRR10316370_1_val_1 priority: -10 resources: tmpdir=/tmp, get_data=1 Activating conda environment: /scratch/users/mliu63@jhu.edu/MicroExonator/MicroExonator/.snakemake/conda/b2cf3cb0bb8978aae83afeb8d09ff3fa [Tue Oct 12 17:41:20 2021] Finished job 22. 8 of 44 steps (18%) done Select jobs to execute... [Tue Oct 12 17:41:20 2021] rule Round1_bwa_mem_to_tags: input: Round1/ME_TAGs.fa, FASTQ/SRR10316370_1_val_1.fastq.gz, Round1/ME_TAGs.fa.amb output: Round1/SRR10316370_1_val_1.sam jobid: 21 wildcards: sample=SRR10316370_1_val_1 priority: 100 resources: tmpdir=/tmp Activating conda environment: /scratch/users/mliu63@jhu.edu/MicroExonator/MicroExonator/.snakemake/conda/b2cf3cb0bb8978aae83afeb8d09ff3fa [Tue Oct 12 18:35:02 2021] Finished job 21. 9 of 44 steps (20%) done Select jobs to execute... [Tue Oct 12 18:35:02 2021] rule Round1_alingment_pre_processing: input: Round1/SRR10316370_1_val_1.sam output: Round1/SRR10316370_1_val_1.sam.pre_processed jobid: 20 wildcards: sample=SRR10316370_1_val_1 priority: 100 resources: tmpdir=/tmp Activating conda environment: /scratch/users/mliu63@jhu.edu/MicroExonator/MicroExonator/.snakemake/conda/b2cf3cb0bb8978aae83afeb8d09ff3fa Removing temporary output file Round1/SRR10316370_1_val_1.sam. [Tue Oct 12 18:35:57 2021] Finished job 20. 10 of 44 steps (23%) done Select jobs to execute... [Tue Oct 12 18:35:57 2021] rule row_Micro_Exon_reads: input: /home-3/mliu63@jhu.edu/scratch/hsref/hg38.fa, Round1/SRR10316370_1_val_1.sam.pre_processed, FASTQ/SRR10316370_1_val_1.fastq.gz output: Round1/SRR10316370_1_val_1.sam.row_ME, Round1/SRR10316370_1_val_1.sam.row_ME.fastq jobid: 19 wildcards: sample=SRR10316370_1_val_1 resources: tmpdir=/tmp Activating conda environment: /scratch/users/mliu63@jhu.edu/MicroExonator/MicroExonator/.snakemake/conda/b2cf3cb0bb8978aae83afeb8d09ff3fa Removing temporary output file Round1/SRR10316370_1_val_1.sam.pre_processed. [Tue Oct 12 18:54:15 2021] Finished job 19. 11 of 44 steps (25%) done Select jobs to execute... [Tue Oct 12 18:54:15 2021] rule hisat2_to_Genome: input: Round1/SRR10316370_1_val_1.sam.row_ME.fastq, data/Genome.1.ht2 output: Round1/SRR10316370_1_val_1.sam.row_ME.Genome.Aligned.out.sam jobid: 23 wildcards: sample=SRR10316370_1_val_1 resources: tmpdir=/tmp Activating conda environment: /scratch/users/mliu63@jhu.edu/MicroExonator/MicroExonator/.snakemake/conda/b2cf3cb0bb8978aae83afeb8d09ff3fa [Tue Oct 12 18:54:20 2021] Error in rule hisat2_to_Genome: jobid: 23 output: Round1/SRR10316370_1_val_1.sam.row_ME.Genome.Aligned.out.sam conda-env: /scratch/users/mliu63@jhu.edu/MicroExonator/MicroExonator/.snakemake/conda/b2cf3cb0bb8978aae83afeb8d09ff3fa shell: hisat2 -x data/Genome -U Round1/SRR10316370_1_val_1.sam.row_ME.fastq > Round1/SRR10316370_1_val_1.sam.row_ME.Genome.Aligned.out.sam (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!) Removing output files of failed job hisat2_to_Genome since they might be corrupted: Round1/SRR10316370_1_val_1.sam.row_ME.Genome.Aligned.out.sam output: Round1/SRR10316370_1_val_1.sam jobid: 21 wildcards: sample=SRR10316370_1_val_1 priority: 100 resources: tmpdir=/tmp Activating conda environment: /scratch/users/mliu63@jhu.edu/MicroExonator/MicroExonator/.snakemake/conda/b2cf3cb0bb8978aae83afeb8d09ff3fa [Tue Oct 12 18:35:02 2021] Finished job 21. 9 of 44 steps (20%) done Select jobs to execute... [Tue Oct 12 18:35:02 2021] rule Round1_alingment_pre_processing: input: Round1/SRR10316370_1_val_1.sam output: Round1/SRR10316370_1_val_1.sam.pre_processed jobid: 20 wildcards: sample=SRR10316370_1_val_1 priority: 100 resources: tmpdir=/tmp Activating conda environment: /scratch/users/mliu63@jhu.edu/MicroExonator/MicroExonator/.snakemake/conda/b2cf3cb0bb8978aae83afeb8d09ff3fa Removing temporary output file Round1/SRR10316370_1_val_1.sam. [Tue Oct 12 18:35:57 2021] Finished job 20. 10 of 44 steps (23%) done Select jobs to execute... [Tue Oct 12 18:35:57 2021] rule row_Micro_Exon_reads: input: /home-3/mliu63@jhu.edu/scratch/hsref/hg38.fa, Round1/SRR10316370_1_val_1.sam.pre_processed, FASTQ/SRR10316370_1_val_1.fastq.gz output: Round1/SRR10316370_1_val_1.sam.row_ME, Round1/SRR10316370_1_val_1.sam.row_ME.fastq jobid: 19 wildcards: sample=SRR10316370_1_val_1 resources: tmpdir=/tmp Activating conda environment: /scratch/users/mliu63@jhu.edu/MicroExonator/MicroExonator/.snakemake/conda/b2cf3cb0bb8978aae83afeb8d09ff3fa Removing temporary output file Round1/SRR10316370_1_val_1.sam.pre_processed. [Tue Oct 12 18:54:15 2021] Finished job 19. 11 of 44 steps (25%) done Select jobs to execute... [Tue Oct 12 18:54:15 2021] rule hisat2_to_Genome: input: Round1/SRR10316370_1_val_1.sam.row_ME.fastq, data/Genome.1.ht2 output: Round1/SRR10316370_1_val_1.sam.row_ME.Genome.Aligned.out.sam jobid: 23 wildcards: sample=SRR10316370_1_val_1 resources: tmpdir=/tmp Activating conda environment: /scratch/users/mliu63@jhu.edu/MicroExonator/MicroExonator/.snakemake/conda/b2cf3cb0bb8978aae83afeb8d09ff3fa [Tue Oct 12 18:54:20 2021] Error in rule hisat2_to_Genome: jobid: 23 output: Round1/SRR10316370_1_val_1.sam.row_ME.Genome.Aligned.out.sam conda-env: /scratch/users/mliu63@jhu.edu/MicroExonator/MicroExonator/.snakemake/conda/b2cf3cb0bb8978aae83afeb8d09ff3fa shell: hisat2 -x data/Genome -U Round1/SRR10316370_1_val_1.sam.row_ME.fastq > Round1/SRR10316370_1_val_1.sam.row_ME.Genome.Aligned.out.sam (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!) Removing output files of failed job hisat2_to_Genome since they might be corrupted: Round1/SRR10316370_1_val_1.sam.row_ME.Genome.Aligned.out.sam Job failed, going on with independent jobs. Select jobs to execute... [Tue Oct 12 18:54:20 2021] rule download_fastq: input: download/SRR10316370_2_val_2.download.sh output: FASTQ/SRR10316370_2_val_2.fastq.gz jobid: 13 wildcards: sample=SRR10316370_2_val_2 priority: -10 resources: tmpdir=/tmp, get_data=1 Activating conda environment: /scratch/users/mliu63@jhu.edu/MicroExonator/MicroExonator/.snakemake/conda/b2cf3cb0bb8978aae83afeb8d09ff3fa Waiting at most 5 seconds for missing files. MissingOutputException in line 33 of /scratch/users/mliu63@jhu.edu/MicroExonator/MicroExonator/rules/Get_data.smk: Job Missing files after 5 seconds: FASTQ/SRR10316370_2_val_2.fastq.gz This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait. Job id: 13 completed successfully, but some output files are missing. 13 Job failed, going on with independent jobs. Exiting because a job execution failed. Look above for error message Complete log: /scratch/users/mliu63@jhu.edu/MicroExonator/MicroExonator/.snakemake/log/2021-10-12T122557.847760.snakemake.log