Error when running Platypus on multiple samples: [E::vcf_parse_format] Number of columns at chrXII:451539 does not match the number of samples (41 vs 49) #2414
Comments
|
Sorry about the issue and thanks for the report. The error message looks like something problematic is happening with Platypus on some lines of the VCF where it's not outputting all of the samples at some positions. It's strange that it doesn't happen consistently at the same position and I'm not sure exactly what is happening. My suggestion would be to debug just the platypus call, saving the output to a file and then see if you get well formed output with all positions having all of the samples. If you can reproduce here then we could try and figure out what is happening in producing it and think of a workaround. Hope this helps. |
|
OK, thank you for your help. I've rerun Platypus as you requested, on the sliced BAM files for the region chrXII_0_593953, and it seemed to work fine. I've attached the resulting log file and the VCF file, however nothing really stands out to me. There is a filtered variant present at position chrXII:451539, which is the one mentioned in the error that I saw in Bcbio's _debug.log file. There are also filtered variants at other posions (chrXII:449950 and chrXII:459798) which appeared in errors from prebious Bcbio runs, on the same BAM files. However, I can't find what sets these loci apart from the other entries in the VCF file, such that it could cause the previously mentioned errors. |
|
Apologies about dropping this thread and thank you for the debugging output. The VCF does look correct here, which indicates there is nothing fundamentally wrong with Platypus on these outputs. Given the inconsistent failure positions my guess is that your production runs are hitting some kind of memory issues resulting in truncated output, and you're not seeing this in the test environment with a single run. It's a bit hard to tell what process causes the truncation from what we have now, but my best suggestion would be to look more at the run environment: how are you distributing jobs (is this all on a single machine or using the IPython parallel support with a scheduler)? Is the machine running the jobs isolated from other processes that might also be using memory. Sorry to not have a definite clue about what is happening but hope this helps some with debugging. |
|
Sorry for replying so late to this. Yes, I was distributing jobs with IPython parallel on a heterogeneous cluster. The jobs were not isolated from other processes. |
|
Thanks for following up. My guess here is that other processes on your system are using up memory resulting in less available for Platypus and this leads to this inconsistent truncation. I don't have any great ideas on our end to work around this but have a couple of practical suggestions:
Hope one of these works for your needs to get your analysis finished. |
|
I managed to complete the previous analysis on a dedicated resource. I've recently started using GATK HaplotypeCaller and joint (GVCF) calling. I wanted to compare HaplotypeCaller with e.g. FreeBayes in joint calling mode, but there seems to be an issue with bcbio.variation.recall, which I've mentioned in a separate issue here. Thank you for your help and for all your work with Bcbio-nextgen! |
Hello,
I'm trying to run Platypus on multiple samples and I get an error in a specific chromosome range. I can't figure out why the error only appears in this region, while for other regions things seem to run fine. This is the error that I get:
[E::vcf_parse_format] Number of columns at chrXII:451539 does not match the number of samples (41 vs 49)This is the Bcbio template with the latest settings that I tried so far:
And these are the commands that appear to cause the error:
platypus callVariants --regions=/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/platypus/chrXII/batch1-chrXII_0_593953-regions.bed --bamFiles=/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-13N/P10-13N-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-1N/P10-1N-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10w_1/P10w_1-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/10Pswift/10Pswift-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-M1T/P10-M1T-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/4Pswift/4Pswift-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-M2/P10-M2-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-7/P10-7-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-4/P10-4-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-S3S/P10-S3S-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-1/P10-1-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-M6/P10-M6-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/8Hswift/8Hswift-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-10/P10-10-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-M3T/P10-M3T-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/2Hswift/2Hswift-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-2N/P10-2N-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/16Hswift/16Hswift-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-S2S/P10-S2S-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/13Pswift/13Pswift-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-7N/P10-7N-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-S1S/P10-S1S-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-M5/P10-M5-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-9N/P10-9N-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/1-12-F/1-12-F-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-2/P10-2-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-41/P10-41-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-20N/P10-20N-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-15N/P10-15N-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-M7T/P10-M7T-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/H2-M1/H2-M1-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-U-21/P10-U-21-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-L-21/P10-L-21-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-L-50/P10-L-50-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-U-50/P10-U-50-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10w_3/P10w_3-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/2-A-F/2-A-F-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-17N/P10-17N-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-16/P10-16-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10g/P10g-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-S5T/P10-S5T-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-19/P10-19-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-M4/P10-M4-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10w_2/P10w_2-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-S4T/P10-S4T-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-5N/P10-5N-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-39/P10-39-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-S6T/P10-S6T-prealign.bam,/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/prealign/P10-14N/P10-14N-prealign.bam --refFile=/mnt/storage-brno3-cerit/nfs4/home/amizeranschi/bcbio2/genomes/Scerevisiae/sacCer3/seq/sacCer3.fa --output=- --logFileName /dev/null --verbosity=1 --maxReads 500000000 --bufferSize 5000 --filterDuplicates=0 | awk -F$'\t' -v OFS='\t' '{if ($0 !~ /^#/) gsub(/[KMRYSWBVHDXkmryswbvhdx]/, "N", $4) } {print}' | awk -F$'\t' -v OFS='\t' '{if ($0 !~ /^#/) gsub(/[KMRYSWBVHDXkmryswbvhdx]/, "N", $5) } {print}' | grep -v ^##contig | bcftools annotate -h /mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/bcbiotx/tmpvlwcrs/batch1-chrXII_0_593953-contig_header.txt | vcfallelicprimitives -t DECOMPOSED --keep-geno | vcffixup - | vcfstreamsort | bgzip -c > /mnt/storage-brno3-cerit/nfs4/home/amizeranschi/allDNA-germline2/allDNA-germlineSV-variant/work/bcbiotx/tmpvlwcrs/batch1-chrXII_0_593953.vcf.gzThe errors seems to come from Bcftools, but I can't rerun the previous command entirely because the temp directory with the header file (batch1-chrXII_0_593953-contig_header.txt) has been removed.
I've run the pipeline multiple times and it fails consistently in more or less the same region:
[E::vcf_parse_format] Number of columns at chrXII:451539 does not match the number of samples (41 vs 49)[E::vcf_parse_format] Number of columns at chrXII:449950 does not match the number of samples (34 vs 49)[E::vcf_parse_format] Number of columns at chrXII:459798 does not match the number of samples (38 vs 49)I also remapped the reads entirely, to rule out a possible corrupted BAM file, but I still get the same error, in the same region.
What could be the cause of these errors?
The text was updated successfully, but these errors were encountered: