LoFreq 2.4 indelqual filter error #89
Comments
|
That sounds serious. Can you reproduce this and share logging messages and possibly even a minimal working example / slice of the BAM file? Thanks |
|
I'll try to reproduce this error and post an update. |
|
I've re-installed LoFreq 2.1.4 to the separate conda environment. The error message: If I call variants without filtering then my VCF look like: The toy example - BAM file with 50000nt slice from the alignment: The VCF above was produced from it. The error was produced from the full alignment. |
|
Oh something is very wrong. I will be able to look at this end of next week |
|
Two quick questions:
And thanks for sharing those files! |
|
|
Hm....I can't reproduce this neither using the compiled version nor the conda version. But there is another user with the same problem. Could you help me a bit more please? What is the output of:
Many thanks! |
|
Results: |
|
Thanks! Could you post the contents of /tmp/lofreq2-call-dyn-bonf.V0eDSv please? |
|
|
Weird (but again: confirmed by one other user)...Which platform/distribution are you on? |
|
I'm using CentOS 7. |
|
Sorry, I got to admit I'm still totally in the dark here. Is you setup unusual in any way? What does You don't have this running on a cloud instance to which you could grant me temporary access, do you? Thanks! |
|
I'm not sure if it is unusual. It is quite old lab server (started 4 years ago) with CentOS 7 updated to the last version. I think it has a lot of things historically happened here. The result of the command is: And yes - I don't have a cloud version of the machine. |
|
Hi all, |
* Issue reported on CSB5/lofreq#89
|
Thanks for reporting. We now have a handful of users with the same problem. Unfortunately I cannot reproduce this. Could you tell me as much as possible about your system, e.g. OS name and version, LoFreq installation method (source of conda) etc? Many thanks |
|
I also have this issue on v2.1.4. (Centos 7). Just following up on this to say that I created a Docker container building the same version (2.1.4) from source in a centos7 environment and this does not have the bug, so it seems that there may be something about the conda version that is not working correctly. |
|
Thanks for confirming! Did you by any chance try to install from source as well? @sleyn, could you be so kind and post your conda env details here? |
|
@andreas-wilm sure. My environment for lofreq24. |
|
Thanks @sleyn. Could I ask you to do an experiment? Could update htslib in your LoFreq conda env to version 1.10 and see whether you still experience problems? @tonbar, I noticed you don't have htslib in your LoFreq conda env. That's surprising. Anyway, could you try to install htslib version 1.10 in there and rerun LoFreq to see whether the problem persists? Many thanks to both of you |
|
For some reason I can't install 1.10 in conda... Happens even in the clean environment. I did not find a solution except one advice to downgrade conda, that I don't like to do. |
|
I used conda to install LoFreq, and I am running it on a CentOS 7. Trying to install htslib 1.10, I get the following error: It seems to me the LoFreq bioconda recipe needs to be updated to allow a higher version of htslib. @andreas-wilm What do you think? |
|
Apologies, in my haste I shared the incorrect stripped down env that would allow installation. Here is what gets installed. |
|
Hm yeah. The only question is why I cant reproduce it. Did you try to
install from source and did that work?
…On Fri, 24 Apr 2020, 16:46 Susana Posada Céspedes, ***@***.***> wrote:
I used conda to install LoFreq, and I am running it on a CentOS 7.
Here is the environement:
channels:
- bioconda
- conda-forge
- defaults
- r
dependencies:
- _libgcc_mutex=0.1=conda_forge
- _openmp_mutex=4.5=1_llvm
- bcftools=1.9=h68d8f2e_9
- bzip2=1.0.8=h516909a_2
- ca-certificates=2020.4.5.1=hecc5488_0
- certifi=2020.4.5.1=py37hc8dfbb8_0
- curl=7.69.1=h33f0ec9_0
- gsl=2.5=h294904e_1
- htslib=1.9=h4da6232_3
- krb5=1.17.1=h2fd8d38_0
- ld_impl_linux-64=2.34=h53a641e_0
- libblas=3.8.0=16_openblas
- libcblas=3.8.0=16_openblas
- libcurl=7.69.1=hf7181ac_0
- libdeflate=1.5=h516909a_0
- libedit=3.1.20170329=hf8c457e_1001
- libffi=3.2.1=he1b5a44_1007
- libgcc-ng=9.2.0=h24d8f2e_2
- libgfortran-ng=7.3.0=hdf63c60_5
- libopenblas=0.3.9=h5ec1e0e_0
- libssh2=1.8.2=h22169c7_2
- libstdcxx-ng=9.2.0=hdf63c60_2
- llvm-openmp=10.0.0=hc9558a2_0
- lofreq=2.1.4=py37hc3dfafe_1
- ncurses=6.1=hf484d3e_1002
- openssl=1.1.1f=h516909a_0
- perl=5.26.2=h516909a_1006
- pip=20.0.2=py_2
- python=3.7.6=h8356626_5_cpython
- python_abi=3.7=1_cp37m
- readline=8.0=hf8c457e_0
- samtools=1.9=h10a08f8_12
- setuptools=46.1.3=py37hc8dfbb8_0
- sqlite=3.30.1=hcee41ef_0
- tk=8.6.10=hed695b0_0
- wheel=0.34.2=py_1
- xz=5.2.5=h516909a_0
- zlib=1.2.11=h516909a_1006
Trying to install htslib 1.10, I get the following error:
Package htslib conflicts for:
lofreq=2.1.4 -> htslib[version='>=1.9,<1.10.0a0']
samtools -> htslib[version='>=1.10,<1.11.0a0|>=1.10.2,<1.11.0a0|>=1.9,<1.10.0a0']
bcftools -> htslib[version='>=1.10,<1.11.0a0|>=1.10.1,<1.11.0a0|>=1.9,<1.10.0a0']
htslib=1.10
It seems to me the LoFreq bioconda recipe needs to be updated to allow a
higher version of htslib. @andreas-wilm <https://github.com/andreas-wilm>
What do you think?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#89 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAILSCJ4LYITJICBAMRHMOTROFGXZANCNFSM4KX674RA>
.
|
|
When I was trying out a few things, I used both htslib 1.10 and 1.9 in the DockerFile I created and neither had this bug. (Dockerfile building from source). |
|
Thanks a lot. So this is likely a problem with the conda build then, but
independent of the htslib version. It's still puzzling to me, why I can't
reproduce the problem
…On Fri, 24 Apr 2020, 17:20 S Rey, ***@***.***> wrote:
When I was trying out a few things, I used both htslib 1.10 and 1.9 in the
DockerFile I created and neither had this bug.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#89 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAILSCKG3H2SY52SGJPV743ROFKV3ANCNFSM4KX674RA>
.
|
|
Dear all, a new LoFreq recipe appeared on Bioconda. It seems that with this new recipe it's now possible to upgrade the htslib version. Please install LoFreq version 2.1.4 and upgrade htslib to 1.10 if it's not immediately pulled in (e.g. Andreas |
|
Same issue here. Funny thing is that running Biocontainer version: 2.1.4--py37ha77fa96_2 (from |
|
Thanks Roberto. I wish we had a better idea about what's going on here.
…On Sun, 31 May 2020, 03:15 Roberto Spreafico, ***@***.***> wrote:
Same issue here. Funny thing is that running lofreq from a conda env on
Mac with htslib v1.10 works fine. Running the docker container
(Biocontainer) matching the latest lofreq recipe, which also installs
samtools and htslib (1.10) in the same docker container, doesn't.
Biocontainer version: 2.1.4--py37ha77fa96_2 (from
https://quay.io/repository/biocontainers/lofreq?tab=tags)
samtools/htslib version in Biocontainer: Version: 1.10 (using htslib
1.10.2)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#89 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAILSCP64GPNX2SCPNNGSTLRUFLL3ANCNFSM4KX674RA>
.
|
|
The pain is real: with --enable-debug the OSXbuilds do not finish, so deploying the binary with this option needs more hacks and different compile arguments on different archs. |
|
That's almost good news, because it gives me a handle. I'll look into this ASAP and will report back here |
|
The failing OSX build looks like a separate issues though: |
|
Yes, but please note that this only occurs with --enable-debug |
|
Okay, that compilation error was easily fixed. |
|
|
|
Quick observation: as noted above the faulty version segfaults, because the intermediately produced vcf files has incomplete INFO columns, but only at indel positions, e.g. Hence the subsequent filter command fails. The above was found with biocontainers/lofreq:2.1.4--py37h3a01142_3 |
|
Running a locally compiled version ( @bgruening, sorry beginner question: what's the easiest for me to build an image that contains the faulty setup and also some debug tools, like valgrind? |
|
Not that easy, those containers are super minimal and don't have anything included. What do you need? Maybe I can build you one? Maybe you can mount in Valgrind as binary? |
|
Getting valgrind as (compatible) binary is the problem. It would obviously be awesome, if you could build an image with valgrind. I still don't understand why my conda installed LoFreq version simply works... |
|
If its a wired buffer-overflow this can be anything and really dependent on the system, no? |
|
Depends. But without valgrind and -g -O0 compiled code it's going to be hard to find out. Do I understand correctly, that your committed changes fix the problem? One approach would be to publish it and find the bug afterwards (if it's a LoFreq bug) |
|
I don't think it fixes it. I think it hides it behind the optimization flag. This smells like memory allocation issues. |
|
The code that's responsible for the vcf formatting hasn't changed in five years... |
|
New container with valgrind is here: curl "https://112272-42372094-gh.circle-artifacts.com/0/tmp/artifacts/images/lofreq:2.1.4--py37h3a01142_1000.tar.gz" | gzip -dc | docker load |
|
|
New container build with: And it still fails: |
- Refer to #89 - Also fixed some outdated asserts
|
Thanks for the hard work @bgruening. I think I found the cause: ages old and shoddily written code with undefined results. That explains why the problem sometimes magically appear. It's committed as #8206db2. Any chance you can try so that I can bump the version? Andreas |
|
This crash was caused by bad effects of the |
|
Thanks again John. Your changes would have prevented the execution from failing. Info fields for indels in the vcf would still be truncated though. My fix (8206db2 or more exactly 1e44677) hopefully addresses this. Will merge your PR once it's confirmed, that the vcf is no longer truncated. Never managed to reproduce the problem at my end, so can't tell whether the fix actually works and therefore rely on someone's help. Pretty please @bgruening |
|
On it ;) |
|
🥇 seems to work! If you release a new version I will update the conda package and container. |
|
Super cool. Thanks so much @bgruening. And many thanks to everyone else as well! God, I love this stuff I should be able to push a new version over the weekend. |
|
Dear all, I just pushed 2.1.5. As far as we know this should address the issue. Thanks so much everyone for your help! |
|
Thanks! New bioconda package is out. You might want to update your blog :) |
Hi,
It is the continuation of my previous BI/DB issue.
I've tried to do
lofreq indelqualwith the new 2.4 version on my BAM file. The subsequentlofreq call-parallelfailed saying something about error on the filtering command (sorry - I did not copy the exact error message). With--no-default-filterparameter I found that in VCF files indels had only HRUN field, but no DP,AF,SB and DP4.I've tried to look in BAM file, but for me it looked good - BI/BD tags were added.
With the LoFreq 2.3.1 everything works perfect. So I'll stay with this version for now.
The text was updated successfully, but these errors were encountered: