Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BAM output in ADAM appears to be corrupt #962

Closed
fnothaft opened this issue Feb 25, 2016 · 7 comments
Closed

BAM output in ADAM appears to be corrupt #962

fnothaft opened this issue Feb 25, 2016 · 7 comments
Assignees
@fnothaft
Labels
bug
Milestone
0.20.0

Comments

@fnothaft
Copy link
Member

Crosslinking to BD2KGenomics/toil-scripts#117.
@fnothaft fnothaft self-assigned this Feb 25, 2016
@fnothaft fnothaft mentioned this issue Feb 25, 2016
Closed
@heuermh heuermh modified the milestone: 0.19.1 Feb 25, 2016
@heuermh heuermh mentioned this issue Feb 25, 2016
Closed
@fnothaft
Copy link
Member Author

No dice on reading the per-partition BAM file:

~/adam$ samtools view part-r-00000
[E::hts_hopen] fail to open file 'part-r-00000' 
[E::hts_open_format] fail to open file 'part-r-00000' 
samtools view: failed to open "part-r-00000" for reading: Success

I don't think we write the BAM header with each partition, which might be scoobying this test. Let me change that and retry...

@fnothaft
Copy link
Member Author

OK, same thing if I write the BAM header as part of each partition. IDK...?

Time to go all stone ages and toLocalIterator this up...

@fnothaft
Copy link
Member Author

OK, have a WIP patch I am testing now at https://github.com/fnothaft/adam/tree/single-bam-localiter-ehf. So far, seems to work. Ugly as sin, but I'll clean it up later.

@fnothaft
Copy link
Member Author

The patch works but the performance is terrible because SAMRecordWritable seems to fall back on Java serialization. I'm working on a new patch... TBD.

@fnothaft
Copy link
Member Author

OK, got a working fix! PR is on the way shortly...

fnothaft added a commit to fnothaft/adam that referenced this issue Feb 26, 2016
@fnothaft
[ADAM-962] Fix corrupt single-file BAM output.
941da09 
It seems like we were doing something incorrectly when writing the header.
Additionally, we now write a correct end-of-file. Resolves bigdatagenomics#962.
fnothaft added a commit to fnothaft/adam that referenced this issue Feb 26, 2016
@fnothaft
[ADAM-962] Fix corrupt single-file BAM output.
f27ffe6 
It seems like we were doing something incorrectly when writing the header.
Additionally, we now write a correct end-of-file. Resolves bigdatagenomics#962.
@fnothaft fnothaft mentioned this issue Feb 26, 2016
Merged
fnothaft added a commit to fnothaft/adam that referenced this issue Feb 26, 2016
@fnothaft
[ADAM-962] Fix corrupt single-file BAM output.
551c559 
It seems like we were doing something incorrectly when writing the header.
Additionally, we now write a correct end-of-file. Resolves bigdatagenomics#962.
fnothaft added a commit to fnothaft/adam that referenced this issue Feb 26, 2016
@fnothaft
[ADAM-962] Fix corrupt single-file BAM output.
3687d7e 
It seems like we were doing something incorrectly when writing the header.
Additionally, we now write a correct end-of-file. Resolves bigdatagenomics#962.
fnothaft added a commit to fnothaft/adam that referenced this issue Feb 26, 2016
@fnothaft
[ADAM-962] Fix corrupt single-file BAM output.
46dde87 
It seems like we were doing something incorrectly when writing the header.
Additionally, we now write a correct end-of-file. Resolves bigdatagenomics#962.
fnothaft added a commit to fnothaft/cgl-docker-lib that referenced this issue Feb 26, 2016
@fnothaft
Hotfix for ADAM/BAM output issue.
8f4c8e0 
See:

* bigdatagenomics/adam#962
* BD2KGenomics/toil-scripts#117
@fnothaft fnothaft mentioned this issue Feb 26, 2016
Merged
@fnothaft fnothaft modified the milestones: 0.20.0, 0.19.1 May 18, 2016
@jpdna
Copy link
Member

jpdna commented Jun 6, 2016

Can this issue be closed?

@jpdna jpdna added the bug label Jun 6, 2016
@fnothaft
Copy link
Member Author

fnothaft commented Jun 6, 2016

No, the open PR still needs a fix.

@heuermh heuermh mentioned this issue Jun 7, 2016
Closed
61 tasks
fnothaft added a commit to fnothaft/adam that referenced this issue Jul 16, 2016
@fnothaft
[ADAM-962] Fix corrupt single-file BAM output.
75c64b4 
It seems like we were doing something incorrectly when writing the header.
Additionally, we now write a correct end-of-file. Resolves bigdatagenomics#962.
fnothaft added a commit to fnothaft/adam that referenced this issue Jul 19, 2016
@fnothaft
[ADAM-962] Fix corrupt single-file BAM output.
4de31aa 
It seems like we were doing something incorrectly when writing the header.
Additionally, we now write a correct end-of-file. Resolves bigdatagenomics#962.
cket pushed a commit to cket/cgl-docker-lib that referenced this issue Sep 7, 2016
@fnothaft
Hotfix for ADAM/BAM output issue.
bd1bc42 
See:

* bigdatagenomics/adam#962
* BD2KGenomics/toil-scripts#117
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fnothaft fnothaft

Labels
bug
Projects
None yet
Milestone
0.20.0
Development

No branches or pull requests
3 participants
@heuermh @fnothaft @jpdna