Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to htslib 1.10.2 #184

Merged
merged 81 commits into from
Mar 17, 2020
Merged
Changes from 1 commit
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
d3db4fd
Pull 1.10.2 submodule. Initial pass on i32->i64 type conversion, will…
brainstorm Feb 3, 2020
bb34638
Fixes issue #109 while I'm at it, long hanging fruit fix
brainstorm Feb 3, 2020
1636ef5
Fix compression levels
juliangehring Feb 3, 2020
18e8e57
Merge pull request #1 from juliangehring/fix-compression
brainstorm Feb 3, 2020
9c2a6d6
Going through https://github.com/samtools/htslib/blob/develop/README.…
brainstorm Feb 4, 2020
115c634
Repoint submodule to upstream release tag
brainstorm Feb 4, 2020
6bf1760
Switch from rust-bio hosted htslib to upstream's samtools htslib.
brainstorm Feb 4, 2020
83c0656
Try to amend submodule
brainstorm Feb 4, 2020
2173a9e
Handle submodules manually on TravisCI, since it is failing now: http…
brainstorm Feb 4, 2020
d7c5355
Odd balls that would need review marked with XXX. New htslib seems to…
brainstorm Feb 4, 2020
dac8565
Add curl-sys crate for hplugin_curl, aws s3 et al
brainstorm Feb 4, 2020
5678f70
Add conditional libcurl compilation flags
brainstorm Feb 4, 2020
bad400b
Be nice with cargo fmt [ci skip]
brainstorm Feb 4, 2020
9e1a315
Try installing libcurl meta-package instead of the openssl dependent …
brainstorm Feb 4, 2020
4c9fddf
Does not work with libcurl-dev metapackage: https://travis-ci.org/rus…
brainstorm Feb 4, 2020
417b672
Add matrix compile support for TravisCI, I would like to know why it …
brainstorm Feb 4, 2020
22fae28
PPA source no longer needed for ubuntu bionic? [ci skip]
brainstorm Feb 4, 2020
1a19c69
Bionic packages htslib with gnutls instead of openssl: https://packag…
brainstorm Feb 4, 2020
6eb623a
sed is not gsed on OSX... not entirely sure this substitution is 100%…
brainstorm Feb 4, 2020
5c31eb8
Add libssl-dev to the mix for the openssl-sys: https://travis-ci.org/…
brainstorm Feb 4, 2020
d3eba71
Add build-essential in hopes to fix libcurl issue, incorporate https:…
brainstorm Feb 4, 2020
eb2ae94
Add curl on default features for hts-sys
brainstorm Feb 4, 2020
92df799
Point to the right htslib upstream repo, thanks @dlaehnemann [skip ci]
brainstorm Feb 5, 2020
4389e3f
Appease OSX musl-cross toolchain requirements: https://travis-ci.org/…
brainstorm Feb 8, 2020
4f64d69
TravisCI timesout installing brew deps: https://travis-ci.org/rust-bi…
brainstorm Feb 8, 2020
39010b1
Try workaround for https://twitter.com/braincode/status/1226103002550…
brainstorm Feb 8, 2020
9fe358b
Do not update homebrew (time intensive), switch from travis_wait to v…
brainstorm Feb 8, 2020
9d2f381
brew install --verbose is not verbose enough for travis, reverting tr…
brainstorm Feb 8, 2020
949499f
Explicitly install system libcurl on linux and osx
brainstorm Feb 8, 2020
850011e
This .cargo/config makes it test right locally: https://github.com/ru…
brainstorm Feb 8, 2020
db42104
Cargo.toml missing quote misshap
brainstorm Feb 8, 2020
89608c6
Same linker flags should apply for Linux builds? build-essential not …
brainstorm Feb 8, 2020
e534df6
tid is i64 now, CRAM vs BAM test l_data is off by 3 and m_data is off…
brainstorm Feb 9, 2020
bbe490f
There should be no need for travis_wait with -v
brainstorm Feb 10, 2020
a741700
Debugging htslib structs with @jkbonfield guidance...
brainstorm Feb 20, 2020
29747e5
Disable osx and musl-cross homebrew on the TravisCI matrix since it t…
brainstorm Feb 21, 2020
77d4765
Mystery solved for test_read_cram, testing some of the struct fields,…
brainstorm Feb 24, 2020
4f2c90c
Extract selective CRAM/BAM field comparison to test helper function
brainstorm Feb 24, 2020
6bd85e0
Formatting
brainstorm Feb 24, 2020
de35991
Temporarily commenting the hfile format category detection makes .bed…
brainstorm Feb 24, 2020
d89c216
Format detection seems to be broken upstream on htslib, see https://g…
brainstorm Feb 25, 2020
f5dd07f
Merge branch 'master' into htslib-1.10.2
brainstorm Feb 25, 2020
f6d3679
Revert to libcurl4-openssl-dev instead of gnutls. All TravisCI script…
brainstorm Feb 25, 2020
4ffb8fe
Merge github.com:brainstorm/rust-htslib into htslib-1.10.2
brainstorm Feb 25, 2020
6e92156
Bump up curl-sys version
brainstorm Feb 26, 2020
6c9454b
Experimenting musl cross-compilation with rustembedded cross. Ideally…
brainstorm Feb 26, 2020
642ceba
Remove assert_ne since cannot guarantee that l_data is different on a…
brainstorm Feb 26, 2020
7cf834f
Add minimum deps needed for rustembedded cross docker container
brainstorm Feb 26, 2020
e6d0131
Run docker in TravisCI, even if it is only supported on linux jobs :/
brainstorm Feb 26, 2020
c0dde9a
Add CFLAGS suggested by @nlhepler
brainstorm Feb 26, 2020
571b457
Separate musl from GNU toolchains on Rust cross
brainstorm Feb 27, 2020
b10c48f
Bump up bindgen, vendorize openssl, further experimentation with comp…
brainstorm Feb 27, 2020
81ab53c
Deprecate the mixed GNU/MUSL container
brainstorm Feb 27, 2020
7f87d97
Add minimal explanation on how to use the new docker files [ci skip]
brainstorm Feb 27, 2020
20f6fb1
cross 0.2.0 with openssl vendored. Progress but more htslib compile t…
brainstorm Feb 27, 2020
77d06f9
Make sure libzma is present in both containers [ci skip]
brainstorm Feb 27, 2020
47fee89
Focusing on LLVMv8 GNU cross build first, musl will come later on (if…
brainstorm Feb 28, 2020
78c41d2
Two more usize that should be u64
brainstorm Feb 28, 2020
6b79b72
Amend and settle BED header debate
brainstorm Feb 28, 2020
116c3dd
Rename test name accordingly
brainstorm Feb 28, 2020
f6f2f64
Transition to cross instead of cargo for builds and tests
brainstorm Feb 28, 2020
8127679
All GNU/clang/llvm8 builds and tests seem to succeed, transitioning t…
brainstorm Feb 28, 2020
2f14727
Remove stray (repeated) musl build
brainstorm Feb 28, 2020
0ca9400
Remove @filoSottile musl-cross homebrew formula since it takes too lo…
brainstorm Feb 28, 2020
d0837cf
Experiment with toolchain setup in https://gitlab.com/rust_musl_docke…
brainstorm Feb 28, 2020
d218b27
Remove build.rs flags noise and different tests, containers should ha…
brainstorm Feb 28, 2020
e293ad3
Indicate dependencies and build flags required for OSX [ci skip].
brainstorm Mar 1, 2020
ebd1d2e
enable musl builds
johanneskoester Mar 10, 2020
fcbbafc
add libclang
johanneskoester Mar 10, 2020
6882797
minor
johanneskoester Mar 10, 2020
e0d1edb
dbg
johanneskoester Mar 10, 2020
e97d47a
revert
johanneskoester Mar 10, 2020
cecbd4a
dbg
johanneskoester Mar 10, 2020
e065263
dbg
johanneskoester Mar 10, 2020
8505cfe
Revert cross/docker madness, bottled up musl-cross myself over here: …
brainstorm Mar 11, 2020
665a38e
Remove .travis.yml
brainstorm Mar 11, 2020
999fa0f
Merge branch 'htslib-1.10.2' of github.com:brainstorm/rust-htslib int…
brainstorm Mar 11, 2020
09aec34
Add in openssl to hts-sys
brainstorm Mar 13, 2020
6042c05
Make clippy boolean logic happy
brainstorm Mar 13, 2020
fc84cff
Remove homebrew OSX check since we'll just use rustembedded cross
brainstorm Mar 13, 2020
24f694f
Merge branch 'musl-action' of https://github.com/rust-bio/rust-htslib…
brainstorm Mar 13, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
…large_positions.md to spot 64 bit reference position changes and sam_* function renaming. Only 4 errors left, but need to closely check types afterwards very carefully. Specially for the binary formats where this change does not apply and (sub)fields that do not necessarily require big representations
brainstorm committed Feb 4, 2020

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit 9c2a6d686746b152ae91c699b74b81955f704597
36 changes: 18 additions & 18 deletions src/bam/ext.rs
Original file line number Diff line number Diff line change
@@ -21,30 +21,30 @@ pub trait BamRecordExtensions {
/// this happens on insertions.
///
/// pysam: blocks
fn aligned_blocks(&self) -> Vec<[i32; 2]>;
fn aligned_blocks(&self) -> Vec<[i64; 2]>;

/// find intron positions (start, stop)
///
/// This scans the CIGAR for reference skips
/// and reports their positions.
///
/// pysam: get_introns
fn introns(&self) -> Vec<[i32; 2]>;
fn introns(&self) -> Vec<[i64; 2]>;

/// a list of aligned read and reference positions
///
/// No entry for insertions, deletions or skipped pairs
///
/// pysam: get_aligned_pairs(matches_only = True)
fn aligned_pairs(&self) -> Vec<[i32; 2]>;
fn aligned_pairs(&self) -> Vec<[i64; 2]>;

/// a list of aligned read and reference positions
///
/// None in either the read positions or the reference position
/// for insertions, deletions or skipped pairs
///
/// pysam: aligned_pairs(matches_only = False)
fn aligned_pairs_full(&self) -> Vec<[Option<i32>; 2]>;
fn aligned_pairs_full(&self) -> Vec<[Option<i64>; 2]>;

/// the number of nucleotides covered by each Cigar:: possibility
///
@@ -67,19 +67,19 @@ pub trait BamRecordExtensions {
/// or unaligned positions within the read
///
/// pysam: get_reference_positions(full_length=False)
fn reference_positions(&self) -> Vec<i32>;
fn reference_positions(&self) -> Vec<i64>;
///
/// a Vec of reference positions that this read aligns to
///
/// include soft-clipped or skipped positions as None
///
/// pysam: get_reference_positions(full_length=True)
fn reference_positions_full(&self) -> Vec<Option<i32>>;
fn reference_positions_full(&self) -> Vec<Option<i64>>;

/// left most aligned reference position of the read on the reference genome.
fn reference_start(&self) -> i32;
fn reference_start(&self) -> i64;
/// right most aligned reference position of the read on the reference genome.
fn reference_end(&self) -> i32;
fn reference_end(&self) -> i64;

/// infer the query length from the cigar string, optionally include hard clipped bases
///
@@ -162,23 +162,23 @@ impl BamRecordExtensions for bam::Record {
for entry in self.cigar().iter() {
match entry {
Cigar::Match(len) | Cigar::Equal(len) | Cigar::Diff(len) => {
for i in pos..(pos + *len as i32) {
for i in pos..(pos + *len as i64) {
result.push([Some(qpos), Some(i)]);
qpos += 1;
}
pos += *len as i32;
pos += *len as i64;
}
Cigar::Ins(len) | Cigar::SoftClip(len) => {
for i in qpos..(qpos + *len as i32) {
for i in qpos..(qpos + *len as i64) {
result.push([Some(i), None]);
}
qpos += *len as i32;
qpos += *len as i64;
}
Cigar::Del(len) | Cigar::RefSkip(len) => {
for i in pos..(pos + *len as i32) {
for i in pos..(pos + *len as i64) {
result.push([None, Some(i)]);
}
pos += *len as i32;
pos += *len as i64;
}
Cigar::HardClip(_) => {} // no advance
Cigar::Pad(_) => panic!("Padding (Cigar::Pad) is not supported."), //padding is only used for multiple sequence alignment
@@ -4304,11 +4304,11 @@ mod tests {

//
//for the rest, we just verify that they have the expected amount of None in each position
fn some_count(pairs: &Vec<[Option<i32>; 2]>, pos: usize) -> i32 {
pairs.iter().filter(|x| x[pos].is_some()).count() as i32
fn some_count(pairs: &Vec<[Option<i64>; 2]>, pos: usize) -> i64 {
pairs.iter().filter(|x| x[pos].is_some()).count() as i64
}
fn none_count(pairs: &Vec<[Option<i32>; 2]>, pos: usize) -> i32 {
pairs.iter().filter(|x| x[pos].is_none()).count() as i32
fn none_count(pairs: &Vec<[Option<i64>; 2]>, pos: usize) -> i64 {
pairs.iter().filter(|x| x[pos].is_none()).count() as i64
}

let pairs = it.next().unwrap().unwrap().aligned_pairs_full();
6 changes: 3 additions & 3 deletions src/bam/mod.rs
Original file line number Diff line number Diff line change
@@ -839,7 +839,7 @@ impl HeaderView {

pub fn tid(&self, name: &[u8]) -> Option<u32> {
let c_str = ffi::CString::new(name).expect("Expected valid name.");
let tid = unsafe { htslib::bam_name2id(self.inner, c_str.as_ptr()) };
let tid = unsafe { htslib::sam_hdr_name2tid(self.inner, c_str.as_ptr()) };
if tid < 0 {
None
} else {
@@ -881,7 +881,7 @@ impl HeaderView {
impl Clone for HeaderView {
fn clone(&self) -> Self {
HeaderView {
inner: unsafe { htslib::bam_hdr_dup(self.inner) },
inner: unsafe { htslib::sam_hdr_dup(self.inner) },
owned: true,
}
}
@@ -891,7 +891,7 @@ impl Drop for HeaderView {
fn drop(&mut self) {
if self.owned {
unsafe {
htslib::bam_hdr_destroy(self.inner);
htslib::sam_hdr_destroy(self.inner);
}
}
}
8 changes: 4 additions & 4 deletions src/bam/record.rs
Original file line number Diff line number Diff line change
@@ -914,7 +914,7 @@ custom_derive! {

impl CigarString {
/// Create a `CigarStringView` from this CigarString at position `pos`
pub fn into_view(self, pos: i32) -> CigarStringView {
pub fn into_view(self, pos: i64) -> CigarStringView {
CigarStringView::new(self, pos)
}

@@ -1065,20 +1065,20 @@ pub struct CigarStringView {

impl CigarStringView {
/// Construct a new CigarStringView from a CigarString at a position
pub fn new(c: CigarString, pos: i32) -> CigarStringView {
pub fn new(c: CigarString, pos: i64) -> CigarStringView {
CigarStringView { inner: c, pos }
}

/// Get (exclusive) end position of alignment.
pub fn end_pos(&self) -> i32 {
pub fn end_pos(&self) -> i64 {
let mut pos = self.pos;
for c in self {
match c {
Cigar::Match(l)
| Cigar::RefSkip(l)
| Cigar::Del(l)
| Cigar::Equal(l)
| Cigar::Diff(l) => pos += *l as i32,
| Cigar::Diff(l) => pos += *l as i64,
// these don't add to end_pos on reference
Cigar::Ins(_) | Cigar::SoftClip(_) | Cigar::HardClip(_) | Cigar::Pad(_) => (),
}
2 changes: 1 addition & 1 deletion src/bcf/mod.rs
Original file line number Diff line number Diff line change
@@ -443,7 +443,7 @@ pub mod synced {
let record = *(*(*self.inner).readers.offset(idx as isize))
.buffer
.offset(0);
if (*record).rid != (rid as i64) || (*record).pos >= (end as i64) {
if (*record).rid != (rid as i32) || (*record).pos >= (end as i32) {
return Ok(0);
}
}