Reading bam from s3

[wdecoster]

Hi,

Based on the documentation I have the impression it should be possible to read bam files from s3, but I cannot figure out how to do it. My cargo.toml contains as dependency rust-htslib = { version = "0.44.1", features = ["s3"] } I am trying to read the publicly available bam file here, with two variations of the url, but in both cases I get the same error (see below)

// let bam = String::from("https://s3.amazonaws.com/1000g-ont/aligned_data_minimap2_2.24/HG01312/aligned_bams/HG01312.STD_eee-prom1_guppy-6.3.7-sup-prom_fastq_pass.phased.bam");
let bam = String::from("s3://1000g-ont/aligned_data_minimap2_2.24/HG01312/aligned_bams/HG01312.STD_eee-prom1_guppy-6.3.7-sup-prom_fastq_pass.phased.bam");
let mut bam_reader = bam::IndexedReader::from_url(&Url::parse(bam).expect("Failed to parse s3 URL"))
            .unwrap_or_else(|err| panic!("Error opening remote BAM: {err}"))

[E::easy_errno] Libcurl reported error 60 (SSL peer certificate or SSH remote key was not OK)
[E::hts_open_format] Failed to open file "s3://1000g-ont/aligned_data_minimap2_2.24/HG01312/aligned_bams/HG01312.STD_eee-prom1_guppy-6.3.7-sup-prom_fastq_pass.phased.bam" : Input/output error
thread 'call::tests::test_genotype_repeat_s3' panicked at 'Error opening remote BAM: unable to open SAM/BAM/CRAM file at s3://1000g-ont/aligned_data_minimap2_2.24/HG01312/aligned_bams/HG01312.STD_eee-prom1_guppy-6.3.7-sup-prom_fastq_pass.phased.bam', src/call.rs:242:35

What am I doing wrong?

[wdecoster]

To add to this: wget and samtools view work for the https link, but not through Rust with the code above.

[kvg]

I'm not a rust-htslib developer, but I had a very similar problem. I was trying to read a BAM file in Google Cloud Storage, and while samtools worked properly, both pysam and rust-htslib failed with libcurl error 77 and 60, respectively. Notably, all three work on my laptop; the failure only manifests in a Jupyter notebook running in a cloud environment (https://app.terra.bio/).

The fix for me was setting the CURL_CA_BUNDLE environment variable that allows htslib to override cURL's certificate bundle path (samtools/htslib#599):

export CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

This solved both my pysam and rust-htslib errors simultaneously. I don't quite understand why samtools's htslib doesn't need this to be specified but pysam's and rust-htslib do, but at least it works now. Hopefully my issue is the same as yours, @wdecoster .

[wdecoster]

Oh damn, that worked! That is great! Thank you so much!

I have changed my code to include the following before bam::IndexedReader::from_url():

if env::var("CURL_CA_BUNDLE").is_err() {
    env::set_var("CURL_CA_BUNDLE", "/etc/ssl/certs/ca-certificates.crt");
}