anvi-profile improvements #1362

ekiefl · 2020-02-26T22:19:40Z

This is a big change to anvi'o so I wanted to open up the pull request early to get critique from others early on throughtout the process.

As of this post, I am currently focused on developing tests that compare results to v6.1

Instructions to access the tests run below

Download and unzip this: testing.zip
Set your anvio repository to a

a=/Users/evan/Software/anvio

Your working directory should look like

SNV-IMPROVEMENTS/
├── test_SKIP_SNV.py
├── test_SNV.py
├── test_SCV.py
├── run.sh
├── ANE_004_05M.bam
├── ANE_004_05M.bam.bai
├── 05_CONTIGS.db
└── with.sh

Fire away

./run.sh

class fetches all its attributes from self.read to mimick the AlignedSegment class

messy, add test upon instantiation

successful; aligned_sequence matches with reference sequence

return a pysam.AlignmentFile object. No more get() method and no more run and progress attributes :(

except trims reads if they overhang the start and start

- stop range

returns a pysam.AlignmentFile object

positions as additional data to ProcessAlleleCounts and it filters them alongside its normal operations

contig in one function call. For a contig of length 1M, this info is retreived in 0.03s, compared 4s if calling get_nt_info for each position

ekiefl · 2020-03-04T17:00:46Z

The above solves SNV calculations from the memory leak, but it does not save the SCVs routine since the routine calls other numba-ized functions. After digging in, I discovered that it was caused by a numba generators in utils, so I made this reproducible example in the numba repo: numba/numba#5350

For now, I will avoid the use of numba generators :\

0.48.0

…umba/numba#5350

write buffer size should depend on the number of threads, since typically if you have more threads you have more memory. I am noticing this because gc.collect() enables the GIL, which creates very large stalls when using mutliple threads in anvi-profile.

arguments to be passed to bamops.BAMFileObject

someone complains

ekiefl · 2020-03-05T01:03:49Z

@meren, I have reached the point where I am just adding emojis to the codebase (2d326b6). I think this means I have nothing more to do here. I'm not sure how much you have reviewed this branch, but at this point I'll wait for you to review as much as you can. If everything is good I want to merge tomorrow :S

meren · 2020-03-05T03:26:02Z

I say go for it and merge, Evan, and we all play with it for a while.

convention"

completion_score if we want

ekiefl added 30 commits February 26, 2020 16:09

Handles reference_positions too. Create __getattr__ method so that the

4462a5f

class fetches all its attributes from self.read to mimick the AlignedSegment class

Add get_aligned_sequence, remove fancy __getattr__ bc it was getting

c56f08e

messy, add test upon instantiation

Looking beautiful as ever

06bd7de

major checkpoint commit: demonstrates the that trimming reads is

6aba9e1

successful; aligned_sequence matches with reference sequence

Make a lil process my variability please class

b9bdda9

Change to bamops.BAMFileObject

f0b27df

bamops.BAMFileObject now inherits from pysam.AlignmentFile, rather than

c896e85

return a pysam.AlignmentFile object. No more get() method and no more run and progress attributes :(

BAMFileObject gets new iterator fetch_and_trim, that acts like fetch,

90ef5e3

except trims reads if they overhang the start and start

Utilize new fetch_and_trim in Auxiliary

25c0e06

Add *args and **kwargs to function call

c558f47

Get rid of pileup method--there's never a good reason to use it

6cb037f

Add get_blocks() as method to sequence.Read

93c3844

docstring

769193a

Coverage now will not break if there is a read that overhangs the start

8051867

- stop range

overwrite reference_start and reference_end upon trim()

75640e3

get() function no longer exists, since BAMFileObject inherits instead of

d17eed3

returns a pysam.AlignmentFile object

Add gen_report() to Timer class

9074cb0

Add timer statistics for each contig if --debug

e534107

The beginning of the end for the old way of calculating variability

9bcf0cf

Requesting documentation

7a274ec

that feels amazing

772adfc

Broken state

2f5885a

Wrapper classes for each engine type

ea68551

Super quick docstring

6363be7

make wrapper classes more readable

2340441

Add docstring to ProcessAlleleCounts

1cb0c98

refactor processallelecounts to accomodate an initial data dict

f09abd3

compatible with new ProcessAlleleCounts

6af683f

ProcessAlleleCounts is a pretty powerful class. Here we pass outlier

2a7e1fc

positions as additional data to ProcessAlleleCounts and it filters them alongside its normal operations

new ContigsSuperClass function returns nt info for ALL positions in a

2615512

contig in one function call. For a contig of length 1M, this info is retreived in 0.03s, compared 4s if calling get_nt_info for each position

ekiefl added 2 commits March 3, 2020 21:09

Remove the cache round 2

b06ce10

add gc.collect() to multithread

f27645c

ekiefl added 12 commits March 4, 2020 11:55

Remove get_blocks. No one is using it and it had a memory leak in numba

fae5514

0.48.0

No need for sass

b14b5be

Convert numba generator to numba function to avoid memory leak: see n…

aecacd5

…umba/numba#5350

Convert numba generator to a function

1d25bf1

Replace pysam.Samfile with bamops.BAMFileObject, allow additional

e380fd0

arguments to be passed to bamops.BAMFileObject

typos

10eec4c

Remove legacy "ignore_orphans". One day we will bring it back when

48be7ed

someone complains

No more bam option for get-codon-frequencies

da61be4

Update some comments; Add FIXME to Linkers class

95bdde0

We were silently clipping off the last position...

2de9540

Add update when writing to DB. Add some emojis.

2d326b6

ekiefl and others added 9 commits March 4, 2020 22:33

change to floppy disk -- real important stuff here

c71b658

Reformat all the terminal docstrings with what I will call a "spreading

884653b

convention"

completion score is really just any number. We can use it as

07dbdbd

completion_score if we want

Add option to report raw memory

87bffd1

A little memory tracker :))

624981d

Utilize new TrackMemory class

54d6d78

Let user know they are writing to DB at this time

ddfaec7

Mo powa

475e90b

Merge branch 'master' into snv-improvements

29dbabe

ekiefl merged commit 107ae4c into master Mar 5, 2020

meren added a commit that referenced this pull request Mar 5, 2020

the new numba requirement related to #1362

5c9c32f

ekiefl mentioned this pull request Mar 6, 2020

Anvi-Profile on .bam files taking long time #1343

Closed

meren deleted the snv-improvements branch July 31, 2020 20:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

anvi-profile improvements #1362

anvi-profile improvements #1362

ekiefl commented Feb 26, 2020 •

edited

Loading

ekiefl commented Mar 4, 2020 •

edited

Loading

ekiefl commented Mar 5, 2020

meren commented Mar 5, 2020

anvi-profile improvements #1362

anvi-profile improvements #1362

Conversation

ekiefl commented Feb 26, 2020 • edited Loading

Instructions to access the tests run below

ekiefl commented Mar 4, 2020 • edited Loading

ekiefl commented Mar 5, 2020

meren commented Mar 5, 2020

ekiefl commented Feb 26, 2020 •

edited

Loading

ekiefl commented Mar 4, 2020 •

edited

Loading