Add --max-memory-usage argument, and associated khmer_args refactoring. #1050

ctb · 2015-05-31T12:36:11Z

This pull request should make little practical difference in behavior, but does several major refactors:

first, we introduce -M/--max-memory-usage which is a single parameter to automatically set
graph sizes;
second, as part of this, we change the way that hashtable sizes are calculated so that prime numbers
below the specified number are chosen. This necessitated many small changes to tests that
relied on specific behavior and numbers.
third, I made CountingHash take the same args as Hashbits, and put in the same new hack in
Python-land to subclass the CPython class _CountingHash;
fourth, while we were changing so many things anyway, I went ahead and replaced
new_counting_hash and new_hashbits with CountingHash and Hashbits.
fifth, since test_hashbits and test_hashbits_obj now contained basically the same tests, I merged them
and then removed test_hashbits_obj.

and of course many minor adjustments throughout.

ctb · 2015-05-31T12:37:10Z

Fixes #980 in passing.

ctb · 2015-05-31T12:43:12Z

This is one for @mr-c to gaze upon. So far, I am happy with the direction it is taking.

…args2 Conflicts: ChangeLog scripts/normalize-by-median.py

ctb · 2015-06-05T01:38:48Z

I think I will wait on @mr-c to take a look-see before continuing with fixing up the documentation etc. etc.

…args2 Conflicts: ChangeLog khmer/__init__.py khmer/khmer_args.py oxli/build_graph.py sandbox/collect-reads.py sandbox/saturate-by-median.py scripts/abundance-dist-single.py scripts/do-partition.py scripts/filter-abund-single.py scripts/load-into-counting.py scripts/normalize-by-median.py scripts/trim-low-abund.py tests/test_script_arguments.py

mr-c · 2015-06-14T08:23:25Z

ChangeLog

+   counting hash/hashbits creation with new khmer_args create* functions.
+   * oxli/build_graph.py,sandbox/{sweep-files.py,sweep-reads.py},
+   scripts/{count-overlap.py,do-partition.py,},khmer/khmer_args.py:
+   changed hashtype over to 'nodegraph' and 'bitgraph' in call to


s/bitgraph/countgraph/

…args2 Conflicts: ChangeLog

ctb · 2015-06-14T21:32:42Z

ARGH this one is going down the rabbit hole... fine, I'll just do it all, then.

* make CountingHash a Python wrapper around _CountingHash; * replace new_counting_hash with CountingHash through codebase; * replace new_hashbits with Hashbits throughout codebase; Still some tests failing, though.

mr-c · 2015-06-15T19:55:24Z

Well, all of these inter-related issues provide a lot of value and are a long time coming :-)

Conflicts: khmer/__init__.py tests/test_counting_hash.py tests/test_hashbits_obj.py

ctb · 2015-06-23T14:48:47Z

test coverage issues fixed.

ctb · 2015-06-24T16:43:57Z

test this please, jenkins.

SensibleSalmon · 2015-06-25T17:16:13Z

retest this, please

ctb · 2015-06-26T14:39:33Z

Ready for review and merge @mr-c.

ctb · 2015-06-26T14:41:07Z

Note: we will probably need to redo things in doc/user/scripts.rst and choosing-hash-sizes a bit once #1106 lands; please evaluate current doc updates in this light :)

mr-c · 2015-06-27T00:14:37Z

doc/whats-new-2.0.rst

+New parameter for tablesize/number of table parameters.
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There is now a `-M/--max-memory-usage` parameter that sets the number


Should use the style

:option:`-M`/:option:`--max-memory-usage`

throughout this document.

mr-c · 2015-06-27T00:20:08Z

Great! I'm going to fix the two small nits I found and merge for you.

ctb · 2015-06-27T00:21:38Z

fixed the doc one

On Fri, Jun 26, 2015 at 05:20:09PM -0700, Michael R. Crusoe wrote:

Great! I'm going to fix the two small nits I found and merge for you.

Reply to this email directly or view it on GitHub:

#1050 (comment)

C. Titus Brown, ctbrown@ucdavis.edu

mr-c · 2015-06-27T00:27:33Z

sigh Thanks :-P

On Fri, Jun 26, 2015 at 5:21 PM C. Titus Brown notifications@github.com
wrote:

fixed the doc one

On Fri, Jun 26, 2015 at 05:20:09PM -0700, Michael R. Crusoe wrote:

Great! I'm going to fix the two small nits I found and merge for you.

Reply to this email directly or view it on GitHub:

#1050 (comment)

C. Titus Brown, ctbrown@ucdavis.edu

—
Reply to this email directly or view it on GitHub
#1050 (comment).

Michael R. Crusoe: Programmer & Bioinformatician crusoe@ucdavis.edu
The lab for Data Intensive Biology; University of California, Davis
https://impactstory.org/MichaelRCrusoe http://twitter.com/biocrusoe

Add --max-memory-usage argument, and associated khmer_args refactoring.

ctb · 2015-06-27T00:29:13Z

lol

On Fri, Jun 26, 2015 at 05:27:33PM -0700, Michael R. Crusoe wrote:

sigh Thanks :-P

On Fri, Jun 26, 2015 at 5:21 PM C. Titus Brown notifications@github.com
wrote:

fixed the doc one

On Fri, Jun 26, 2015 at 05:20:09PM -0700, Michael R. Crusoe wrote:

Great! I'm going to fix the two small nits I found and merge for you.

Reply to this email directly or view it on GitHub:

#1050 (comment)

C. Titus Brown, ctbrown@ucdavis.edu

???
Reply to this email directly or view it on GitHub
#1050 (comment).

Michael R. Crusoe: Programmer & Bioinformatician crusoe@ucdavis.edu
The lab for Data Intensive Biology; University of California, Davis
https://impactstory.org/MichaelRCrusoe http://twitter.com/biocrusoe

Reply to this email directly or view it on GitHub:

#1050 (comment)

C. Titus Brown, ctbrown@ucdavis.edu

ctb · 2015-06-27T00:29:26Z

great, thanks! cc @bocajnotnef

ctb added 6 commits May 31, 2015 07:14

update explicit hashtype types to nodegraph and bitgraph

380e167

removed environment variable override for hash size defaults

4f89e31

pass IOError through extract_*_functions

2e8984c

fixed FP rate reporting

157fd85

added --max-memory-usage and create_nodegraph/create_countgraph fns

7bf9b43

updated ChangeLog

3a98a99

fix pep8

5883b5f

ctb added 3 commits May 31, 2015 10:36

replaced new_* hash creation functions with khmer_args.create*

2d398fd

fixed check_space_for_hashtable

a0e6b01

updated sandbox files, too

bce7b0a

ctb force-pushed the update/memory-args2 branch from 98da767 to bce7b0a Compare May 31, 2015 15:07

fix up oxli build_graph as well

dbb081d

ctb added this to the 2.0 milestone May 31, 2015

ctb added 2 commits June 4, 2015 21:34

removed redundant line

f30b2ba

Merge branch 'master' of github.com:ged-lab/khmer into update/memory-…

596c5f2

…args2 Conflicts: ChangeLog scripts/normalize-by-median.py

This was referenced Jun 13, 2015

Choosing hash table size and numbers according to number of unique k-mers #347

Closed

Complain on graph creation when k-mer size requested is > 32 #1094

Closed

mr-c reviewed Jun 14, 2015
View reviewed changes

Merge branch 'master' of github.com:dib-lab/khmer into update/memory-…

d7b4b5f

…args2 Conflicts: ChangeLog

ctb added 4 commits June 15, 2015 18:04

Merge remote-tracking branch 'origin/master' into update/memory-args2

9739e5a

Conflicts: khmer/__init__.py tests/test_counting_hash.py tests/test_hashbits_obj.py

minor cleanup

782b8e8

added more create_*graph tests

0c31a8f

many small test fixes for new hashtable size behavior

4364988

ctb added 4 commits June 22, 2015 14:58

add test coverage

55ffdf8

updated ChangeLog

9f68822

updated whats-new-2.0 with description of -M/--max-memory-usage

af13a5f

changed --min-tablesize to --max-tablesize

7b44ee5

ctb mentioned this pull request Jun 23, 2015

Vulture of #621: "add estimate_optimal_hash.py" #1106

Merged

fix collect-reads and add test

251132a

SensibleSalmon mentioned this pull request Jun 23, 2015

Implementing --max-mem and auto'd hashtable agrs #1117

Closed

fix stupid method name mistake

04a6418

ctb added 3 commits June 26, 2015 07:09

fixes for recipes; added test for saturate-by-median exec

6ab3775

updated documentation

01d6afb

add multiplier for creating countgraph/nodegraph

a15f810

mr-c reviewed Jun 27, 2015
View reviewed changes

fixed option

9295052

mr-c added a commit that referenced this pull request Jun 27, 2015

Merge pull request #1050 from dib-lab/update/memory-args2

b069826

Add --max-memory-usage argument, and associated khmer_args refactoring.

mr-c merged commit b069826 into master Jun 27, 2015

mr-c deleted the update/memory-args2 branch June 27, 2015 00:27

ctb mentioned this pull request Jun 29, 2015

User experience upgrades for khmer #732

Closed

SensibleSalmon mentioned this pull request Jun 29, 2015

Implement auto'd optimal hashtable args and memory limitation checking #1126

Merged

ctb mentioned this pull request Jun 30, 2015

removed incorrect warning for max_tablesize when -M is used #1133

Merged

ctb mentioned this pull request Jul 18, 2015

Deal appropriately with --unique-kmers argument #1179

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add --max-memory-usage argument, and associated khmer_args refactoring. #1050

Add --max-memory-usage argument, and associated khmer_args refactoring. #1050

ctb commented May 31, 2015

ctb commented May 31, 2015

ctb commented May 31, 2015

ctb commented Jun 5, 2015

mr-c Jun 14, 2015

ctb commented Jun 14, 2015

mr-c commented Jun 15, 2015

ctb commented Jun 23, 2015 via email

ctb commented Jun 24, 2015

SensibleSalmon commented Jun 25, 2015

ctb commented Jun 26, 2015

ctb commented Jun 26, 2015

mr-c Jun 27, 2015

mr-c commented Jun 27, 2015

ctb commented Jun 27, 2015

#1050 (comment)

mr-c commented Jun 27, 2015

#1050 (comment)

ctb commented Jun 27, 2015

#1050 (comment)

#1050 (comment)

ctb commented Jun 27, 2015 via email

Add --max-memory-usage argument, and associated khmer_args refactoring. #1050

Add --max-memory-usage argument, and associated khmer_args refactoring. #1050

Conversation

ctb commented May 31, 2015

ctb commented May 31, 2015

ctb commented May 31, 2015

ctb commented Jun 5, 2015

mr-c Jun 14, 2015

Choose a reason for hiding this comment

ctb commented Jun 14, 2015

mr-c commented Jun 15, 2015

ctb commented Jun 23, 2015 via email

ctb commented Jun 24, 2015

SensibleSalmon commented Jun 25, 2015

ctb commented Jun 26, 2015

ctb commented Jun 26, 2015

mr-c Jun 27, 2015

Choose a reason for hiding this comment

mr-c commented Jun 27, 2015

ctb commented Jun 27, 2015

#1050 (comment)

mr-c commented Jun 27, 2015

#1050 (comment)

ctb commented Jun 27, 2015

#1050 (comment)

#1050 (comment)

ctb commented Jun 27, 2015 via email