Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Partition sweep #216

Merged
merged 162 commits into from
Jan 13, 2014
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
162 commits
Select commit Hold shift + click to select a range
8844c12
Merge github.com:ged-lab/khmer into bleeding-edge
camillescott Jul 25, 2013
e32703b
Merge branch 'bleeding-edge' of github.com:ged-lab/khmer into bleedin…
camillescott Jul 25, 2013
2ee322c
added threading params to filter-abund.py
camillescott Jul 25, 2013
a7d6343
Merge branch 'graceful-fail2' into bleeding-edge
camillescott Aug 2, 2013
4378a7b
Merge branch 'bleeding-edge' of github.com:ged-lab/khmer into bleedin…
camillescott Aug 2, 2013
d9fda8f
Merge pull request #1 from ged-lab/bleeding-edge
camillescott Aug 8, 2013
d3bb3d2
Merge branch 'bleeding-edge' of github.com:ged-lab/khmer into bleedin…
camillescott Aug 28, 2013
32fa1c6
Merge branch 'bleeding-edge' of github.com:cswelcher/khmer into bleed…
camillescott Sep 2, 2013
c9467fb
Added typedefs to hashtable.hh
camillescott Sep 4, 2013
05edfb8
Added function prototype for consume and tag with colors
camillescott Sep 4, 2013
0e92a8d
added code to insert colors into colormap
camillescott Sep 5, 2013
1465c74
added consume_fasta functions
camillescott Sep 6, 2013
e19e139
merged in refactor from ged-lab/graph_partitioning_on_abundance
camillescott Sep 6, 2013
b33c71a
promoted new code to hashtable to be in line with refactor
camillescott Sep 6, 2013
c097fb5
added _cmap_contains back in after being nuked by merge
camillescott Sep 6, 2013
90e8306
fixed for proper pointers/refs, added reverse color map
camillescott Sep 6, 2013
ce1d027
added sweep function, getters for colors and tags, started color reco…
camillescott Sep 6, 2013
24f7380
color sweep added
camillescott Sep 9, 2013
a961a21
added parse function to python glue
camillescott Sep 9, 2013
1c07a01
added a spin lock for tag_colors (not strictly necessary, but ctb lik…
camillescott Sep 9, 2013
e0f7dca
adding sweep glue
camillescott Sep 9, 2013
a64264d
woot compiles and segfaults i win
camillescott Sep 9, 2013
db935a1
consuming, coloring, and sweeping functions successfully integrated w…
camillescott Sep 10, 2013
e6ba546
silly testing python script added (need to add nose tests)
camillescott Sep 10, 2013
0e350c0
added function to tag fasta by partition id
camillescott Sep 13, 2013
2b1b6b2
a few comments on @cswelcher code
ctb Sep 16, 2013
bfc0896
Merge pull request #2 from ctb/partition_sweep
camillescott Sep 17, 2013
67c1b45
exposing more stuff for python glue
camillescott Sep 18, 2013
b7c5e97
Merge branch 'partition_sweep' of github.com:cswelcher/khmer into par…
camillescott Sep 18, 2013
d948a6e
added more glue functions
camillescott Sep 19, 2013
ba9cab4
finished tag export to python land and fixed tag parsing to ull int. …
camillescott Sep 24, 2013
90f9dba
mucking about with get_all_tags function, temporarily broken...
camillescott Sep 24, 2013
5689442
fixed issues with get_all_tags
camillescott Sep 25, 2013
18ebd4a
fixed in-call increment re ctb comment
camillescott Sep 25, 2013
22333f2
added n_colors python function
camillescott Sep 25, 2013
b51df47
added test_consume_fasta_and_tag_with_colors, test passes
camillescott Sep 25, 2013
d131776
fixed consume_partitioned_fasta_and_tag_with_colors to properly check…
camillescott Sep 25, 2013
afd1792
added delete of temp colorptrmap
camillescott Sep 25, 2013
0f10ae5
added khmer colormap changes
camillescott Sep 26, 2013
6043518
added dealloc func
camillescott Sep 26, 2013
96a572d
Merge branch 'partition_sweep' of github.com:cswelcher/khmer into par…
camillescott Sep 26, 2013
eb94496
changed delete of tmp colorptrmap to clear
camillescott Sep 26, 2013
3684974
updated color_tst, remove soon
camillescott Sep 26, 2013
0b07d7f
Merge branch 'partition_sweep' of github.com:cswelcher/khmer into par…
camillescott Sep 26, 2013
bc767df
added bunch more tests, TODO fix n_colors
camillescott Sep 26, 2013
d29061c
added a sweep-reads script to scripts, TODO add tests
camillescott Sep 26, 2013
feb0f39
added test for correctness of color tagging and traversal
camillescott Sep 27, 2013
01f233d
started traversal optimization
camillescott Sep 27, 2013
1b97d8c
fixed color allocation error; added persistent color to color pointer…
camillescott Sep 30, 2013
2113c23
changed back deref on color sweep after bugfixing, switched python ex…
camillescott Sep 30, 2013
072adfd
working on new traversal code, broke off neighbor finding to its own …
camillescott Sep 30, 2013
88154dc
implemented new perimeter traversal as part of sweep opimization
camillescott Oct 1, 2013
ab23874
fixed error with new perimeter alg
camillescott Oct 1, 2013
74c9a88
fixed error in traversal params
camillescott Oct 1, 2013
05cbe35
fixed func prototype to match prev change
camillescott Oct 1, 2013
235b746
fixed bug in color allocation during consume_fasta, fixed unexpected …
camillescott Oct 1, 2013
4806913
added new tests for n_colors and get_color_dict
camillescott Oct 1, 2013
15d7a88
fixed incorrect memory estimation on sweep-reads-by-partition
camillescott Oct 2, 2013
b20d00a
squashed bug with using a traversal range of 0
camillescott Oct 2, 2013
add2eab
couple debugging things added to subset
camillescott Oct 2, 2013
bf5a544
added new test file
camillescott Oct 3, 2013
e8d06aa
final optimizations to traversal
camillescott Oct 3, 2013
76e91d7
added temporary testing scripts, different sweep scripts, etc
camillescott Oct 3, 2013
23c197e
addded fix for tag color correctness test not working on HPCC, still …
camillescott Oct 3, 2013
b2f6164
c++ land color test
camillescott Oct 3, 2013
fd9728a
changes to test-Colors
camillescott Oct 3, 2013
4c419ed
resync with hpcc
camillescott Oct 7, 2013
e8466b1
changes to test-colors
camillescott Oct 7, 2013
c23c26d
fixed test-colors issue
camillescott Oct 7, 2013
639099f
testing out buffer based spitter
camillescott Oct 10, 2013
aca8b2c
some changes to buffered splitting
camillescott Oct 10, 2013
b71e086
fixed conflict with ktable test in lib Makefile
camillescott Oct 17, 2013
fa8d5bd
added debugging option to make
camillescott Oct 22, 2013
4e30cec
fixed color test build params
camillescott Oct 22, 2013
cd86674
couple changes to color-Test
camillescott Oct 22, 2013
e54b216
changes to original sweep reads, now deprecated...
camillescott Oct 22, 2013
b871ad6
Merge branch 'master' of github.com:cswelcher/khmer
camillescott Oct 22, 2013
d13cb52
added combined sweep and file output script
camillescott Nov 1, 2013
412eec9
Merge branch 'master' of github.com:ged-lab/khmer
camillescott Nov 4, 2013
8120362
changed bad env line
camillescott Nov 5, 2013
0d89921
important change in traversal code: removed optimization which trunca…
camillescott Nov 12, 2013
c6035a1
Merge branch 'master' into partition_sweep
camillescott Nov 12, 2013
bd2fcdb
added default parameters to buffered sweep
camillescott Nov 13, 2013
9278ecd
added error handling to file opening and buffer flushing
camillescott Nov 13, 2013
ac63a8b
added warning output for errors, updated description
camillescott Nov 13, 2013
d524257
added minimum k and hashsizes to prevent inanely complex useless sear…
camillescott Nov 13, 2013
3a7ff00
started tests
camillescott Nov 13, 2013
37ac9d8
working on tests
camillescott Nov 14, 2013
0ccb623
changed something...
camillescott Nov 14, 2013
eb7b346
removed old sweep scripts
camillescott Nov 14, 2013
c253a91
sweeped reads properly puts output files in source dir
camillescott Nov 14, 2013
3b6f9af
fixed key error in error checking code
camillescott Nov 14, 2013
3278230
was not flushing buffer at end of run! oops...
camillescott Nov 14, 2013
0f8fb29
fixed screwy extra spaces in output
camillescott Nov 14, 2013
e7ece33
added initial test for sweep
camillescott Nov 14, 2013
409fa78
test data for sweep tests
camillescott Nov 14, 2013
2b312a8
added checking multi and orphan reads to sweep tests
camillescott Nov 14, 2013
51255b6
added case for missing file to sweep
camillescott Nov 14, 2013
6e06219
removed old color sweep functions
camillescott Nov 14, 2013
d94f707
fix for stochastic color order in tests
camillescott Nov 14, 2013
33c711c
updated multi file for standard name
camillescott Nov 14, 2013
297ca68
updated test data
camillescott Nov 14, 2013
b65ce5c
fixed horked mem usage estimate
camillescott Nov 14, 2013
3758564
added error handling for too-short reads
camillescott Nov 18, 2013
48a0e8c
fixed error in error handling code FACEPALM
camillescott Nov 19, 2013
b43328b
rewrote buffering code for better performance and simplicity, changed…
camillescott Nov 19, 2013
a915e7b
removed deprecated reference to fps
camillescott Nov 19, 2013
750cc36
changed all names using color to use label
camillescott Nov 19, 2013
c9f4b10
forgot to commit test changes
camillescott Nov 19, 2013
d40d868
aaaaaand the script
camillescott Nov 19, 2013
8b28ba3
writing script to build sparse graph
camillescott Nov 19, 2013
048e542
done for the day
camillescott Nov 19, 2013
a693a87
removed references to debugger
camillescott Nov 22, 2013
d6e4088
playing with sparse graph viz
camillescott Nov 22, 2013
32c0c39
changed buffering parameters to make more sense
camillescott Nov 22, 2013
d79a5ba
fixed env line
camillescott Dec 4, 2013
7766bcc
Merge branch 'partition_sweep' of github.com:ged-lab/khmer into parti…
camillescott Dec 4, 2013
7a57bd2
fixed merge conflicts in hashbits, test_filters, and _khmermodule
camillescott Dec 4, 2013
7d80ee9
fixed derped up merge from partition_on_abundance, properly 3 way mer…
camillescott Dec 4, 2013
72b945e
fixed error in buffer flushing
camillescott Dec 4, 2013
36b6a80
beginning subclassing of label stuff
camillescott Dec 10, 2013
902cc2b
added labelhash.cc
camillescott Dec 11, 2013
605c04e
moved parse pid to read_parsers file, fixed syntax errors in labehash…
camillescott Dec 11, 2013
4127089
added labelhash defs to khmermodule, started stripping labeling metho…
camillescott Dec 12, 2013
09dd66a
finished most of integration, added new tests, dealing with linker er…
camillescott Dec 17, 2013
ebf8a9e
stripped label stuff from hashtable.cc
camillescott Dec 17, 2013
6ee7f91
switched include ordering back
camillescott Dec 17, 2013
7e8a1ac
added setup.py which had been left out (still doesn't work)
camillescott Dec 17, 2013
5bbb493
commented out undefined destructor
camillescott Dec 17, 2013
b07e3f0
fixed namespaces
camillescott Dec 17, 2013
a2fd39a
now the namespaces are nice fishjord
camillescott Dec 17, 2013
1ac3d6f
stripped final remnants of labeling code out of hashtable
camillescott Dec 18, 2013
7478d69
some inlined functions in hashtable.hh removed
camillescott Dec 18, 2013
e75a838
narrowing down freezup: all_tags spinlock in labelhash
camillescott Dec 20, 2013
13bff6e
tracked down SIGBUS error to labelhash_dealloc function not calling t…
camillescott Dec 20, 2013
9ac22f0
swapped out tst-Colors in lib Makefile
camillescott Dec 20, 2013
747dab0
removed rogue test files?
camillescott Dec 20, 2013
9289e0c
added same tp_free call to dealloc for Hashbits
camillescott Dec 20, 2013
3ec384d
exposed Hashbits object to python with constructor providing primes
camillescott Dec 20, 2013
a50061d
fixed Hashbits dealloc by removing old PyObject_Del, added tests for …
camillescott Dec 22, 2013
035659c
merge in master
mr-c Jan 10, 2014
10598e4
killed noisy init code
camillescott Jan 10, 2014
acb6348
autopep8
mr-c Jan 10, 2014
9a511c4
cpychecker fixes
mr-c Jan 10, 2014
ee8e819
fixed referenced count errors in pylist_setitem and pydict_setitem ca…
camillescott Jan 10, 2014
ab67608
add some error messages
mr-c Jan 13, 2014
fc444d2
Merge branch 'partition_sweep' of github.com:ged-lab/khmer into parti…
mr-c Jan 13, 2014
9a0e71e
fixed typo and cleaned up old code comments
camillescott Jan 13, 2014
ad71bec
one more error message
mr-c Jan 13, 2014
e9afda7
added PyErr_Setstring usage to exception in consume_...tag_with_labels
camillescott Jan 13, 2014
9c08a57
reverted change in setup.cfg
camillescott Jan 13, 2014
4b1c98a
inlined _parse_partition_id
camillescott Jan 13, 2014
5f38587
updated description in comments and argparse to match function
camillescott Jan 13, 2014
c85a6e8
Merge branch 'partition_sweep' of github.com:ged-lab/khmer into parti…
camillescott Jan 13, 2014
62a6504
moved sandboxy script to sandbox, removed explicit file path
camillescott Jan 13, 2014
769ae39
fixed long lines
camillescott Jan 13, 2014
c99a471
fixed more long lines
camillescott Jan 13, 2014
7bbe460
made fastp non-optional, shortened lines, removed includes
camillescott Jan 13, 2014
c35ef38
updated tests to reflect new script option format
camillescott Jan 13, 2014
69447b6
removed deprecated sweep_sequence_for_labels, added test for consume_…
camillescott Jan 13, 2014
037115d
shortened remaining long lines:
camillescott Jan 13, 2014
3c4158f
greatly expounding on help
camillescott Jan 13, 2014
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions khmer/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@
from _khmer import reverse_hash
from _khmer import get_config
from _khmer import ReadParser
from _khmer import _LabelHash
from _khmer import _Hashbits

from ._version import get_versions
__version__ = get_versions()['version']
Expand Down Expand Up @@ -148,4 +150,27 @@ def get_n_primes_above_x(n, x):
i += 2
return primes

'''
Expose the cpython objects with __new__ implementations.
These constructors add the functionality provided by the existing
factory methods to the constructors defined over in cpython land.
Additional functionality can be added to these classes as appropriate.
'''


class LabelHash(_LabelHash):

def __new__(cls, k, starting_size, n_tables):
primes = get_n_primes_above_x(n_tables, starting_size)
c = _LabelHash.__new__(cls, k, primes)
c.primes = primes
return c


class Hashbits(_Hashbits):

def __new__(cls, k, starting_size, n_tables):
primes = get_n_primes_above_x(n_tables, starting_size)
c = _Hashbits.__new__(cls, k, primes)
c.primes = primes
return c
Loading