[MRG] Move greyhound-core into sourmash

[luizirber]

This PR moves the greyhound-core (RevIndex and gather) into sourmash. It doesn't bring the CLI, web server or browser frontend.
(The CLI should probably be exposed here at some point, the web server and frontend should go into wort).

I created a new feature on the Rust side, "experimental". The idea is to allow experimentation without making stability guarantees, including passing all checks required for merging (like wasm support). #1221 is another example of an "experimental" feature.
In order to avoid piling up experimental features, I also propose a requirement that sourmash-Python CAN'T use the "experimental" feature. This keeps us honest, and force stabilization in the Rust side =]
This is sort of equivalent to the nightly features in the Rust compiler.

Lots left to do, but a small list:

• properly use the parallel feature to expose rayon
• fix wasm compilation

[codecov]

Codecov Report

Merging #1238 (f016123) into latest (f9a2e96) will increase coverage by 6.35%.
The diff coverage is 65.17%.

@@            Coverage Diff             @@
##           latest    #1238      +/-   ##
==========================================
+ Coverage   83.58%   89.94%   +6.35%     
==========================================
  Files         113       88      -25     
  Lines       12185     8592    -3593     
  Branches     1684     1705      +21     
==========================================
- Hits        10185     7728    -2457     
+ Misses       1743      600    -1143     
- Partials      257      264       +7     

Flag	Coverage Δ
python	89.94% <65.17%> (-0.32%)	⬇️
rust	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/sourmash/utils.py	78.94% <0.00%> (ø)
src/sourmash/index/revindex.py	63.46% <63.46%> (ø)
src/sourmash/index/__init__.py	95.96% <100.00%> (ø)
src/core/src/encodings.rs
src/core/src/ffi/mod.rs
src/core/src/ffi/utils.rs
src/core/src/index/linear.rs
src/core/src/index/mod.rs
src/core/src/index/sbt/mhbt.rs
src/core/src/index/sbt/mod.rs
... and 20 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f9a2e96...f016123. Read the comment docs.

[ctb]

In order to avoid piling up experimental features, I also propose a requirement that sourmash-Python CAN'T use the "experimental" feature. This keeps us honest, and force stabilization in the Rust side =]

Hmm. 👎

I instead propose that we add an "experimental" subcommand in sourmash CLI that is explicitly not stable.

Alternatively, as long as it's exposed by the Rust API to Python, I can use it in scripts. So we could keep it away from the CLI, at the cost of not being able to evolve a better CLI as we play with use cases.

Motivation: I do most of my experimentation in Python, as do others, and I think sourmash has benefited substantially from this kind of experimentation. So I want to keep this available!

[luizirber]

In order to avoid piling up experimental features, I also propose a requirement that sourmash-Python CAN'T use the "experimental" feature. This keeps us honest, and force stabilization in the Rust side =]

Hmm. -1

I instead propose that we add an "experimental" subcommand in sourmash CLI that is explicitly not stable.

Alternatively, as long as it's exposed by the Rust API to Python, I can use it in scripts. So we could keep it away from the CLI, at the cost of not being able to evolve a better CLI as we play with use cases.

Motivation: I do most of my experimentation in Python, as do others, and I think sourmash has benefited substantially from this kind of experimentation. So I want to keep this available!

Fair point. Note that we can request "experimental" features from the Rust side on any sourmash PRs just fine (add "--features experimental" to the cargo invocation in setup.py), what I'm blocking is merging PRs that use the experimental feature without also "stabilizing" it on the Rust side.

[ctb]

is there value in combining this with #1131 at all? (I don't really know, am wondering.)

[luizirber]

is there value in combining this with #1131 at all? (I don't really know, am wondering.)

Probably... I see the LcaDB as a superset of RevIndex, and could probably extend the latter with the taxinfo bits from the former, but this becomes a larger PR than this one.

[luizirber]

as I suspected, shoehorning greyhound into how sourmash does gather is more
complicated than it looks.

First hurdle is that I did the greyhound demos using on-disk signatures (and paths),
but sourmash indices also work with in-memory signatures
(create an empty index, add in-memory sigs).
So, I've been modifying greyhound to support both (not hard, but does change
some of the internal logic).

Next, the gather issue discussed in #1263
I was thinking about making another method in Index for now, counter_gather,
for the greyhound behavior (returning a Counter for the matches in the index),
with a blanket implementation behaving like a LinearIndex (which... I think it
would also work by default for the SBT?), and figuring out later how to
integrate back into the regular Index.gather.
(side benefit: testing how well counter-gather works on SBTs, which I think
might not be terrible on runtime and actually stellar on memory consumption?)

[ctb]

Is fine by me!

[mr-eyes]

🎉 🎉 🎉 🎉 🎉 🎉

[ctb]

After a frustrating 15 minutes of debugging, I feel like the reorganization of index.py into index/ should have received a mention in the PR description :)