[MRG] distance utility functions to support ANI estimation

[bluegenes]

This PR adds distance utility functions, ultimately to support #1788

A few notes:
For ANI from either jaccard and containment, we can assess the probability that the sketches have nothing in common based on chance alone (== false negative). Ideally, we would keep count of how many times we encounter a too-high probability, and recommend that the user retains more hashes (decrease scaled).

For containment, we can optionally output confidence intervals, which actually take into account the impact of the scaled factor on ANI estimation (point estimate does not). I would like to retain the ability to output these, and output them only when desired (python API or maybe sourmash sig overlap).

For jaccard, we only calculate the point estimate, but there is a small error associated with this estimation (it was suggested that > 10^-4 should be handled with caution). We can warn the user when this situation occurs, but I'm also currently returning this, as it might be useful to have as output, so users can eliminate untrustworthy comparisons.

Note that containment --> ANI will always be more accurate than jaccard --> ANI, but it seems useful to support jaccard, as 1. in my experience, the point estimate is actually quite similar. and 2. there are still occasions where users may prefer jaccard. Both formulas do require scaled sketches, so num sketches will not be compatible with ANI estimation

• finish updating all tests to reflect changes in function return values
  • by default, do not return CI values
  • no CI values from jaccard dist
• add test for get_exp_probability_nothing_common
• run pyflakes; black formatting for distance_utils.py
• add minhash fns

Since the signature functions are really similar to the minhash functions, I'm going to save those for a separate PR, so this one is smaller/easier to review.

[codecov]

Codecov Report

Merging #1934 (fb54506) into latest (0d04cca) will increase coverage by 8.19%.
The diff coverage is 99.47%.

@@            Coverage Diff             @@
##           latest    #1934      +/-   ##
==========================================
+ Coverage   82.94%   91.14%   +8.19%     
==========================================
  Files         125       95      -30     
  Lines       13763     9674    -4089     
  Branches     1877     1910      +33     
==========================================
- Hits        11416     8817    -2599     
+ Misses       2075      584    -1491     
- Partials      272      273       +1     

Flag	Coverage Δ
python	91.14% <99.47%> (+0.16%)	⬆️
rust	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/sourmash/distance_utils.py	99.32% <99.32%> (ø)
src/sourmash/minhash.py	93.20% <100.00%> (+0.64%)	⬆️
src/core/src/ffi/signature.rs
src/core/src/index/bigsi.rs
src/core/src/index/search.rs
src/core/src/cmd.rs
src/core/src/ffi/hyperloglog.rs
src/core/src/encodings.rs
src/core/src/index/linear.rs
src/core/tests/test.rs
... and 23 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0d04cca...fb54506. Read the comment docs.

[bluegenes]

Ready for review @ctb

[ctb]

Overall looks really good - only a few test-this-please requests, and some suggested changes to function parameters!

[bluegenes]

@ctb I think this is ready for re-review, assuming tests pass

[bluegenes]

@ctb is there a way to check this here that I'm missing? I ended up adding a direct test in test_distance_utils.py, so I could rm this test if we want. But it's kinda nice to know that this situation triggers it...

[ctb]

looks good! just one suggestion, and one comment! merge as you will ;)

[bluegenes]

@ctb py tests were passing, but now they're not and I'm not really sure why...

[ctb]

@ctb py tests were passing, but now they're not and I'm not really sure why...

you merged my quote adjustment but didn't fix the tests, looks like -

assert("Error: distance estimation requires input of either `sequence_len_bp` or `n_unique_kmers`") in str(exc)
  E       assert 'Error: distance estimation requires input of either `sequence_len_bp` or `n_unique_kmers`' in '<ExceptionInfo ValueError("Error: distance estimation requires input of either \'sequence_len_bp\' or \'n_unique_kmers\'") tblen=2>'
  E        +  where '<ExceptionInfo ValueError("Error: distance estimation requires input of either \'sequence_len_bp\' or \'n_unique_kmers\'") tblen=2>' = str(<ExceptionInfo ValueError("Error: distance estimation requires input of either 'sequence_len_bp' or 'n_unique_kmers'") tblen=2>)

[ctb]

🎉