Discussion : Roadmap to 1.0.0

[fulmicoton]

This ticket is here to open a discussion. Feel free to chime in, especially if you are using tantivy somewhere.

@petr-tik is advocating for a clear roadmap to tantivy 1.0.0.

I am not against really against the idea, as long as we don't rush it.
I'm ok to put a 1.0.0 stamp on tantivy if

• we are reasonably confident ppl can use this in production and sleep at night.
• we reach a version for which we are confident we won't break backward compatibility within a year or so.

We both agree being super featureful is not a problem.

Required because related to the use of tantivy in production or the backward compatibility story:

• Having a proper footer in files add checksum check in ManagedDirectory #605
• Having checksum in files add checksum check in ManagedDirectory #605 (note this was not a thing in Lucene until very recently)
• Handle deletes and large segments correctly in the merge policy Add an optional upper limit for the number of docs in a segment that is to be merged with another. #615 (note this was a problem that was addressed in Lucene 4 or something)
• Fix or ditch directory watching Remove watching directory introduced in 0.9 #602
• Rollback should get its own opstamp Rollback and Delete all docs should get their own opstamp #564 - im not sure if the status of this. investigating would be nice.
• Have a story for schema modification. Probably refactoring field to field names.
• Improve Errors.

Nice to have:

• Have a layer that does unicode normalization and possibly different operations on char Add a unicode normalizer. #407 Add support for CharFilter #409
• Test for race conditions with loom Test for race conditions with loom #497
• Make sure Term's debug can print term names
• Split tantivy into several crates to improve compilation times and split the different crate to different maintainer Splitting tantivy and call for official maintainers #618
• Near real time indexing Near Real Time indexing. #494. Lucene used not to have this but the concept of commit seems to really confuse people and I am scared of people misusing tantivy.
• Improve logging.

Note that prioritizing reaching 1.0.0 is very different from optimizing for adoption.

[sunface]

nice hearing this ,especially for production-ready roadmap

[petr-tik]

Thanks for opening this Paul.

To emphasise, I don't want to rush the 1.0.0 either. My main reason to start this conversation is to help us prioritise feature/bug work and broadcast serious intentions to prospective users of tantivy.

Apart from a consistent index format and a stable API, I believe security to be an equally important feature of tantivy.

Raph's recent post about cargo bloat and our long-ish compile times gave me time to think about another aspect of stability - security and dependencies.

Cargo makes it easy to import crates

Rust enables us to build systems software using the tooling of high-level languages like cargo and using great crates like fst (thanks BurntSushi), serde and crossbeam to name a few.

However, to the users of tantivy the buck of security stops with us. People adding tantivy to their Cargo.toml should have a great user experience and be confident that their applications will do nothing unexpected/nasty.

The sprawl of dependencies and the implicit rules about updating/reviewing might let us down. If any of our dependencies misbehave, it will make tantivy look bad and the Rust ecosystem less reliable (cough npm)

We control some dependencies: census, tantivy-fst, bitpacking, but most of them we import.

I suggest we do a light audit as part of 1.0.0 and a hard audit just before releasing 1.0.0.

Dependency light audit

Look for low hanging fruit in removing dependencies or updating their versions to reduce duplication.

For example, the scoped_pool crate looks dormant with outstanding PRs and no updates in the last 2 years.

One of the open PRs updates crossbeam to 0.4 (we are on 0.7), because there was an ordering bug in the MsQueue implementation. Scoped_pool occurs only in one file - our implementation of the search executor (src/core/executor.rs).

I don't know enough about executors to estimate the amount of effort to upgrade/replace this, but the security implications of using an out-of-date dependency seem reason enough to do so.

Minimising dependencies also carries the benefit of using rustc/cargo's caching mechanisms and speeding up builds, which improves user experience.

Upgrade regex-syntax dep in tantivy-fst to match that of regex

Using cargo-tree I found that we tantivy-fst uses a different version of regex-syntax than regex. If the API isn't too different, this should be doable.

regex-syntax v0.4.2 
└── tantivy-fst v0.1.0 
└── tantivy v0.11.0 

regex-syntax v0.6.6 
└── regex v1.1.5 
└── tantivy v0.11.0 

Won't make a massive impact, but looks like a small enough change with limited downside.

Move to Rust std futures

I expect Rust 1.39 and Futures to be released before tantivy 1.0.0. Currently, we use futures and futures_cpupool in the following files

src/indexer/segment_updater.rs
src/indexer/merger.rs 
src/indexer/index_writer.rs

If the functionality allows it, I think it's best we move to std-lib futures. Given how few files rely on it, it should be a relatively easy change.

Freeze dependencies for v1.0.0

Once we have completed the feature roadmap for 1.0.0, I suggest pinning tantivy 1.0.0 to exact versions (or version ranges like we have with combine) of dependencies in Cargo.toml.

Establish and automate future dependency audits

We can add dependency review like cargo-geiger and crev to CI.
Run it either:

• on changes to Cargo.toml (new dependency or correction of a current dependency)
• on every commit (since cargo automatically uses new minor versions unless specified otherwise).

If we pin 1.0.0 to exact versions, the latter should be redundant.

Publish a yank/replace procedure

If a problem arises with any of our dependencies, we should have a replacement procedure.

If the dependency is small, we can internalise the functionality in tantivy. If the dependency is large, we will need a way to push a hotfix.

[fulmicoton]

Once we have completed the feature roadmap for 1.0.0, I suggest pinning tantivy 1.0.0 to exact versions (or version ranges like we have with combine) of dependencies in Cargo.toml.

I wonder if this is a good idea...
Of course, we should not reexport symbol of crates external to our workspace, so conflict should not happen, but do we want to inflate clients binary because they use two libraries which import two different minors of the same library?

I thought the best practise was to not force the version down to the minor in libraries.

[petr-tik]

Fair point re minor version numbers, didn't think of that.

[fulmicoton]

Apart from that I agree with most of the stuff above.

I don't care much about having an official Publish a yank/replace procedure specific to dependencies. I think it should be the same story as for regular hotfixes. It is certainly worth having some guide somewhere, if other dev end up having to write or even (who knows?) publish hotfix

[Fiedzia]

I suggest pinning tantivy 1.0.0 to exact versions

Tantivy is a library, it should work with whatever application author will decide to use it with. Excluding versions that are known to be broken is fine if there are any, but pinning is not.