gix-index towards 1.0

[Byron]

Only interpret and make accessible all values and figure out how to bridge different formats in a single implementation ideally.

Reading

• V3 reading + V4 reading with multi-threading
• read extensions
  • needed for multi-threading
  • TREE
  • mandatory link and sdir
  • remaining optional extensions
• use bitflags instead of u32 for type safety
• test for long paths and extended flags
• publish v 0.1 of index and bitmap crates
• improve performance - there seems to be a bottleneck when reading large indices, reading a 53MB file takes 2s!
• evaluate using std::thread::scope() instead of statically scoped threads, which should make it easier (needs Rust 1.63 MVP)

Instantiation

• verification/warm up
  • visualize/print index content
  • implement verification first (to double check what we produce from a tree)
  • handle verification on linked indices
• index from tree
  • it looks like a basic tree traversal to create entries, maybe it's smart by creating them sorted right way.
  • git can actually traverse multiple trees at the same time, maybe taking advantage of them being sorted. It seems to unpack the same entry on each level for comparison.
  • it inserts without special knowledge of tree traversal, and typically does a binary search unless it naturally inserts past the end
    • It will invalidate the cache tree and update the untracked cache.
    • It checks every path and rejects .git (on macos with case-insensitivity, should be a configuration flag). Handles backslashes in paths, too, on windows, rejecting them.
    • '.git' is always 'outlawed' without case checking, everywhere.
  • The tree cache (extension) is created after inserting all entries, and thus not created on the fly
• create cache tree
  • see here - interesting is to see promisor objects and 'quick-fetches' without negotiation.
• multi-threaded index reading (now) creates too many threads, and is actually slower than single-threaded reading

Writing

Definitely round-trip tests with what we are reading.

• Write tree from index
  • when writing trees, assure these do not contains relative path components or .git named entries
  • See if TREE extension can be used to accelerate the tree generation.
  • Maybe see Git code here
• write V4
• write entire tree after clone
  • 'tree' extension
  • what about racy timestamps? Needs modification dates after checkout to learn more
  • parallelization support (via extension)
  • … see if other extensions should also get write support, even though there isn't much use in them after clone in particular.
• generalized writing - an index after various modifications
  • protections, while we don't support it
    • [CE_REMOVED flags](https://github.com/git/git/blob/master/read-cache.c#L2894:L2894
    • extended flags
  • V2 index, no extensions
    • automatic version - use version that makes sense (2 or 3, depends on extended flags), or preset.
    • maybe handle the above with some sort of prepare step to do the cleanup like git does, needs mutation

Plumbing

• entries
• info, with details for extensions…
  • TREE
  • …others

Research

• There is an fs-monitor implementation in Rust, it might be a basis for a pure-Rust notify based implementation, avoiding the watchman tool.
• the ignore crate is probably interesting for .gitignore/exclude files. And it's a big dependency but most definitely worth it.