big crates are slow to use with new archive-storage

[syphar]

For crates with many files we have a performance regression.

Example is stm32ral, which has 1.2 Million files in the archives.

Currently we are reading the whole index from disk for every file-exists or get-from-archive request, which might take too much time.

[jyn514]

@syphar did you want to keep this open to track not having to read the full index into memory?

[syphar]

@syphar did you want to keep this open to track not having to read the full index into memory?

yes. But most likely it's good enough afterwards for the 99% case on docs.rs.

more optimizations would involve a new index format and I'll gather more data in a benchmark and the S3 inventory first

[jyn514]

Current status for 99th percentile:

The deploy of #1541 happened at 13:38 and the spike at 14:05 is me testing https://docs.rs/stm32ral/0.7.0/stm32ral/.

[syphar]

After:

• introduce new archive index format based on SQLite #2068
• add subcommand to migrate old CBOR archive indexes to SQLite #2154
• migrating the old indexes this week (01.08./02.08.)

this is solved.

#2179 will drop the old code