Remove the raw feature and make RawTable private

[Amanieu]

This will give more freedom for the internal implementation details of hashbrown to evolve without the need for regular releases with breaking changes.

All existing users of RawTable should migrate to the HashTable API which is entirely safe while providing the same flexibility as RawTable.

This also removes the following features which were only exposed under RawTable:

• RawTable::iter_hash
• RawIter::reflect_insert and RawIter::reflect_remove
• RawTable::clone_from_with_hasher
• RawTable::insert_no_grow and RawTable::try_insert_no_grow
• RawTable::allocation_info
• RawTable::try_with_capacity(_in)
• HashMap::raw_table(_mut) and HashSet::raw_table(_mut)

If anyone was previously relying on this functionaly, please raise a comment. It may be possible to re-introduce it as a safe API in HashTable and/or HashMap.

[Amanieu]

cc @clarfonthey

[Dandandan]

I think https://github.com/Apache/arrow-datafusion

uses:

• RawTable::try_insert_no_grow
• RawTable::allocation_info

[clarfonthey]

Since you're already doing this, would you mind pushing a release after too so rust-lang/rust#128711 can go through? Please and thank you. <3

IMHO the sooner we do this, the better, since it gives people time to migrate away from the API before we start following up on the threat to improve the code. And, since it's a published crate, it's not a big deal, since the old (non-breaking) versions will continue to work correctly.

[Amanieu]

I think Apache/arrow-datafusion uses:

Datafusion seem to actually require the full functionality of RawTable since it directly stores bucket indices in a BinaryHeap so that it can delete arbitrary elements later (and rebuilds the heap on rehash to keep the indices valid).

This isn't something that can be supported on HashTable since it relies on low-level implementation details of RawTable such as the fact that buckets are never moved except in a rehash. Perhaps in the future the new raw API that @clarfonthey is planning (see #545) will be publicly exposed, but in the meantime datafusion can just stay on the previous hashbrown version.

[Dandandan]

FYI @alamb @avantgardnerio

[alamb]

We have also been discussing implementing our own custom hash table in apache/datafusion#7095, so perhaps this would be another potential reason to pursue that idea

I agree that using the low level details of how the hash table in hashbrown is implemented is not ideal (e.g. it constrains how hashbrown can version releases)

@Amanieu I wonder if you would consider a feature flag on hashbrown like hashbrown_unstable (following the model of tokio_unstable). That would let DataFusion keep using hashbrown internals (and it would be on us to deal with any changes during upgrade)

[avantgardnerio]

This will require work in DataFusion to fix TopK aggregations, and then after it is fixed it will cause significant (40% IIRC) performance regressions.

[Amanieu]

I don't think that's a good idea because every minor release of hashbrown could break users of your crate.

However you can just keep using the 0.14 version of hashbrown which will still have the raw feature. It's only being removed in 0.15.

[alamb]

However you can just keep using the 0.14 version of hashbrown which will still have the raw feature. It's only being removed in 0.15.

Another option is for us to to fork the crate which might be reasonable if we can get more performance by doing so

[Dandandan]

I think we should create some tickets in DataFusion for moving our usage towards HashTable / own hashtable (e.g. better vectorized) / forked code.

[Amanieu]

@bors r+

[bors]

📌 Commit 26ef4a1 has been approved by Amanieu

It is now in the queue for this repository.

[bors]

⌛ Testing commit 26ef4a1 with merge aa1411b...

[bors]

☀️ Test successful - checks-actions
Approved by: Amanieu
Pushing aa1411b to master...

[the-mikedavis]

I was using RawTable::iter_hash for a multi-map-like type (kind of like HashMap<K, Vec<T>>) and I don't believe there's a way to look up multiple entries with the same key from HashTable<(K, V)>: HashTable::find stops after the first matching element. Would it be possible to introduce something safe in HashTable to take its place? I can share more code if you'd like more context and maybe work on a PR if you think it's a reasonable addition.

[Amanieu]

I think iter_hash makes sense to expose on HashTable. Feel free to send a PR, you can do a partial revert of this one to get iter_hash back in RawTable.

[xacrimon]

@Amanieu dashmap relies on RawTable functionality. With this merged we're be effectively locked to 0.14 forever.

[Amanieu]

What specific functionality do you need that isn't available though HashTable which is the replacement for RawTable?

[cuviper]

I took a brief look, and the main roadblock that I see will be iterators and entry structs that currently contain RwLock guards with corresponding RawIter / Bucket / InsertSlot. With HashTable, those second parts all have lifetimes that would need to borrow through the guard, so they become self-referential.

[clarfonthey]

Added a comment to #545 mentioning to look into dashmap's internals when I go about ripping out the raw table API. It may be the case that we need to offer some kind of lifetime-erased version of the various types to get it to work, but I'm hoping we can get around that.

[Amanieu]

I took a brief look, and the main roadblock that I see will be iterators and entry structs that currently contain RwLock guards with corresponding RawIter / Bucket / InsertSlot. With HashTable, those second parts all have lifetimes that would need to borrow through the guard, so they become self-referential.

I thought about this a bit and fundamentally it's not a problem with HashTable but rather a limitation of the MutexGuard API. I think you can work around this by using a Mutex<()> and an UnsafeCell<HashTable> where you manage the locking manually.