Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Hash and Hasher update for faster common case hashing #28044

Closed
wants to merge 2 commits into from
Closed

WIP: Hash and Hasher update for faster common case hashing #28044

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e8695f9
Hasher and Hash updates
bluss Aug 27, 2015
a661c5c
Fixup hash tests and benchmarks
bluss Aug 27, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions src/libcore/hash/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,13 @@ pub trait Hasher {
#[stable(feature = "rust1", since = "1.0.0")]
fn write(&mut self, bytes: &[u8]);

/// Emit a delimiter for data of length `len`
#[inline]
#[unstable(feature = "hash_delimit", since = "1.4.0", issue="0")]
fn delimit(&mut self, len: usize) {
self.write_usize(len);
}

/// Write a single `u8` into this hasher
#[inline]
#[stable(feature = "hasher_write", since = "1.3.0")]
Expand Down Expand Up @@ -230,8 +237,9 @@ mod impls {
#[stable(feature = "rust1", since = "1.0.0")]
impl Hash for str {
fn hash<H: Hasher>(&self, state: &mut H) {
// See `[T]` impl for why we write the u8
state.delimit(self.len());
state.write(self.as_bytes());
state.write_u8(0xff)
}
}

Expand Down Expand Up @@ -272,7 +280,7 @@ mod impls {
#[stable(feature = "rust1", since = "1.0.0")]
impl<T: Hash> Hash for [T] {
fn hash<H: Hasher>(&self, state: &mut H) {
self.len().hash(state);
state.delimit(self.len());
Hash::hash_slice(self, state)
}
}
Expand Down
8 changes: 8 additions & 0 deletions src/libcore/hash/sip.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,14 @@ impl Hasher for SipHasher {
self.write(msg)
}

#[inline]
fn delimit(&mut self, len: usize) {
// skip the first delimiter
if self.length > 0 {
self.write_usize(len);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this works fine with siphash and still defends against hash flooding by tuples or structs. The first string or slice doesn't get its length emitted (to quicken the common case -- only one field is hashed).

("aa", "a") and ("a", "aa") don't collide in this model, but maybe there's something I'm not thinking of?

There's also the option to do something that is outside the data stream so to speak -- we could insert an extra sip compress round here for the self.length > 0 case.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about ( &[1usize,2usize], None::<usize> ) and ( &[1usize], Some(1usize) ) ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would advise against inserting extra sip compress rounds in any circumstance. Cryptographers today tend to frown on level-crossing things like that; SipHash is designed to achieve its security goals when fed a byte stream, and nobody knows whether it still will if fed an arbitrary mixture of "bytes" and "bonus rounds".

Even if we had a hash function designed for use on a 257-element alphabet, there would be a collision between vec![vec![],vec![]] and vec![vec![vec![]]] : Vec<Vec<Vec<i8>>>.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good points.

Yes I think (&[x, 1], None) and (&[x], Some(0)) would both hash to the same sequence of x, 1, 0 (in for example 64-bit integers), That's not good, and it allows arbitrary many values to produce the same hash, so SipHasher cannot permit this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or wait, is that a problem? It's just two elements with the same hash, not arbitrary many of one hash. So the example it's not enough for a hash flooding attack.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can make more than two, although it's limited to no more than N + 1 interpretations for an N-byte string. This example achieves N / usize::BYTES:

#[derive(Hash)]
enum Foo { One, Two(Box<Foo>) }

(vec![2usize; n], One)

(My experiments with --pretty expanded tell me that discriminators are one-based, not zero-based).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aha! The actual discriminant of None is 0, but maybe deriving doesn't use that, I didn't know.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I made a critical reading comprehension error earlier. The discriminants are 0, both for user-defined types and the builtin Option; see http://is.gd/YjOcxM (playpen).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

About the One / Two example, since the type Foo is actually variable length (a linked list), it must emit a delimiter of its own. ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's a good example, it seems to be a flooding attack. It's properly delimited by One on its own.

}
}

#[inline]
fn finish(&self) -> u64 {
let mut v0 = self.v0;
Expand Down
2 changes: 1 addition & 1 deletion src/libcoretest/hash/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ fn test_writer_hasher() {
assert_eq!(hash(&'a'), 97);

let s: &str = "a";
assert_eq!(hash(& s), 97 + 0xFF);
assert_eq!(hash(&s), 1 + 97);
// FIXME (#18283) Enable test
//let s: Box<str> = box "a";
//assert_eq!(hasher.hash(& s), 97 + 0xFF);
Expand Down
8 changes: 4 additions & 4 deletions src/libcoretest/hash/sip.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -239,23 +239,23 @@ fn test_hash_no_concat_alias() {
fn bench_str_under_8_bytes(b: &mut Bencher) {
let s = "foo";
b.iter(|| {
assert_eq!(hash(&s), 16262950014981195938);
hash(&s)
})
}

#[bench]
fn bench_str_of_8_bytes(b: &mut Bencher) {
let s = "foobar78";
b.iter(|| {
assert_eq!(hash(&s), 4898293253460910787);
hash(&s)
})
}

#[bench]
fn bench_str_over_8_bytes(b: &mut Bencher) {
let s = "foobarbaz0";
b.iter(|| {
assert_eq!(hash(&s), 10581415515220175264);
hash(&s)
})
}

Expand All @@ -268,7 +268,7 @@ irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nul
pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui \
officia deserunt mollit anim id est laborum.";
b.iter(|| {
assert_eq!(hash(&s), 17717065544121360093);
hash(&s)
})
}

Expand Down
19 changes: 2 additions & 17 deletions src/libstd/sys/common/wtf8.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ impl CodePoint {
///
/// Similar to `String`, but can additionally contain surrogate code points
/// if they’re not in a surrogate pair.
#[derive(Eq, PartialEq, Ord, PartialOrd, Clone)]
#[derive(Eq, PartialEq, Ord, PartialOrd, Clone, Hash)]
pub struct Wtf8Buf {
bytes: Vec<u8>
}
Expand Down Expand Up @@ -382,6 +382,7 @@ impl Extend<CodePoint> for Wtf8Buf {
///
/// Similar to `&str`, but can additionally contain surrogate code points
/// if they’re not in a surrogate pair.
#[derive(Hash)]
pub struct Wtf8 {
bytes: [u8]
}
Expand Down Expand Up @@ -796,22 +797,6 @@ impl Hash for CodePoint {
}
}

impl Hash for Wtf8Buf {
#[inline]
fn hash<H: Hasher>(&self, state: &mut H) {
state.write(&self.bytes);
0xfeu8.hash(state)
}
}

impl Hash for Wtf8 {
#[inline]
fn hash<H: Hasher>(&self, state: &mut H) {
state.write(&self.bytes);
0xfeu8.hash(state)
}
}

impl AsciiExt for Wtf8 {
type Owned = Wtf8Buf;

Expand Down