Remove dependence upon PYTHONHASHSEED

[mbauman]

The result from the search for predicates depends upon the initial order of the set of predicates. That order depends upon the value of PYTHONHASHSEED since it's using a set. I've not thoroughly looked for other places where this causes a dependency, but it'd be nice to support fully reproducible runs with just random.seed and numpy.random.seed.

[fgregg]

what would that look like?

[mbauman]

Something like orderedset would be sufficient — it just uses insertion order. Of course the predicate selection algorithm still shuffles it, but the initial state is now deterministic. It's not a big deal, but it'd make reproducible runs a little easier since it wouldn't require starting a new python instance.

[fgregg]

what's the performance cost for that?

[mbauman]

I'm not sure; I've not evaluated any potential solutions yet. Theoretically it may not have a major cost — in fact Python's builtin dictionaries recently moved to an ordered implementation since they found its performance to be advantageous.

[fgregg]

If you provide a PR with benchmarks, this can move forward.

…

On Wed, Apr 12, 2017 at 1:16 PM, Matt Bauman ***@***.***> wrote: I'm not sure; I've not evaluated any potential solutions yet. Theoretically it may not have a major cost — in fact Python's builtin dictionaries recently moved to an ordered implementation since they found its performance to be advantageous. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#541 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAgxbY0nyAMhIfYZqEBvr3bMtSIUVrsYks5rvRUWgaJpZM4M7wIV> .

-- 773.888.2718

[mbauman]

Do you have a benchmark suite? Or any micro-benchmarks? Or would you just be interested in the overall run time for a complete deduplication run?

[fgregg]

dedupe-examples would be fine.

…

On Wed, Apr 12, 2017 at 1:59 PM, Matt Bauman ***@***.***> wrote: Do you have a benchmark suite? Or any micro-benchmarks? Or would you just be interested in the overall run time for a complete deduplication run? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#541 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAgxba8k0P7ylIuDQKfOFitRlMBAydceks5rvR8ugaJpZM4M7wIV> .

-- 773.888.2718

[fgregg]

no response on this, so I'm closing