Feat: (pseudo-) unique hash for dataset

[MerlinSmiles]

I'd like to implement a hash for each dataset, that is unique enough for our general applications, and short enough for humans to read :)

Reasons for that:

• easily identify a dataset by its hash so that it is independent of folder structure and things like that
• add that (human readable) hash to i.e. a plot or my logbook so that I am always sure that I know where the data was exactly
• eventually in the far future have a way that I in my logbook have a link qc://notebook/S6N09W65ZL which loads a data-analysis notebook

If we think that is useful we'd need to make sure we can generate smart hashes, where it is unlikely that we have a double version.
We could:

import random
import string
N = 10
hash = ''.join(random.SystemRandom().choice(string.ascii_lowercase + string.digits) for _ in range(N))

I have no idea how unique it is though...

we can of course make a really really long string but thats hard to put in a plot title or something like that.

Any ideas/comments about that?

[peendebak]

What about uuid?

Also: why a hash and not a timestring, perhaps with a letter for the location, e.g. qd20161109003910

The time will make it unique, and is human readable.

[AdriaanRol]

@MerlinSmiles , @peendebak we also use the date_time format as an implicit unique ID. However we sometimes run into the problem that there are multiple files with the same date-timestamp (some experiments generate multiple datasets per second). I guess the uuid solves this by going to a finer resolution.

Another idea (that can be incorporated with the above) would be to use a certain aspect of git-hashes. The git hashes are generally long. However if you provide enough characters (~7) to be unique the software will automatically select the right hash. We could implement the same functionality for the qcodes hashes so that if you specify yymmddhhmm (and no seconds) it will still find it because there are no other datasets corresponding to it.

tl;dr

• timestamp based hashes 👍
• Allow reference by partial hash (like git hash)

[MerlinSmiles]

@peendebak @eendebakpt @AdriaanRol
Yes I was looking into that uuid too, also apparently it can be shortened whith XORing parts of it and it should still be random. I had a thought to put a long and a short version in the snapshot (to not depend on a correct implementation of a shortening algorithm elsewhere).

I use timestamps currently in file / folder names and it works ok, but I might want to rename, reorganize and do stuff.
I am not very fond of timestamps for uniqueness, if you have many measurement setups and many datasets, there might be collisions.
Also a string based on characters and numbers will be much shorter (and readable) than timestamps.

[AdriaanRol]

The hash does not need to be a part of the filename (that's just what we do now), so you should still be free to reorganize, rename and do stuff :).

I agree that shorter generally means more readable, however I think timestamps are generally way more readable than random hashes.
It would also ensure that the timestamp is always stored, allowing automating some kind of data retrieval system.

[giulioungaretti]

I'd just use the git way, as we talked before. 🖌️

[MerlinSmiles]

Where would the right place be to generate and save that hash?
upon initialization of the dataset?
And then add it to the snapshot when saving? Or an empty file which filename is the hash, for easy finding later?

[alan-geller]

One thing to be careful about with random hashes is that you need a large hash to avoid the birthday problem. That's why GUIDs/UUIDs are so big, and cryptographic hashes are typically very large.

A timestamp plus a short random string often works well, unless you're generating a very large number of data sets each second.

And supporting substring match like Git makes a lot of sense no matter how we generate the unique IDs.

[AdriaanRol]

unless you're generating a very large number of data sets each second.

@alan-geller Just to make sure no hidden assumptions slip into the framework, we generally do generate multiple datasets per second. I'm not worried about this creating any problems but just to be sure.

[WilliamHPNielsen]

Closing this issue since the desired feature has been added in #1002 (and we no longer develop the old dataset).