Has the data been shuffled?

[Lisennlp]

Hello, I see your batch_view.py, found that the data does not use a shuffle, but in the gpt-neox library, the data is shuffled.
So I want to make sure that the author did or did not shuffle during the training? Hope to get your answer, thank you！

[pietrolesci]

I think this might provide an answer #123 (comment)

[itsnamgyu]

The data is shuffled in terms of documents. The repo-id says preshuffled in https://github.com/EleutherAI/pythia#exploring-the-dataset, i.e., EleutherAI/pile-standard-pythia-preshuffled.

I'm actually not sure about https://huggingface.co/datasets/EleutherAI/pythia_deduped_pile_idxmaps mentioned in https://github.com/EleutherAI/pythia#reproducing-training. I will add a quesiton about this on #123.