Reduce memory requirement while precessing the data

[hsajjad]

Hi, is there any easy way to fix the memory hungry step during the pre-processing of the data? I guess this command is the culprit.
output_chars[split] = torch.ones(split_counts[split], max_word_l):long()

I have 40M sentences of data and it requires more than 200GB of RAM

Cheers - Hassan