Migration tutorial

[zhangguanheng66]

Add the migration tutorial in examples/legacy_tutorial folder

[cpuhrsch]

https://colab.research.google.com/github/pytorch/text/blob/9a0ff4569ec481b0382406714aaf2aca607b1651/examples/legacy_tutorial/migration_tutorial.ipynb is the link colab link associated to the latest comment at time of commenting

[zhangguanheng66]

In the README file, I use the link - https://colab.research.google.com/github/zhangguanheng66/text/blob/master/examples/legacy_tutorial/migration_tutorial.ipynb>.

[codecov]

Codecov Report

Merging #1203 (906e9cc) into master (db8da95) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1203   +/-   ##
=======================================
  Coverage   73.23%   73.23%           
=======================================
  Files          67       67           
  Lines        3718     3718           
=======================================
  Hits         2723     2723           
  Misses        995      995           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a1c8b50...9d09f9d. Read the comment docs.

[cpuhrsch]

Also, can you update the tutorial to be built against the RC?

[cpuhrsch]

As a side note

def bucket_iter_func(pool, batch_size=64):
   for rand_item in pool:
      sorted_item = sorted(rand_item,
                           key=lambda x: len(tokenizer(x[1])))  # x is a tuple of (label, text)
      sorted_dataloader = DataLoader(sorted_item, batch_size=batch_size,
                                     shuffle=False,  # shuffle is set to False to keep the order  
                                     collate_fn=collate_batch)
      for item in sorted_dataloader:
         yield item

train_iter = IMDB(split='train')
train_list = list(train_iter)
batch_size = 8 # A batch size of 8
rand_pools = DataLoader(train_list, batch_size=batch_size*100,
                        shuffle=True, collate_fn=lambda x: x)
sorted_train_dataloader = bucket_iter_func(rand_pools, batch_size=batch_size)

If you just want sublists of size batch_size*100 you can also use Python builtins

shuffle(train_list)
train_lists = [train_list[i:i+ batch_size*100] for i in range(0, len(train_list), batch_size*100)]
train_lists = [sorted(train_list, key=lambda x: len(tokenizer(x[1])))  for train_list in train_lists]
train_lists = sum(train_lists, []) # Very slow way of flattening
dataloader = DataLoader(train_lists, batch_size=batch_size,
                                     shuffle=False,  # shuffle is set to False to keep the order  
                                     collate_fn=collate_batch)

[zhangguanheng66]

As a side note

Yep. Add to the tutorial.