Adding support for mcore generate

[shanmugamr1992]

What does this PR do ?

Supports generating though megatron core

Collection: NLP

Changelog

• Added deprecation notice to old generate.
• Added support for generation using mcore.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

• New Feature
• Bugfix
• Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

• Related to # (issue)

[marcromeyn]

Can't we just add these methods to the TokenizerSpec in nemo so we don't need this wrapper-class?

[shanmugamr1992]

Hmm, this is very specific to mcore generate, so I didnt want to corrupt the original tokenizer spec class.

[shanmugamr1992]

We are also reworking tokenizers later maybe for mcore 0.9 or the next release, where nemo and mcore will use the same tokenizer. At that point will unify the apis

[shanmugamr1992]

Tested that it works. The output of the old code and the new code are exactlly the same. The old code for tp2pp2 gpt3 43b, takes 15 seconds for generating 100 tokens . The new code takes 8.5 seconds all same settings. The text outputs are exactly the same:
OLD CODE :
TOTAL TIME TAKEN 14.98s

---

[{'sentences': ["Q: How are you?\n\nA: I'm doing well. How are you?", 'Q: How big is the universe?\n\nA: The observable universe is 93 billion light years in diameter.']
['Q: Explain the concept of linear algebra to a five year old?\n\nA: Linear algebra is a way to solve problems by using numbers and shapes. It helps us understand how things move and change, like how a ball rolls down a hill or how a car turns a corner. We can use linear algebra to make things move in the right direction, without having to try lots of different ways first.', 'Q: Write a poem about the sun and moon.\n\nThe Sun and Moon']

NEW CODE :
TOTAL TIME TAKEN 8.68s
------------- RESULT FOR PROMPT 0 ---------------
{'id': '0', 'input_prompt': 'Q: How are you?', 'generated_text': "\n\nA: I'm doing well. How are you?",

------------- RESULT FOR PROMPT 1 ---------------
{'id': '1', 'input_prompt': 'Q: How big is the universe?', 'generated_text': '\n\nA: The observable universe is 93 billion light years in diameter.',

------------- RESULT FOR PROMPT 2 ---------------
{'id': '2', 'input_prompt': 'Q: Explain the concept of linear algebra to a five year old?', 'generated_text': '\n\nA: Linear algebra is a way to solve problems by using numbers and shapes. It helps us understand how things move and change, like how a ball rolls down a hill or how a car turns a corner. We can use linear algebra to make things move in the right direction, without having to try lots of different ways first.',

------------- RESULT FOR PROMPT 3 ---------------
{'id': '3', 'input_prompt': 'Q: Write a poem about the sun and moon', 'generated_text': '.\n\nThe Sun and Moon',

[ericharper]

LGTM. Thanks!

Noting here that we still need ci tests and docs for this.