[Torchscript] Enable inference optimizations on the scripted model

[pooyanamini]

Patch description
Enable inference optimization on the torchscripted model. This also
fixes a crash in inference from the torchscripted model when we had
multi-line context.

Testing steps
Tried scripting a model and running inference on it for multi-turn context and made sure we don't see the crash and the output is correct.
buck run //deeplearning/projects/parlai:parlaicmd -- torchscript --model-file manifold://cair_models/tree/cairaoke/experimental/christopherhlin/f335033955/cstudio_reminder_r4 --model fb:bart/cairaoke_bart --scripted-model-file manifold://cair_models/tree/cairaoke/experimental/test/script_reminder_debug.pt --no_cuda --input 'what are my reminders for tomorrow|api_resp: get_entity.date = 2021-07-12T00:00:00.000-07:00'

Other information

[EricMichaelSmith]

@pooyanamini This makes sense to me, but the CI check at https://app.circleci.com/pipelines/github/facebookresearch/ParlAI/11294/workflows/18cf7bc2-284a-43f0-908e-477f2ce3f78e/jobs/92335 is failing now - can you address? I wonder if maybe we should make inference optimization a command-line flag if it wouldn't be backcompatible otherwise?

[pooyanamini]

I tried reproducing the error by running this command:

buck run //deeplearning/projects/parlai:parlaicmd \
-- torchscript \
--model bart \
--scripted-model-file manifold://cair_models/tree/cairaoke/experimental/test/script_debug.pt \
--no_cuda \
--input "Don't have a cow, man!"

but the command above worked(the output of scripted and unscripted model matched and it did not crash). I will try to see if I can reproduce it and debug it. If not, I will add a flag as you suggested

[stephenroller]

Can we check which version of PyTorch that inference-fast thing was introduced in?

[stephenroller]

Also I don't know about --no-cuda

[pooyanamini]

Can we check which version of PyTorch that inference-fast thing was introduced in?

optimize_for_inference was introduced in Pytorch 1.9.0 . However, there are changes for it as recent as Pytorch 1.11.

Also I don't know about --no-cuda

This flag is enabled in the test also code

[pooyanamini]

@stephenroller these failures seem to be unrelated to the PR. can you please check if any other changes are needed to the PR?

[github-actions]

This PR has not had activity in 30 days. Closing due to staleness.

[pooyanamini]

@stephenroller , @EricMichaelSmith it seems that finally the CI checks are passing. Can you please take a look?