Model Suite: Future Work

[PaliC]

• Numeric stability support over some set of iterations (ie. compare kernelized model against eager for 1000 iterations of training)
• Performance testing (jiannanWang/BackendBenchExamples) is a great reference point
• Allowing folks to run custom ops for the model and just testing end to end correctness / performance (we will need to think this through more thoroughly to ensure correctness). Effectively this is how we'll end up handling fusions later on.
• More models. The two main ones are one that uses all of the backwards passes we currently support and a second is a norms one.
• Support loading models from huggingface
• Better integrate correctness results with our logging
• Add a simple profiling script where given the model + args, it autogenerates the forward and backward ops for the config.