Remove XXOutput loss operators

[eric-haibin-lin]

Description

These operators are often used with mx.module APIs. Removing them for mxnet 2.0. You can find equivalent loss functions in mx.gluon.loss namespace.

These APIs will still be maintained in the 1.x branch
#17676

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage:
• Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
• Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
• Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
• Code is well-documented:
• For user-facing API changes, API doc string has been updated.
• For new C++ functions in header files, their functionalities and arguments are documented.
• For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
• Check the API doc at https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
• To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

• Feature1, tests, (and when applicable, API doc)
• Feature2, tests, (and when applicable, API doc)

Comments

• If this change is a backward incompatible change, why must this change be made.
• Interesting edge cases to note here

[mxnet-bot]

Hey @eric-haibin-lin , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

• To trigger all jobs: @mxnet-bot run ci [all]
• To trigger specific jobs: @mxnet-bot run ci [job1, job2]

---

CI supported jobs: [centos-gpu, windows-cpu, miscellaneous, unix-gpu, windows-gpu, clang, website, sanity, centos-cpu, edge, unix-cpu]

---

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

[fhieber]

Will there be an efficient equivalent for SoftmaxOutput in 2.0 that also supports label smoothing? Reason being that the approach in GluonNLP was still quite a bit slower than SoftmaxOutput when evaluated last time for Sockeye.

[eric-haibin-lin]

@fhieber thanks for the inputs. Could you provide some reference latency number for what you evaluated before? Since MXNet now supports registering c++ operators dynamically without having to compile mxnet source library https://github.com/apache/incubator-mxnet/tree/master/example/extensions/lib_custom_op, it should be easy to copy-paste and compile the softmax operator code even if it's not inside the official repo.
@yzhliu @sxjscience does the numpy-nlp branch also uses softmax + label smoothing and have plan to include such an operator in the mx.npx namespace?

[sxjscience]

@eric-haibin-lin @fhieber I’ve written a new loss layer in the numpy API: https://github.com/dmlc/gluon-nlp/blob/b714eaccc67619d7bdcbd1574d30be87d9c73f0c/src/gluonnlp/loss.py#L4

The speed + memory cost is better than calculating the cross-entropy between softmax and smoothed label (the GluonNLP's original implementation).

[fhieber]

@eric-haibin-lin Thanks for the pointer. We had tried custom operators in a separate project (int8 quantization) but found them to be not suitable for performance reasons.
@sxjscience Thanks! I will take a look. I haven't benchmarked SoftmaxOutput and a Gluon implementation recently. Mid 2019 we still observed a ~15-20% slowdown when not using SoftmaxOutput. I will give your label smoothing implementation a try, but my guess is that the combination of Softmax forward pass, (smoothed) cross-entropy gradient backward pass, and support for easy masking without creating an explicit mask, all implemented in a single operator, will still have a slight edge over an implementation with multiple operators.

[marcoabreu]

CC @sandeep-krishnamurthy

Are custom operators actually causing a slow-down? Speak, the same code, one time directly compiled and the other one dynamically loaded are not creating the same performance?

[eric-haibin-lin]

@samskalicky is there any evidence that dynamic ops are slower based on your experience?

[samskalicky]

No, dynamic C++ custom ops are significantly faster than Python custom ops. I believe @fhieber and his team found dynamic C++ custom ops to be within 10% of built in operators. There are some overheads that are a result of our design choices (we chose easier implementation rather than speed). But our goal was never equal performance as built in operators.

[fhieber]

Just to follow up on this and provide some numbers (@eric-haibin-lin). I tried the label smoothing implementation by @sxjscience (thanks!) in Sockeye, making sure scaling and normalization is equivalent to how we currently use SoftmaxOutput.
While this is much faster than a naive implementation of label smoothing with dense, smoothed labels, SoftmaxOutput is still 5-10% faster than the optimized implementation.