Introduces Masked Language Modeling for Transformers by gabrielspmoreira · Pull Request #780 · NVIDIA-Merlin/models

Alternative implementation of MLM as a transform for data loader

gabrielspmoreira

committed

Implemented SequencePredictMasked and MaskSequenceEmbeddings

gabrielspmoreira

committed

Fixed broken test

gabrielspmoreira

committed

Renaming method and attribute of SequencePredictMasked

gabrielspmoreira

committed

As when targets._keras_mask is set outside the model it is not cascaded through the blocks, changed the masking approach introducing ExtractTargetsMask()

gabrielspmoreira

committed

Added tests to train Transformers with Causal LM and Masked LM

gabrielspmoreira

committed

Fix on ListToDense() to deal with cases where last dim is None

gabrielspmoreira

committed

Ensures target ragged tensors are converted to dense and are one-hot. Fixed and added tests on the special input mask approach

gabrielspmoreira

committed

Fixed comments

gabrielspmoreira

committed

Making TopKMetrics work with preds and targets with rank > 2 (e.g. sequences of predictions). Added mask support to TopKMetrics

gabrielspmoreira

committed

Solved issues with graph mode

gabrielspmoreira

committed

Fixed support to Keras Masking in metrics, incluing the TopkMetric ones

gabrielspmoreira

committed

Fixes tests by making adjust_predictions_and_targets ensure targets has the same shape and dtype as predictions

gabrielspmoreira

committed

Adding test that checks if Keras masking is correctly considered by default keras losses and metrics (including our TopkMetrics and TopKMetricsAggregator)

gabrielspmoreira

committed

Fixed test on masking

gabrielspmoreira

committed

Makes SequenceMaskRandom available only to be used in the model (not in the Loader)

gabrielspmoreira

committed

Created base class SequenceTargetAsInput which is parent of SequenceMaskRandom and the new SequenceMaskLast

gabrielspmoreira

committed

Fixed tests

gabrielspmoreira

committed

Removing not necessary constant

gabrielspmoreira

committed

Removing not necessary constant

gabrielspmoreira

committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduces Masked Language Modeling for Transformers#780

Commits on Oct 9, 2022

Alternative implementation of MLM as a transform for data loader

Implemented SequencePredictMasked and MaskSequenceEmbeddings

Fixed broken test

Renaming method and attribute of SequencePredictMasked

As when targets._keras_mask is set outside the model it is not cascaded through the blocks, changed the masking approach introducing ExtractTargetsMask()

Added tests to train Transformers with Causal LM and Masked LM

Fix on ListToDense() to deal with cases where last dim is None

Ensures target ragged tensors are converted to dense and are one-hot. Fixed and added tests on the special input mask approach

Fixed comments

Making TopKMetrics work with preds and targets with rank > 2 (e.g. sequences of predictions). Added mask support to TopKMetrics

Solved issues with graph mode

Fixed support to Keras Masking in metrics, incluing the TopkMetric ones

Fixes tests by making adjust_predictions_and_targets ensure targets has the same shape and dtype as predictions

Adding test that checks if Keras masking is correctly considered by default keras losses and metrics (including our TopkMetrics and TopKMetricsAggregator)

Fixed test on masking

Makes SequenceMaskRandom available only to be used in the model (not in the Loader)

Created base class SequenceTargetAsInput which is parent of SequenceMaskRandom and the new SequenceMaskLast

Fixed tests

Removing not necessary constant

Removing not necessary constant

Commits on Oct 10, 2022

Trying to fix failing tests

Trying to invalidate compile-cache when pre is provided

Fixed tests

Merge branch 'mlm_alt' of github.com:NVIDIA-Merlin/models into mlm_alt

Merge branch 'main' into mlm_alt