[RMP] Tensorflow support for session based recommendations integration in Merlin

[viswa-nvidia]

Problem:

Session-based and sequential-based models are an active research area for providing personalized recommendations. Transformers4Rec library was built to support the definition of such architectures and the results of our experiments conducted in the T4Rec paper showed the effectiveness of Transformers in modeling short sequences observed in session-based tasks. The T4Rec library was also used to win various RecSys challenges. We also observe a growing interest and active engagement from customers in using Transformers4Rec.

T4Rec was not actively updated for several months as the team shifted its focus to developing the Merlin Model library. MM does not currently support sequential and session-based recsys architectures. MM should support these sequential architectures and provide all necessary support to our users so that they can build such effective models.

Goal:

• Port the Transformers4Rec TF API to MM.
• The main blocks are: Masking, Transformer, RNN, and NextItemPredictionTask
• Provide an example that demonstrates < @jsohn-nvidia to clarify>

Definition of Done

Have an example that serves TF session based model in conjunction with a NVT workflow where the session based models scores the whole catalog

Constraints:

• T4Rec TF API is not as stable and complete as torch API. The main missing points are:
  - Two out of the 4 masking classes are missing: PLM and RTD
  - Support for training techniques embedding in HuggingFace trainer class: multi-gpu, early stopping, checkpoints saving...
  - Conduct experiments with real-world datasets (like the one conducted in the T4Rec paper with the Pytorch API)

Starting Point:

Training

Proposed API:

inputs = InputBlock(
  schema=con_schema + seq_schema,
  post=BroadcastToSequence(con_schema, seq_schema)
)
model = RetrievalModel(
  XLNetEncoder(inputs, n_head=4, n_layer=2),
  CategoricalOutput(inputs.select_by_tag(Tags.ITEM_ID))
)

topk = TopKEncoder(model)
topk.evaluate(...)

loader = mm.Loader(
  dataset, 
  batch_size=1000,
  transforms=PredictMasked(seq_schema, target=Tags.ITEM_ID)
)

model.fit(loader)

Inputs

Add support for sequential-inputs & shared embeddings
@edknv, @oliverholworthy & @marcromeyn

• [Task] Allow for pre-filtering of features a branch needs in a ParallelBlock models#674
• [TASK] Add select_by_tag to ParallelBlock models#694
• [TASK] Add capability to EmbeddingTable to hold multiple features that share id-encoding models#697
• [TASK] Add from_dataset & to_dataset to EmbeddingTable models#696

Masking

Add training strategies for sequence models
@gabrielspmoreira

• [TASK] Extract unrolling logic out of Aggregation into Unroll & make it mask-aware models#713
• [TASK] Implement PredictNext, PredictLast & PredictRandom models#691
• [TASK] Implement PredictMasked (BERT-like masking) models#692

RetrievalModel

Make RetrievalModel more generic to session-based use-cases + allow encoders being served
@marcromeyn

• [TASK] Add EncoderBlock models#698
• Refactor RetrievalModel to support Session-based models models#729
• Refactor utility that ensures a Dataset passed into embedding-extraction contains the right rows & features models#730

Outputs

Improve model-outputs to handle session-based recsys
@marcromeyn & @sararb

• Introducing Outputs-V2 models#715
• [TASK] Add TopKEncoder models#718
• Add from_dataset & to_dataset to CategoricalPrediction & ContrastivePrediction models#731

Port Sequence Architectures

Add sequence-encoding blocks like transformers
@sararb

• [TASK] Add Huggingface transformers as a backend models#719
• Port LSTMBlock models#732
• Port GRUBlock models#733
• Port TransformerBlock models#721
• [Task] Add RNN-based model apis for session-based recommendation models#819
• [Task] Add LSTM and BiLSTM unit tests for session-based recommendation models#818

Fixes for the GTC tutorial on session-based recommendation

• Fixes lower train metrics when using Keras Masking (SequenceMaskRandom, SequenceMaskLast) models#983
• Fix BroadcastToSequence to enable context features in sequential models models#991
• Renaming ProcessList to PrepareFeatures models#992
• Fixes support of sequential continuous features for sequential and non-sequential models models#969

Inference support

Save schema on model save

• Session-base model can be used as a candidate generation model, for that we need the following options:

• Define NextITemPredictionTask as a sub-class of RetrievalModel.Note: Make sure we can export the encoder block (Transformer or RNN) together with a SequenceSummary post layer as the query tower. models#736
  Note: Make sure we can export the encoder block (Transformer or RNN) together with a SequenceSummary post layer as the query tower.

• Export the item embeddings table for the ANN index models#735

• Constraint: Provide information about input list features to Merlin System [RMP] Establish a metadata standard for serializing information about Merlin components #489

Systems

• Investigate what work is required in systems side for serving TF session based models. What kind of SLA do we need for practical performance. Under what batch size/data. systems#202
• Ragged column support:
  • Remove explicit DictArray reference from merlin.core.dispatch core#163
  • Introduce and apply a basic version of the Column abstraction systems#223
  • ...

Documentation

• Review API documentation for TF based session based recsys #905
• [Task] Add/Fix the doc strings for TF based session based API documentation models#1078

Examples

Small scale (see #352)

• (WIP) Switch ragged multi-hot columns with two Triton arrays to a single ColumnSchema systems#173
• [FEA] Add list column support in Merlin Systems systems#135 ( Note: Ability to server session based depends on this work )
• [Task] Transformer-Based End-To-End Example with TensorFlow Merlin Models models#734

Blocker

• [INF] Unresolved architectural decisions
• [BUG] Model schemas and serving signatures don't align with each other models#924

[EvenOldridge]

@sararb @gabrielspmoreira can you flesh this out as best as possible in @marcromeyn's absence.

[EvenOldridge]

@gabrielspmoreira what does the architecture look like for the system for session based? Are we planning to use session generation to feed into a candidate generation stage?

[gabrielspmoreira]

@

@gabrielspmoreira what does the architecture look like for the system for session based? Are we planning to use session generation to feed into a candidate generation stage?

@EvenOldridge the session-based recommendation works as a next-item prediction task. It can be seen as a retrieval model, where the query tower users a sequential model (e.g. RNN, Transformer) and outputs a query representation/vector. During inference, such vectors can be used to retrieve the similar items from ANN the same way a retrieval model does.
So we believe we wouldn't need anything special on Merlin Systems related to the output of a session-based recommendation model. The main different is in the input, as such sequential models expect list features as input. NVTabular already supports processing and storing such list features as you know, but there might be some challenges on Systems building the Triton ensemble with list features support.

[viswa-nvidia]

@karlhigley , is the systems section here updated. Pleaes review

[karlhigley]

It is up to date with the current state of our knowledge of the work

[viswa-nvidia]

@marcromeyn , in one of the meetings, I made a note that this task is dependent on some tasks covered in RMP479-EMBEDDINGS initiative [RMP] Enable users to pass embedding tables directly into the input block in order to more easily support new functionality (non-trainable embeddings, different dimensions, model parallel, etc) . Is this correct ? which are these tasks. ? @EvenOldridge for vis.

[viswa-nvidia]

@rnyak , please link the systems - multi hot related development

[viswa-nvidia]

@oliverholworthy , please add the input output schema related tickets to this ticket

[oliverholworthy]

@viswa-nvidia For the saving method:

This is the parent issue for that:

• [FEA] Save input and output schema when .save methods are called on models models#669

We have implemented the save method to save input schema, but currently missing output schema.

[oliverholworthy]

Also identified an error in Merlin Models that may impact ability to serve Transformer-based models that affects issues with saving a model after loading.

NVIDIA-Merlin/models#878

[viswa-nvidia]

@karlhigley , please add the Triton related PR ( serving signatures haven't matched up with what model expects ) in the ticket.

[viswa-nvidia]

@rnyak to follow up with @radekosmulski for blocker ( 23.04 )

[EvenOldridge]

It was pointed out to me that we should be consistent about when we consider something done or not so I'm going to reopen this and move it to 22.05. @bbozkaya You've had the only remaining ticket (review the API) on your todo for the past two weeks with no progress. Is this something you're able to take on so that we can close the ticket. If not let us know and we can reassign.