Learning-to-rank is a ranking task that takes fixed-size data with numerical features as input, and predicts the relevance of each input instance. These relevance scores are then used to rank the items from highest to the lowest. The ranking models used to define the relevance of the items are usually implemented as either gradient boosted trees, multi-layer perceptrons and tranformers, and can be categorized as pointwise, pairwise or listwise. The pointwise approach frames the ranking task as a simple regression or classification of the true relevancy for input entities regardless of possible interactions between them. In the pairwise approach, pairs of input items are treated as independent variables and the loss functions is optimized around the preference among the pairs. Finally, the listwise approach computes the loss based on the scores of the entire input list.
However, previous research has shown how widely used loss functions are not well connected to the actual Information Retrieval (IR) metrics used for evaluating their performance. That being said, popular metrics like Normalised Discounted Cumulative Gain (NDCG) and Mean Average Precision (MAP) present the challenge of not being differentiable, making them not suitable for being loss functions. This challenge has been subject of different researches which contributed to the development of differentiable approximations of such metrics that can be used as loss functions in Learning-to-rank tasks. On top of this, in recent years, the progress in the training of neural networks introduced an additional perspective of effectiveness and scalability of deep network architectures applied to Learning-to-rank. Therefore, understanding the behavior and the performance of more suitable loss functions combined with more sophisticated neural models is a topic that should not be overlooked.
Another topic of concern for Learning-to-rank is the influence that the used datasets have on the quality of a model when inconsistency and bias are present. As the performance of supervised Machine Learning models is highly dependent on the quality of the training dataset, it is important that the used data presents consistency in the labels assigned to similar items. This translates to Learning-to-rank as well, where previous research has shown how inconsistency in the used datasets can negatively impact the performance of the model. On top of this, the bias present in the datasets cannot be ignored either. As Unbiased Learning-to-rank techniques have been introduced to mitigate the position bias and the trust bias, some limitations are still present. For example, it has been shown how the bias related to other display information has been overlooked. This can be seen as a consequence to the fact that most existing datasets used for Learning-to-rank lack real-world user feedback. Therefore, it is fundamental for this work to be aware of the potential bias and inconsistency a dataset can carry over, which can bring challenges when trying to generalize a model over different scenarios.
The aim of this project is to present and discuss an empirical comparison of the performance of different loss functions against different ranking metrics, using the same underlying neural network architecture. On top of this, the experiments are run on different datasest to inspect if the eventual presence of bias and/or inconsistency can affect the loss functions performance. Thus, the following research questions will be answered:
- What is the best loss functions for Learning-to-rank applied to a neural network for each considered metric?
- Can the results of one dataset be generalized to other datasets?
For a full report of the project see this document.