[💡SUG] Remove Duplicated User and Item Interaction

[rowedenny]

Is your feature request related to a problem? Please describe.
In some cases, we would like to merge the duplicated user-item interactions by keeping the earliest one (referring to the timestamp). The rationality here is to test the performance of a method in recommending novel items that a user did not consume before.

Describe the solution you'd like

1. It is assumed to be an optional function, since it may not be universally true for the general recommendation system;
2. Looking through the implementation of Dataset, I think to add a new function within data_processing is one alternative solution. Here is one example,

def _remove_duplication(self):
    self.inter_feat = self.inter_feat.sort_values(by=[self.time_field], ascending=True)
    self.inter_feat = self.inter_feat.drop_duplicates(subset=[self.uid_field, self.iid_field], keep='first')

Describe alternatives you've considered
None

Additional context
None

[hyp1231]

Thanks for your suggestion!

If you use the 27 benchmarks we have collected, we have put the deduplication in the process of generating atomic files, with the optional argument --duplicate_removal. Detailed, we merge repeated interaction records, and record the latest timestamp and the number of repetitions. Besides, you can also download the processed atomic files in different processing strategy (not_merged and merged).

If you want to use your own dataset, you can also generate atomic files in this way.

We will also add this feature in future versions.

[rowedenny]

First of all, thanks for the quick response. That is exactly what I am looking for.

May I suggest that record optionally like pandas.drop_diplicates, either keep the first or the last?
Referring to:
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.drop_duplicates.html

[hyp1231]

No problem for the new feature of _data_processing, and that's also we'd like to see. 😃

[hyp1231]

We have made a improvement (refer to #492 ), which removes duplicated interactions if arg rm_dup_inter is not None.

Note that If arg TIME_FIELD exists, interaction records in inter_feat will be sorted by values of inter_feat[TIME_FIELD]. Otherwise it will remain unchanged. After that, if rm_dup_inter == first, we will keep the first user-item interaction in duplicates; if rm_dup_inter == last, we will keep the last one.

Thanks for the insightful suggestion. I hope this can help you.