Implement non-accuracy metrics

[abellogin]

Like novelty, diversity, etc.
We should try to involve someone else that already measured them (e.g., Saúl Vargas @saulvargas)

Milestone: later than 0.5?

[alansaid]

If @saulvargas has the time to spend on this, it would be great! 👍

[alansaid]

I think we should strive to have some basic metrics, e.g. ILS earlier than 0.5. Definitively no later than 0.5 at least.

[saulvargas]

Hi! I'd be happy to contribute. Implementing the basic version of some common*** diversity and novelty metrics is easy and straightforward. I propose to implement the following:

• Intra-list distance (ILD), which is basically 1 - ILS (so that higher numbers are better results).
• Profile distance (PD), which measures the average distance from the recommendations to the items of the profile.
• Expected popularity complement (PC), i.e., the average probability of the items in the recommendation not to be known from the users in training.
• Other system-level metrics that consider the RS as a whole (they don't define a user-level score): aggregate diversity (Adomavicius and Kwon), Gini Index...

[alansaid]

@saulvargas awesome! 👍
Please do the following:

1. fork the project
2. do your magic
3. submit pull requests
4. go to 2

[abellogin]

Btw @saulvargas, if during this process you have any suggestion to improve the evaluation module, please, let us know and we can discuss about it (a comment here should be enough).

[alansaid]

@saulvargas any progress here? Can we (me and/or @abellogin) do anything to help you?

[saulvargas]

Done! I implemented the basic version of some novelty and diversity metrics, that is, without relevance or rank discount. If you were interested in these functionalities, then we would have to talk...

[alansaid]

👍

[alansaid]

Current iteration closed in 47b2347

[saulvargas]

I would like to release an implementation of the "binomial diversity" metric of the paper in RecSys 2014 (http://ir.ii.uam.es/saul/pubs/recsys2014-vargas-tid.pdf). Do you think it could be included in RiVal?

[alansaid]

Sure, go for it.
I think that for more esoteric/non standard metrics it would make sense to update the documentation/wiki Evaluation Metrics

[alansaid]

I suggest you open an issue with the metric you want to implement and assign it to yourself. If you have timeline for when you think you might have it done, assign it to one of the milestones that fits your time estimate

[abellogin]

Awesome! Besides, this would give more publicity to the framework, which is great ;)