Allow participants to prioritize statements

[colinmegill]

The issue of prioritization of statements in Polis has long been considered but never implemented. This is a thread to discuss a possible implementation.

Goals:

• Prioritization would feed into group informed consensus, de-prioritizing "the sky is blue" statements (all groups agree, but insignificant)
• Prioritization would feed into statement prioritization when showing statements to users
• Prioritization would surface how different groups value different issues

Proposal:

• A checkbox on each statement which the user (participant voting) would need to check before agreeing, disagreeing or passing
• Checkbox allows users to flag statements as high priority
• We assume there is no need for low priority as the interface allows a pass
• This functionality would be disable-able in admin config

[ ] this issue is high priority to me

Prioritization matrix

This creates a second comments * participants matrix, in addition to the votes matrix. The value of each value in the second prioritization matrix is, initially, a 1 or a 0 for 'given user did prioritize given comment'.

Upon analysis, however, we might consider: total statements total votes and total uses of prioritization as a means of scaling each user's row in the prioritization matrix. We would then variously apply penalties for overuse given ratios between the three values given some equation (which should be able to approach abs(-1) but never go below it, as the values in the votes matrix are 1 & -1 and priority should not be able to reduce their weight, only increase it - though we might consider whether we factor passes in to decrease the weight). We might consider:

Naive:

priority_ij = arbitraryScalingConstant / totalUsesOfPrioritization + 1

Perhaps better:

priority_ij = (totalParticipantVotes / totalStatements) * (arbitraryScalingConstant / totalUsesOfPrioritization) + 1

let arbitraryScalingConstant = 100
let totalUsesOfPrioritization = 2

(600 votes / 600 statements) *  (100 / 2 prioritizations) + 1 = 51
(300 votes / 600 statements) * (100 / 2 prioritizations) + 1 = 26
(10 votes / 600 statements) * (100 / 2 prioritizations) + 1 = .0166 * 50 + 1 = 1.83

let totalUsesOfPrioritization = 150

(300 votes / 600 statements) * (100 / 150 prioritizations) + 1 = .5 * .666 + 1 = 1.333

Possible improvements:

• Highly penalize certain totalParticipantVotes to totalStatements ratios such as if user has voted 3 times and there are 1000 statements
• Similarly don't heavily reward extremely high vote counts ie., voting on half the comments should probably be about as good as voting on all of them
• The difference between 559th and 560th vote should not be nearly as important as difference between 3rd and 4th vote
• If the number of totalUsesOfPrioritization is very low relative to the totalParticipantVoteCount, we might also find a way to upweight that, but perhaps we're already effectively doing that by watering down other cases

Once final values were computed, the prioritization matrix would then be used to multiply the votes matrix to surface statements various opinion clusters found significant.

Special thanks to @DZNarayanan and @misscs

[patcon]

Also, does this dovetail with crowd moderation #120?

[colinmegill]

This will be handled in 'importance', which is specified in another issue. I will fetch the id...

…

On Sun, May 3, 2020, 12:30 PM Patrick Connolly ***@***.***> wrote: Also, does this dovetail with crowd moderation compdemocracy/polis#120 <https://github.com/pol-is/polis-issues/issues/26>? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <https://github.com/pol-is/polis-issues/issues/123#issuecomment-623137023>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANQGGN7LFBPGCTYFQCMYCLRPWL3BANCNFSM4H6HRKUA> .

[colinmegill]

#133 is closed.
#120 is also an idea we've abandoned to prevent groups from punishing each other's ideas, in favor of comment routing.

[AleksandarPetrov]

Another idea for priority calculation would be to scale depending on whether a person uses more or less prioritization than other people, e.g.

priority_ij = (totalParticipantVotes / totalStatements) * (averageUsesOfPrioritizationAmongAllUsers / totalUsesOfPrioritization) + 1
(variations are possible)

[jucor]

Delicious idea. I think we might be getting to a point where it might make sense to explicitely formulate a latent variable model, which could capture individual's personal use of prioritization and take it into account. Thinking alongside the lines of Blei 2014.

These latent variable models range from low-rank matrix factorization (recommender-system style, a-la Netflix/Spotify, with a single matrix), to more elaborate membership models.

The added advantage is that it lends itself nicely to add the clustering as part of the latent variables.

And because it's Bayesian, there should be a way to do some online update of the posterior without having to recompute everything each time there's a new vote.

It's a chunk of work to formalize it, but I'd be game to try. It should surface "naturally" the kind of formulas mentioned in the comments above, from the form of the posterior distributions corresponding to whichever structure we bake into the model.

[ThenWho]

Do you need help @jucor ? Not my area but always good to look a bit further.

I thought I mentioned it but apparently not here: in tensors, priority would be just another slice, next to the agree/disagree/pass slices. It's just another type of information really 🙂 , orthogonal to normal voting but combines nicely with everything.

As as I said, not my area, but I wouldn't be surprised if it ended being very similar. So excited you're going to look into this! 😊

[jucor]

Thanks Giorgios! Always happy to geek out with others :) Agreed with you, there should be deep connections between both approaches. I need to put pen on paper to formalise a bit -- this week is the conference of Fairness, Accountability, and Transparency, so I don't have 100% time available, but it keeps buzzing in my mind :) Let me get back to you once I have the basic equations written down in latex, then we can iterate.

[ThenWho]

No worries 🙂 If you plan to work in the open, feel free to drop some URL so that me and others can follow up. Otherwise let's sync down the road! 🙌

[jucor]

Happy to work in the open for a good part :) The more eyes, the shallower the problems!

I would normally use Overleaf and Latex, but not sure how good they are for open view, nor how they match what's used by the pol.is community. Any suggestion of tool? Does pol.is have a preferred one that supports equations? @colinmegill ?

[colinmegill]

@jucor I finally have an answer here :) DeepNote is really nice.

[jucor]

Oh you do?? Answer as in:

• method to compute it importance, or
• user-based definition of importance, or
• "Does pol.is have a preferred one that supports equations? " -> yes and it's Deepnote :)