Collective statements

[colinmegill]

DeepMind has dropped a paper which cites as inspiration the "Opportunities and Risks of LLMs for Scalable Deliberation with Polis" https://arxiv.org/abs/2306.11932 paper.

https://www.science.org/doi/epdf/10.1126/science.adq2852

This issue is a stub that will be expanded, but the tldr; is that it's exciting to think about applying this clusters not just to an entire population (like an entire citizen assembly) but to each cluster (opinion group statements) and statements which rank high on group informed consensus (collective statement from entire population). This would definitionally increase the scores and metrics DeepMind measures against here.

[colinmegill]

This could be further broken down by topic.

[colinmegill]

Collective Statement Example: SJI Landbank

Here's a trivial example of generating collective statements from polis data using GPT4o.

Files:

1. Summary: https://github.com/compdemocracy/openData/blob/master/ssis.land-bank-farmland.2rumnecbeh.2021-08-01/summary.csv
2. Comments: https://github.com/compdemocracy/openData/blob/master/ssis.land-bank-farmland.2rumnecbeh.2021-08-01/comments.csv
3. Handwritten report (included to increase quality since we already have it, obviously this wouldn't be available, arguable that this simulates some degree of 'supervision' by participants or a facilitator, since the goal here is just to get some/any example of collective statements) https://sjclandbank.org/wp-content/uploads/2021/10/Final-Report_-Public-Discussion-on-the-San-Juan-County-Land-Bank-2.pdf
4. Screenshot of groups from pol.is report
5. Screenshot of top group informed consensus statements from pol.is report

Statements which make groups unique

Statements with high group informed consensus

Prompt (generated by GPT4o1-preview)

Task:

Using the provided handwritten pol.is report and pol.is data, generate "collective statements" for each identified pol.is cluster, as well as an overall informed consensus statement for the entire pol.is group. These statements should be written as if they are being presented to a governing authority, reflecting the collective voice of the population.

Instructions:

Data Analysis:

Carefully review the pol.is report and data.
Identify the key opinions, sentiments, and themes within each cluster.
Note the main concerns, priorities, and suggestions expressed by participants.
Cluster Statements:

For each cluster, draft a collective statement that encapsulates the group's viewpoints.
Requirements for each statement:
Accuracy: Reflect the true sentiments and opinions from the data.
Clarity: Use clear and concise language.
Formality: Maintain a respectful and formal tone suitable for communication with a governing body, but remain authentic to the substance and tone of the comments themselves and the voices of the population. Always err on the side of representing the population, over formality. 
Perspective: Write from the collective "we" perspective of the population.
Overall Consensus Statement:

Identify common ground and areas of agreement across all clusters.
Draft an overall consensus statement that represents shared views and collective recommendations.
Emphasize points with broad support significant for policy considerations.
Formatting and Tone:

Use formal language appropriate for addressing a governing authority.
Ensure statements are objective and free from personal biases.
Present the information in a structured and organized manner.
Final Review:

Verify that all statements accurately represent the provided data.
Ensure the tone and content are appropriate for the intended audience.

Output: Collective Statements (generated by GPT4o)

Group A (266 Participants):

"We, as a collective, believe that the San Juan County Land Bank is a critical institution for preserving the islands' natural beauty and heritage. Our confidence in the Land Bank stems from its commitment to community engagement, responsible land management, and transparent governance. We appreciate its efforts in listening to the needs of the community, as evidenced by its proactive stewardship and conservation practices. We see the Land Bank as a vital entity for the future of San Juan County, ensuring public access to preserved spaces and supporting ecological balance. Though there may be areas for refinement, we believe the Land Bank is responsive and continues to honor its original mandate."

Group B (92 Participants):

"We collectively express concern that the San Juan County Land Bank has strayed from its mission, becoming an entity lacking transparency and effective management. Our group believes that the institution has failed to manage its properties efficiently and lacks accountability to the public it serves. We feel that the Land Bank should adopt a more community-oriented approach, with increased clarity in its operations and decision-making. Additionally, we are concerned about its operational structure, which we perceive as detached from the original conservation and stewardship goals. To restore trust, we call for a reassessment of the Land Bank's policies and management practices."

Overall Consensus Statement:

"As residents of the San Juan Islands, we acknowledge the San Juan County Land Bank’s pivotal role in preserving and protecting our unique natural landscapes. Across diverse perspectives, there is broad agreement that the Land Bank should operate with the highest level of transparency and accountability to foster trust and community support. The majority of us value both conservation and public access, supporting a balanced approach that prioritizes environmental stewardship while allowing for responsible recreational use. We endorse ongoing efforts to protect native ecosystems, safeguard water resources, and consider soil health in property management. Additionally, there is strong consensus on the need for the Land Bank to incorporate diverse community voices, particularly through greater representation on its commission. While we recognize the positive impacts of the Land Bank, we also encourage continued dialogue and improvements to better align with the evolving needs of our community."

[colinmegill]

Polis' Visualization

Polis' visualization was built with the intention of reflecting groups' opinion back to each group and to other groups, and to highlight consensus in the context of division. Here are two examples, again from the SJI Land Bank conversation. The intention of the visualization was to surface "worldviews" of each group and of a population, in the form of a set of statements.

Given a list of statements (as in the pasted images in the report, above), it's the responsibility of the reader to "reconstruct" a worldview for a group, and for the consensus of the population.

Consensus

Group Divisions

[colinmegill]

A worthy experiment related to the Science paper above could be replacing the visualization with collective statements which can be rated by groups. Each participant could rate a collective statement about their own group, and see a statement about other groups.

Here's a low fidelity example, with intentionally silly button language.

Alternatives to the buttons, which might be familiar to voting, or might be too close to voting:

• A 0-5 star rating ⭐️⭐️⭐️⭐️⭐️ could also be used, as it immediately implies quality rating.
• "clause knock out" in which the participant could highlight just a part of the summary and invalidate it.
• "free form text response" in which the participant writes freeform back to the group summaries
• Replacing the voting interface itself with rating a collective statement, and doing this regularly, as clusters change and new groups emerge and statements improve. Perhaps once every 15 votes, or something like that, would be a good starting point and could be optimized through testing whether and when it makes people stop voting. Disadvantages seem clear: don't interrupt the critical path, don't take people off task, don't confuse people with different tasks
  • One intuition is "growing" what is agreed disagree pass on. agreeing disagreeing passing on a larger chunk could be preserved at the individual level as data, even if the clusters changed.
    • this might be amenable to hclust... agglomerative... annealing, at each step you're expanding it by trying to find a more broadly receptive version of it...
• Instead badger Community Notes to implement this experiment
• Just do the group informed consensus statement to start
  • this is much more amenable to this approach because it doesn't matter if the groups split, the existence of groups... could instruct this. Same for topics.
  • a group informed consensus statement, that people rate with stars, PER TOPIC, seems likely to avoid most problems.

[colinmegill]

Rated collective statements could be seen as a separate, and complementary, deliverable to #915, just as the report has always been distinct from the visualization, with different goals and different audiences. Delivering these side by side stapled together would continue to try to strike a balance between emergent / informal / fast and formal / quantitative.

[raulsperoni]

Hey! We applied a version of this idea to the report on our latest conversation in Uruguay. This went about Social Security and a plebiscite to amend the constitution. The result of the plebiscite ended up being negative but the discussion will continue.

1. Web report
2. Full report

Those are in Spanish but browser translate should do the trick

We chose to describe the groups in the third person and got something like this:

Group A: Primary focus on fiscal responsibility (208)
Group A is made up of people who, while recognizing the challenges of the social security system, prioritize fiscal sustainability in any reform. This group supports a greater role for individual savings in supporting pensions and is skeptical of using referendums to solve complex social security problems. They advocate a balanced approach that respects individual contributions and maintains the financial viability of the system for future generations.

Group B: Main focus on equity (538)
Group B brings together people who recognise the urgent need to reform the social security system to ensure greater equity and sustainability. This group emphasises the importance of ensuring a decent minimum pension for all retirees and stresses that the retirement age should be adjusted to the diversity of occupations. In addition, they promote an inclusive debate involving academics, civil society and social policy experts to achieve fair solutions adapted to an ageing society .

Consensus
Overall, participants agree on the need for a reform that combines equity and sustainability in the social security system. There is broad support for establishing a minimum pension and adjusting retirement policies according to the particularities of each occupation. The importance of a transparent and well-informed public debate, allowing the participation of all sectors of society, is also emphasized. The general vision advocates a system that balances individual responsibility with social justice, to ensure a secure retirement for present and future generations.

Areas of Uncertainty
Despite points of agreement and disagreement between the groups, there are areas of uncertainty that raise questions among participants. These areas reflect issues where more than 30% of voters chose to "pass" when giving their opinion, suggesting a lack of consensus or clarity.

Among the issues of greatest uncertainty are the definition of the limits between a social security reform and a pension reform, the ability of individuals to save based on their income in order to contribute to the system, and the fairness of contributions to various professional funds. There are also doubts about how the charging of minimum wages for social benefits should be implemented and the equity in the unification of benefits from all funds.

These areas of uncertainty indicate the need for increased dialogue, information and education around these issues to help the community better understand the implications of each proposal and make informed decisions.

Closing the loop

We have definitely discussed how interesting would be to close de loop and actually ask participants if they recognize themselves in these groups and use that in a second round of conversation. We thought this as an in person exercise but ofc those interfaces look really good. As with other things the challenge would be to get participants to effectively return to the conversation once some time has passed.

[akonya]

We've been using a similar kind of system; where we take the statements with the highest max-min bridging agreement and then use an LLM to create a slate synthesized collective statements; with one statement capturing each of the unique topics. Then we have participants vote on those statements to validate how representative they actually are.

The "tooling" we built for this (early example here, specific to Democratic Inputs to AI stuff with Open AI) can handle doing the statement generation for both individual groups or participants overall, but we've largely focused on doing this across the whole universe of participants and focused on "common ground".

We used this type of pipeline to develop normative objectives for AI alignment, and it works really well at crisply articulating common ground; the final set of normative objective statements had >95% approval from participants. We've also been using a similar pipeline in peacebuilding work and it routinely generates statements that get >80% approval from people on both sides of active conflicts.

Few thoughts on where this stuff could go though beyond the awesome ideas above...

One direction is to focus on the loop closing and work back:

• Collective letter Start by saying "we want to end with a letter from [constituency] to [center of power] that everyone will sign" .. then design the collective statement pipeline towards that goal. This could mean trying to one-shot a collective letter, which is sorta the vibe of the Habermas machine paper. When we've done this in peacebuilding though, we've designed the system to generate a set of statements that can then be compiled into a letter. I think a cool extension of this collective dialogue -> collective statements -> collective letter pipeline would be to build it end-to-end including the final "signing" of the letter.
• Collective decision The thing I have most in mind here is DAO's, where there is a formal mechanism to ratify a decision. In that case, the idea would be to engineer the path to a 'collective statement' that took the shape of a collective decision that could be formally ratified. But you could imagine a version of this with a path directly into a referendum or something.

Or from a pretty output / way of interacting with the data you want to arrive at:

• The great slider A cool thing buried in the Habermas machine paper was this idea of defining a continuum in embedding space in terms of two statements. You could imagine doing this such that one end of the continuum is one "side" (eg learned cluster, or dimension if your using continuous bridging) and corresponds to a collective statement best representing each side. Then in the middle is a collective statement that best represents everyone. Now, because these statements define a continuum, you could then map a range of synthetically generated collective statements on that continuum. This could enable to things a slider where as you move it from one side to the other, the "collective statement" evolves in real time capturing where on the spectrum from "common ground" to "either side" you are. And if you want to capture more than one side you could move from a 1D slider to something like a cursor you move in 2D (eg like the existing opinion space shown on polis).

Or maybe just YOLO it all on synthetic?

• Synthetic bootsrapping Do a full e2e process for collective statement generation using just synthetic participants. Then just drop those collective statements in as seed statement and let er rip. Or just cut straight to a simple voting process on those collective statements.