Add semi-automated DAO reporting to website

[m52go]

Proper DAO reporting is long-overdue, but probably won't be implemented as originally envisioned (fully automated) for a while.

Hence this initiative to add semi-automated reporting, which takes place at intervals (e.g., per-day, per-cycle) instead.

Mid-term workaround

Reporting BTC+BSQ burning, qualitative issuance, BSQ supply, and more in real-time is not a trivial task.

However, thanks to the cycle results JSON dump provided by Bisq itself, reporting basic DAO voting cycle results in a semi-automated way is quite simple. By "semi-automated" I mean creating a script that parses the JSON dump -- this script will need to be run manually after cycles end, but can write most of the report automatically.

Here's what I'm currently working on to get this done:

• write script to generate YAML markup for a custom Jekyll collection for Bisq DAO cycles (obtain data, create new page, write report)
• redirect old dao cycle reports to new dao cycle reports
  • dao cycle reports are currently blog items (/blog/cycle-N-results), but after this change, they will become their own custom collection type available at (/dao/cycle-N)
• overhaul /dao page
  • remove all existing content except "Want to learn more?" links at the bottom
  • add list of automatically-generated dao cycle reports
  • see mockup here, but just imagine that each cycle section will be a link to a dao results post instead of an expandable section (having all cycle data details listed on one page doesn't make sense -- too many cycles)
• remove /stats page
  • move volume graphs to /markets page
  • move bsq supply graph to top of new /dao page

I'll have the script leave blank spaces for proposal commentary and general commentary, both to be filled in by hand wherever appropriate.

This workflow will make DAO reporting more punctual, as I won't have to spend time getting simple data points such as DAO cycle block numbers, time periods, voting statistics, and other items. Currently these numbers must be obtained from Bisq itself and a bunch of script hacked together over time, which makes doing this by hand tedious and prone to pushing off.

Next steps

Once this basic DAO result reporting is completed, the next step will be to add supply dynamics (issuance details, BSQ burn, BTC burn, etc) to help put DAO results into context.

• issuance details will require GitHub issue parsing (partly already done)
• BSQ burn will require BSQ explorer polling
• BTC burn will require BTC address monitoring (method TBD)

Whereas voting cycle results are only released once per cycle and require hand-crafted commentary, fee collection (both BSQ and BTC) happens constantly and needs to be counted in an automated way. Thus this next step will require more technical work and probably some kind of intermediate server that aggregates data on intervals. Depending on the approach used, daily time intervals might work, otherwise, a cycle-based interval may need to be used.

Details are TBD, and will be addressed in a future issue.

Future

As mentioned above, the long-term hope is for mempool nodes to entirely take over and provide DAO reporting. They are technically in the best spot to do so since they already have real-time BTC and BSQ data, so implementing this any other way is a compromise. But it's past time to get a handle on DAO reporting, so this compromised approach will have to suffice for now.

I'll have a pull request soon for the items covered in the mid-term workaround section, as I don't see those changes as controversial (they're merely moving around data that's already on the website and changing a behind-the-scenes workflow).

I'm making this issue to outline the approach and background to put that pull request into context and how it could fit within a broader initiative to achieve better overall reporting for the Bisq DAO.

Please let me know if you have any suggestions or concerns.

[m52go]

Overview of Planned Work

This post outlines the conceptual overviews, data sources, and planned work for 2 pages: a new /dao page and /dao/cycle-n pages with cycle details.

The goal is to auto-generate all items below (exception: proposal descriptions, since those are necessarily subjective and reflective). It assumes updates will take place monthly right after the end of a cycle. Most data should be available locally (e.g. Bisq data dumps), but GitHub and Mempool may be needed for some things...the idea is to get all the data, process it, auto-generate pages as outlined below, add proposal commentary (which would be optional, strictly speaking), and push to GitHub.

The /dao page outlined below should replace the page currently at /dao as well as the current /stats page.

/dao - Conceptual Overview

This overview will be wireframed shortly for design feedback (see task list below).

Graph (single graph with all 3 data series)

• bsq supply (solid main line)
• altcoin volume (dotted line)
• fiat volume (dotted line)

Tiles for each cycle

• cycle number
• timeframe (blocks)
• timeframe (calendar)
• link to cycle-n page for more details

Learn more about the DAO

• link section much like the existing one at /dao

---

/dao/cycle-n - Conceptual Overview

This overview will be wireframed shortly for design feedback (see task list below).

Trading Overview

• volume (+ number of trades) by market
• volume (+ number of trades) by payment method

Network Funds Transfers

• Supply Decreases
  • bsq burned
    • types of transactions (number + sum)
  • btc fees
  • total equivalent supply decrease
• Supply Increases
  • compensation
    • compensation by function
    • compensation total
  • reimbursement
    • reimbursement total (sum + number of reimbursements)
  • total bsq issuance
• Net Supply Change

Governance

• Proposals and Voting, At a Glance
  • number of proposals, number of votes, total voting weight
• Proposals, In Detail
  • proposal details + outcome

---

Task List

What needs to be done...

UI

• Wireframe interface for new DAO status page (/dao)
• Wireframe interface for DAO cycle details pages (/dao/cycle-n)
• Get feedback from pedromvpg and others on UI

Obtain Data

• Establish reliable tools to obtain data for proposed interface (enumerated below)

Plug Data into Interface

• Generate /dao and /dao/cycle-n pages programatically

Document

• Make it easy for others to run the scripts and create reports for new cycles

---

Data Sources

Where will the data come from?

/dao

• BSQ supply (may be easiest to parse current supply from https://mempool.space/bisq/stats, then add back interim burn from existing script)
• Fiat volume (use existing script)
• Altcoin volume (use existing script)
• Cycle numbers, blocks, and time frames (script needed; best data source is probably bisq's dao json dump)

/dao/cycle-n

• Volume by market (use existing script)
• Volume by payment method (use existing script)
• BSQ burned in cycle, types of transactions, etc (use existing script)
• BTC fees collected in cycle (script needed; see discussion at the end of this thread for further details: Running the numbers on Bisq DAO revenues bisq#5171)
• Compensation by function (https://github.com/bisq-network/growth/tree/master/reporting/compensation-bot works, but may have reconciling issues)
• Total compensation and total reimbursements (script needed; best data source is probably bisq's dao json dump)
• Total BSQ issuance (easy to get from bisq's dao json data dump)
• DAO cycle figures (also in the dao json data dump)
• Proposals in detal (will need to be written by hand; only manual part, optional)

[pazza83]

Hi @m52go looks great.

On the /dao/cycle-n page

Do you think it would be possible to pull in the following:

	USD	BTC
Trade volume July 2021	$X	X BTC
BSQ trade revenue fees	$X	X BTC
BTC trade revenue fees	$X	X BTC
Net Profit (Loss)	$X	X BTC

Percentage of fees paid in BSQ was: XX.XX%

When doing it manually I was able to pull in data on a monthly basis but found in harder to pull in the correct data for a cycle.

[m52go]

Merged with bisq-network/bisq-website#449.