politeia v1.4.0

A summary of this feature release is listed below. See the changelog for the
full list of changes.

• Importing legacy proposals. The majority of this release centers around
  importing the legacy git backend proposals into the tlog backend. A CLI tool
  was created to handle the required data conversions and to manage the import
  process. The import also required a number of downstream changes as well.
  Code was added that gives the sysadmin the ability rebuild the plugin caches,
  the caches were moved into the tlog backend database, the trillian version
  was brought up to date, and trillian tree management was added to politeiad
  in order to reduce trillian performance issues that can result from the
  number of trillian trees growing.

• Comment edits. Users now have the ability to edit comments during a set
  period of time after the comment was submitted. The edit period is a plugin
  setting that defaults to 5 minutes.

Note: politeiad requires a restart using the --fsck flag after updating to
v1.4.0 to reflect the changes made to the various plugin caches. The trillian
binaries must be updated to v1.4.1.

Changelog

This release consists of 60 commits from 3 contributors which totals to 242
files changed, 14866 additional lines of code and 4808 deleted lines of code.

[0ed0313] docs: Add v1.3.1 release notes.
[9e8c9a1] comments: Add 'allowextradata' to the policy reply.
[bd78867] comments: Allow retrieving all comment votes.
[b846b94] multi: Add routes page sizes as plugin settings.
[4439e3f] politeiad: Consistent record entries ordering.
[5f8e535] multi: Add temporary comment edits.
[cf53aa7] politeiad: Cleanup deprecated page sizes.
[2668d50] pictl: 'commenttimestamps' improvements.
[d968152] tstore: Allow plugins to use the key-value store.
[22a80f8] tstore: Unprefix returned keys from CacheGet.
[c54ff12] politeiad: Update trillian to v1.4.0.
[be19afb] comments: Cache final timetamps.
[094cff5] ticketvote: Cache final timestamps.
[e5d9b16] politeiawww: Add Records API documentation.
[38b63bd] pi: Add LegacyToken field to ProposalMetadata.
[f8c0449] politeiawww: Add plugin model.
[444cb2d] pi: Fix LegacyToken errors.
[14381e9] docs: Add plugin architecture doc.
[6c0d426] politeiad: Add legacypoliteia convert cmd.
[94dd965] mysql: Fix column name bug.
[b749291] tstore: Add tstore client init function.
[65bee18] mysql: Fix InsertUser identity table bug.
[321813e] politeiad: Add tlog package.
[af61db4] chore: Update github actions.
[18491bc] pi: Fix min start date setting bug.
[cff2406] politeiawww: Fix csrf error check bug.
[2a35645] pi: Fix plugin setting error messages.
[3830b89] comments: Fix fsck bugs.
[69d4b66] Update trillian version (1.4.1).
[58c77b1] pivoter: Correct unix build tags.
[bf02cf0] build: Drop io/ioutil.
[68b24c4] pivoter: Support SIGTERM on Windows.
[9446fc2] multi: Improve identity logging.
[950620c] mysql: Improve error handling.
[8eaa621] pivoter: Cosolidate config error reporting.
[2af8a2a] politeiad: Remove leveldb tstore config option.
[ca6fdfe] tstorebe/store: Allow put overwrites.
[67afd49] tstorebe: Improve fsck logging and docs.
[aea0011] comments: Fix empty comments bug.
[9366695] ticketvote: Add the timestamp to the vote summary.
[6f6bcc0] ticketvote: Move inventory cache to db.
[e5381ad] ticketvote: Move summaries cache to db.
[87860ed] ticketvote: Move runoff vote subs cache to db.
[34adbcb] ticketvote: Implement fsck.
[58d30e1] politeiawww: Remove legacy websockets.
[a893055] Cleanup git backend packages.
[a8bab40] politeiawww: Cleanup legacy packages.
[c1d6bbc] legacypoliteia: Add import command.
[d4ade66] Package and deadcode cleanup
[c196b7f] legacypoliteia: Handle trillian edge case failure.
[cae9d73] tstore: Improve blobs not found error.
[cba058e] dbutil: Improve migrate cmd.
[5bd85d2] Bump version to v1.4.0.
[ae75b0c] multi: Go 1.19 comment formatting.
[3237f4c] version: Support single override and add git hash.
[5e4c32a] build: Updates for Go 1.19.
[cf41fe8] sessions: Del expired sessions on startup.
[bc06896] sessions: Add db tests.
[ec4a9c8] dbutil: Fix mysql flag bug.
[35c283d] dbutil: Update dump cmd.
[be5a17f] tstore: Freeze trillian trees.

Code Contributors (alphabetical order)

• Amir Massarwa (amass01)
• Dave Collins (davecgh)
• Luke Powell (lukebp)

politeia v1.3.1

This patch release includes the following fixes and improvements. See the
changelog for the full list of changes.

• Fixes a MySQL maximum placeholder bug. MySQL allows a maximum of 65,535
  placeholders to be used in a single prepared statement. This limit was being
  hit on a recent proposal vote that had a high voter turnout, causing certain
  clients to be unable to cast votes during the last day of voting.

• Makes various small improvements to politeiavoter. A signal interrupt bug
  and a bug that was causing unnecessary retry attempts were both fixed. The
  --hoursprior UX was also cleaned up.

• Fixes a performance bottleneck on the proposal list views. v1.3.0 added a
  API request to the proposal list views that nearly doubled the list view load
  time. This release reduces the response time of this call by ~86%. The
  proposal list view performance now matches the pre v1.3.0 performance.

Changelog

This release consists of 14 commits from 2 contributors which totals to 44
files changed, 2,595 additional lines of code and 587 deleted lines of code.

[45c7f39] docs: Add v1.3.0 release notes. [lukebp]
[ee56e9c] pictl: Add proposal status opt to seedproposals. [lukebp]
[fdc6e23] sessions: Implement mysql db. [amass01]
[c0d95bd] usermd: Add cache fsck. [amass01]
[df87720] pictl: Add timer flag. [lukebp]
[5921145] politeiavoter: Fix retry loop bug. [lukebp]
[43fe7b3] politeiavoter: Improve summary printing. [lukebp]
[e9cc239] politeiavoter: Fix duplicate vote bug. [lukebp]
[7f42cfe] pi: Cache proposal statuses. [amass01]
[760e539] politeiad: Execute batch reads concurrently. [lukebp]
[ba9da84] pi: Update proposal status cache logging. [lukebp]
[ede2886] politeiavoter: Improve hoursprior UX. [lukebp]
[b7f8a71] mysql: Batch select queries when required. [lukebp]
[e5c7b90] Bump version to v1.3.1. [lukebp]

Code Contributors (alphabetical order)

• Amir Massarwa (amass01)
• Luke Powell (lukebp)

politeia v1.3.0

This release includes the following additions and improvements. See the
changelog for the full list of changes.

• Preps the codebase for the user layer rewrite. The majority of the LOC
  changes made in this release are changes made in preparation for the user
  layer rewrite. This includes pulling apart code and separating concerns that
  were too tightly coupled with the legacy code and moving all of the legacy
  code into a legacy package that will be removed at some point in the
  future.

• Improves the billing status API. Billing statuses were updated to allow
  for error correction. By default, only a single billing status change is
  allowed. This is now configurable by the sysadmin to allow for error
  correction in the event that an admin makes a mistake when updating a
  proposal's billing status.

• Parallelize politeiavoter trickling. The user now has the option to
  parallelize their vote trickling using up to 100 concurrent threads. The
  threads will start trickling votes in at random, uncorrelated times, further
  improving privacy and making the trickling process more robust.

Note: You must update your politeiavoter binary if you use the politeiavoter
trickling feature. Retry attempts for failed votes will not work properly if
you do not update.

Changelog

This release consists of 36 commits from 4 contributors which totals to 211
files changed, 11,364 additional lines of code and 8,724 deleted lines of code.

docs: Add v1.2.0 release notes. [thi4go] (#1516)
tstorebe: Add record inventory fsck. [thi4go] (#1520)
pi: Add API to retreive billing status changes. [amass01] (#1526)
politeiawww: Add legacy package. [lukebp] (#1523)
politeiawww: Add logger package. [lukebp] (#1527)
pi: Allow for billing status error correction. [amass01] (#1528)
pi: Allow zero billingstatuschangesmax. [amass01] (#1533)
pi: Make billing status changes batched. [amass01] (#1535)
politeiawww: Add websockets package. [lukebp] (#1529)
multi: Move identity methods. [lukebp] (#1530)
ticketvote: Fix ineligible status bug. [lukebp] (#1537)
ticketvote: Add cache fsck. [thi4go] (#1531)
politeiawww: Refactor config for rewrite. [lukebp] (#1536)
politeiavoter: Add hours prior config setting. [marcopeereboom] (#1542)
pi: Make status transition error human readable. [amass01] (#1543)
comments: Add cache fsck. [thi4go] (#1544)
pi: Allow comment writes on unvetted proposals. [amass01] (#1546)
logger: Fix nil pointer bug. [lukebp] (#1540)
politeiaverify: Allow short token filenames. [lukebp] (#1539)
politeiawww: Move sessions into legacy package. [lukebp] (#1554)
politeiawww: Add generic sessions store. [lukebp] (#1555)
politeiavoter: Add voting sortition in bunches. [marcopeereboom] (#1556)
politeiavoter: Print proposal names in inventory. [amass01] (#1548)
pictl: Add rfptest command. [amass01] (#1551)
doc: Add note about unverifiable partial records. [amass01] (#1559)
pictl: Fix typo and improve logging in rfptest. [amass01] (#1560)
pictl: Fix print bug and standardize vote params. [lukebp] (#1562)
pictl: Use cmdVoteStart in test commands. [amass01] (#1563)
politeiavoter: Allow up to 100 bunches. [marcopeereboom] (#1565)
Bump version to 1.3.0. [lukebp] (#1572)
pi: Fix RFP billing status bug. [lukebp] (#1575)
pi: Add approved proposal status. [lukebp] (#1578)
ticketvote: Fix status change validation bug. [lukebp] (#1572)
politeiavoter: Improve help output. [amass01] (#1579)
politeiavoter: Fix config bugs. [lukebp] (#1568)
ticketvote: Cast vote error format. [amass01] (#1561)

Code Contributors (alphabetical order)

• Amir Massarwa (amass01)
• Luke Powell (lukebp)
• Marco Peereboom (marcopeereboom)
• Thiago F. Figueiredo (thi4go)

politeia v1.2.0

This release includes the following feature additions, bug fixes and
improvements.

• Proposal author updates (#1491).
  Proposal authors are now able to give updates on the status of their
  proposal. Once the proposal is approved, the author can can start new comment
  threads that server as update threads until the proposal has been marked as
  closed or completed by an admin.

• Proposal billing statuses (#1480).
  Billing statuses (active, closed, completed) have been added to proposals.
  Once a proposal has been approved, the billing status is considered to be
  active. Admins can mark the billing status of a proposal as completed or
  closed in order to prevent further billing against the proposal. The
  completed billing status indicates the proposal has successfully delivered
  all promised work and is no longer being billed against. The closed billing
  status means that the proposal work has not been fully delivered, but further
  billing against the proposal is no longer allowed. These billing statuses
  will allow CMS, Decred's contractor management system, to further automate
  the Decred treasury billing process.

• Proposal statuses (#1515).
  A proposal status type has been added to the pi plugin that is used to map
  all possible paths a proposal can take throughout its lifecycle. This status
  is determined at runtime by combining the metadata from the various plugins
  (record status changes, vote status changes, billing status changes, etc).
  This status removes complexity from the client and serves as the source of
  truth for how the various plugin metadata impacts a proposal.

• OOM hardening.
  The Politeia servers have experienced a few "out of memory" errors since the
  last release. This was due to the combination of a memory leak in politeiad
  and a server config that was too restrictive with regards to allowed memory
  usage of the politeiad process. These issues have been fixed and additional
  measures have been implemented in order to increase the security and
  stability of the Politeia servers. (#1500) (#1505) (#1506) (#1507)

Note: In order to enable proposal author updates, the following plugin
setting must be added to the politeiad config file.

pluginsetting=comments,allowextradata,true

Changelog

This release consists of 26 commits from 7 contributors which totals to 81
files changed, 4,350 additional lines of code and 1,237 deleted lines of code.

multi: Add billing statuses. (#1480)
politeiawww: Add missing code blocks to README. (#1489)
docs: Add v1.1.0 release notes. (#1486)
politeiavoter: Update prompt pass placement. (#1490)
comments: Update signature and settings. (#1487)
Bump to Go 1.17. (#1492)
pictl: Improve human readable dollar printing. (#1488)
multi: Add proposal author updates. (#1491)
records: Add missing error return. (#1496)
multi: Document exported names. (#1497)
Add SECURITY.md file. (#1495)
pictl: cmdVoteStart help message typo. (#1498)
store: Close deblob reader. (#1500)
mysql: fix u_blub user table column bug. (#1499)
records: Fix SetStatusReply wrong username bug. (#1509)
politeiawww: Add server read limits. (#1506)
politeiad: Cleanup middleware. (#1507)
pictl: Add save as csv option to vote results. (#1478)
multi: Add read/write constraints to servers. (#1505)
politeiad: Setup fscks. (#1512)
politeiad: Remove politeiaimport tool. (#1513)
multi: Remove testpoliteiad package. (#1514)
docs: Add plugin overview. (#1519)
pi: Add proposal statuses. (#1515)
politeiawww: Add page sizes to policy replies. (#1518)
pi: Fix typo in Summary.Status field json key. (#1522)

Code Contributors (alphabetical order)

• Amir Massarwa (amass01)
• Đàm Việt Anh (vibros68)
• David Hill (dajohi)
• Degeri (degeri)
• Luke Powell (lukebp)
• Thiago F. Figueiredo (thi4go)
• Victor Guedes (victorgcramos)

politeia v1.1.0

This feature release includes the following feature additions.

• Additional proposal metadata fields. A proposal author is now required
  to provide a funding amount, start date, estimated end date, and domain
  when they submit their proposal.

• politeiawww email rate limit. politeiawww users are now subject to a
  daily email rate limit to prevent malicious users from being able to
  repeatedly perform actions that result in notification emails in an attempt
  to get the email server classified as spam.

• MySQL user database implementation. politeiad must be run using MySQL
  due to trillian (tlog) running on MySQL. The user database was previously
  implemented using CockroachDB. The addition of the MySQL user database
  implemenation allows for the removal of CockroachDB as a dependency, allowing
  Politeia to be run against a single database instance.

Changelog

This feature release consists of 31 commits from 7 contributor which total to
105 files changed, 7,520 additional lines of code, and 1,234 deleted lines of
code.

multi: Update READMEs. (decred/politeia#1421)
Add docs/release-notes directory. (decred/politeia#1420)
www: Add PaywallConfirmations to policy. (decred/politeia#1423)
multi: Add pkg/errors stack traces. (decred/politeia#1428)
politeiavoter: Migrate to new APIs. (decred/politeia#1415)
politeiawww: Add user tests. (decred/politeia#1235)
user: Add mysql userdb implementation. (decred/politeia#1419)
politeiavoter: Retry votes on 500s. (decred/politeia#1438)
politeiad: Fix tstore config validation. (decred/politeia#1427)
d/backendv2: Fix trillian rfc6962 import. (decred/politeia#1433)
politeiavoter: Improve error logging. (decred/politeia#1439)
pictl: Add missing cmd docs. (decred/politeia#1435)
www/pi: Use proposal name in email subjects. (decred/politeia#1440)
build: Update golangci-lint to v1.41.1. (decred/politeia#1447)
politeiawww: Update README. (decred/politeia#1441)
usermd: Fix user tokens sorting bug. (decred/politeia#1449)
politeiad: Improve plugin setting parsing. (decred/politeia#1451)
mysql: Upsert user's identities on user update. (decred/politeia#1458)
pi: Add proposal validation tests. (decred/politeia#1453)
mysql: Consistent table and columns naming convention. (decred/politeia#1459)
comments: Sort votes by timestamp. (decred/politeia#1460)
comments: Return record state on comment votes. (decred/politeia#1461)
politeiad: Add reset docs. (decred/politeia#1462)
multi: Fix ineligible vote status bugs. (decred/politeia#1463)
politeiawww: Add email rate limit. (decred/politeia#1448)
politeiawww: Update password change email subject. (decred/politeia#1475)
multi: Fix set status json typo. (decred/politeia#1476)
Update dcrdata import. (decred/politeia#1472)
multi: Handle duplicate payload errors gracefully. (decred/politeia#1467)
pi: Add additional proposal metadata. (decred/politeia#1444)
politeiawww: Add login to auth router. (decred/politeia#1481)

Code Contributors (alphabetical order)

• Amir Massarwa (amass01)
• Đàm Việt Anh (vibros68)
• David Hill (dajohi)
• Luke Powell (lukebp)
• Thiago F. Figueiredo (thi4go)
• Tiago Alves Dulce (tiagoalvesdulce)
• Victor Guedes (victorgcramos)

politeia v1.0.1

politeia v1.0.1

This patch release fixes various bugs in both Pi and CMS.

Changelog

This patch release consists of 15 commits from 5 contributor which total to 46 files changed, 470 additional lines of code, and 514 deleted lines of code.

Pi (Proposal System)

www: Add totpverified to login reply. (decred/politeia#1405)
politeiawww: Fix missing comment count. (decred/politeia#1408)
politeiad: Move timestamp verify functions. (decred/politeia#1404)
tstore: Optimize map allocation. (decred/politeia#1410)
multi: Cleanup dead code. (decred/politeia#1409)
pi: Add record token to email subjects. (decred/politeia#1417)
Update modules. (decred/politeia#1411)
politeiawww: Handle client conn drops. (decred/politeia#1416)

CMS (Contractor Managment System)

[cms] Sort invoices for proposal billing (decred/politeia#1350)
[cms] Filter invoices by line item domain (decred/politeia#1351)
[cms] Update doco and remove documentation domain (decred/politeia#1352)
[cms] Add more invoice email notifications (decred/politeia#1353)
cms: Add check for outstanding DCCs (decred/politeia#1338)
[cms] Fix code stats user update (decred/politeia#1347)
[cms] Update line items from token for more invoice status (decred/politeia#1418)

Code Contributors (alphabetical order)

• Alex Yocom-Piatt
• Amir Massarwa
• iurii
• Luke Powell
• Tiago Alves Dulce

politeia v1.0.0

politeia v1.0.0

This release of politeia introduces a large number of structural changes to politeia and deprecates many of the existing politeiawww API routes. Listed below are a summary of the changes.

• Adds a new politeiad backend called tstore.
• Adds a politeiad v2 API.
• Formalizes a politeiad plugin architecture.
• Adds the following politeiad plugins: comments, ticketvote, dcrdata, usermd, pi.
• Support has been removed for the politeiawww www/v1 API routes that handle proposals, comments, and dcr ticket votes. Routes that are used by decred clients (decrediton, dcrdata) have only been deprecated and will have support removed in the future.
• Adds the following politeiawww APIs: records/v1, comments/v1, ticketvote/v1.

politeiad tstore backend

A new backend was added to politeiad that is referred to as the tstore backend. It combines a trillian log, commonly referred to as a tlog, and a key-value store. The tstore backend differs from the git backend in three main ways.

1. Scalability
   The git backend relied on git repos saved to the file system to store data. This approach cannot be scaled out since it is constrained to the file system of a single politeiad instances. The tstore backend moves the data into an actual database, allowing for much greater scalability.

2. Separate the timestamps and the data blobs
   The git backend would timestamp the git commit hashes onto the decred blockchain. These commit hashes are merkle roots of all the files in the commit. The timestamp and the data blobs in this setup are interconnected. If you delete a data blob from the commit history, such as if you need to censor a public comment, you destroy the ability to recreate the merkle path for other data blobs included in that commit merkle root, thus destroying the ability to prove inclusion of data in a decred timestamp. Your only option is to remove the data from the git repo in a new commit, but the censored data remains in the git history. The tstore backend seperates out the timestamps from the data blobs. A merkle tree of the data is stored in a tlog instance while the data itself is stored in a key-value database. This allows for the deletion of data from the database, such as if you need to censor a public comment, without impacting any of the timestamps or our ability to prove the inclusion of a piece of data in a timestamp.

3. Ability to retrieve timestamp inclusion proofs
   The tstore backend gives us the ability to retrieve an inclusion proof for any piece of data in the backend, providing cryptographic proof that the data blob was included in a dcr timestamp. There was no easy way to do this in the git backend.

politeiad v2 API

A politeiad v2 API was added. This API is a simplified version of the politeiad v1 API. The tstore backend relies solely on the politeiad v2 API.

politeiad plugin architecture

A formalized plugin architecture was added to politeiad.

A politeiad plugin extends a record with additional functionality. Some examples of this include adding comments to a record, adding the ability to run a dcr ticket vote on a record, and adding record validation that is specific to the decred proposal system.

This plugin architecture allows politeiad to become a configurable system. If a user wants to runs a politeiad instance that allows for specific file types and record comments, they can do so by simply adjusting the politeiad configuration. No new code is required unless they want to add in more granular, application specific validation.

The end goal is to eventually allow plugin code to exist outside of the politeia repository that can be dynamically loaded at runtime. While this is not possible yet, this release lays the foundation for doing so in the future.

politeiad plugins

The following plugins have been added to politeiad:

• comments - extends a record with comment functionality.
• ticketvote - extends a record with dcr ticket vote functionality.
• dcrdata - provides an API that plugins can use to retreive data from dcrdata.
• usermd - extends a record with user metadata (ex. user signatures).
• pi - extends a record with functionality specific to the decred proposal system.

politeiawww APIs

The following politeiawww APIs have been added:

• records/v1
• comments/v1
• ticketvote/v1

The politeiawww APIs have been rewritten to be generic, application agnostic APIs. These APIs were formally tied to specific politeia applications such as the decred proposal system and the decred contractor management system. You would use a proposal route to submit new proposals and a separate invoice route to submit cms invoices. These application specific routes have been replaced with more generic APIs. For example, there now exists a single set of routes for submitting all records, regardless of whether that record is a proposal or an invoice. The application specific validation has been pushed into the politeiad plugins. The validation that is specific to proposals is now performed in the pi politeiad plugin.

The politeiawww APIs now map to the politeiad plugins. For example, the politeiad comments plugin has a corresponding politeiawww API that can be used for submitting comments on any record, regardless of whether its a proposal comment or invoice comment.

These application agnostic APIs allow politeia to become a configurable system. You configure the politeiad plugins for your specific use case and you automatically get the corresponding politeiawww API routes.

Changelog

This release consists of 21 commits with a total of 368 files changed, 48,410 additional lines of code, and 39,948 deleted lines of code.

Add tstore backend and plugin architecture. (decred/politeia#1180)
multi: Break comment timestamps up. (decred/politeia#1374)
multi: Return comment del signature. (decred/politeia#1375)
politeiaverify: Verify comment and vote bundles. (decred/politeia#1377)
Add last error code verification. (decred/politeia#1376)
politeiavoter: Add temp dir to tests. (decred/politeia#1378)
ticketvote: Del ballots channel. (decred/politeia#1379)
tstore: Delete vetted record content bug. (decred/politeia#1384)
ticketvote: Remove testnet linkby path. (decred/politeia#1385)
politeiawww: Fix login reply bug. (decred/politeia#1386)
politeiawww: Fix legacy vote results bug. (decred/politeia#1387)
ticketvote: Fix active votes cache bug. (decred/politeia#1388)
multi: Add receipt to vote details. (decred/politeia#1380)
politeiawww: Fix legacy active votes bug. (decred/politeia#1389)
pictl: Fix various minor issues. (decred/politeia#1390)
build: Test against Go 1.16. (decred/politeia#1366)
multi: Add timestamp to ticket vote. (decred/politeia#1395)
politeiavoter: Fix password bug. (decred/politeia#1396)
politeiawww: Fix legacy pagination bug. (decred/politeia#1397)
ticketvote: Fix json typo. (decred/politeia#1399)
ticketvote: Fix summary best block bug. (decred/politeia#1402)

politeia v0.2.0

This tag marks the last commit that supports the politeiad git backend, the politeiad v1 API, and much of the politeiawww www/v1 and www/v2 APIs.