politeiad performance limitations

[lukebp]

The response times for API calls with larger payloads can be extremely slow at times. I was curious what the bottleneck is for these calls so I did a little digging.

The vote results API endpoint returns all of the cast vote data for a Decred proposal vote. The response time for a mainnet proposal with 20k-30k votes can be between 5-10 seconds.

Listed below is the response time of the vote results endpoint on test-proposals for a proposal with ~3k votes. The response payload size was ~1.4 MB.

Response time logged by client: 2,563ms
 
The majority of this response time was from the vote results plugin command.
Total command execution time was 1.511 seconds. A breakdown of the command
execution is listed below.

cmdResults
  voteResults : 1,472ms
  marshal json:    37ms
              : 1,509ms

voteResults
  BlobsByDataDesc: 1,390ms
  decode         :    78ms
  sort           :     4ms 
                 : 1,472ms

BlobsByDataDesc
  Get leaves  :   197ms (get the trillian tree leaves)
  Find matches:    11ms (find the leaves for the data we're fetching)
  Decode keys :     9ms (decode the key-value store keys)
  Get blobs   :   399ms (get the data blobs from key-value store)
  Deblob      :   773ms (decode the gzipped byte slice)
              : 1,390ms

Conclusion

The largest contributors to the slow response times are the extra database read required to fetch the trillian tree and the extra decoding that we have to do as a result of the generic data structure that is saved to the key-value store. Previous testing revealed that the response time for fetching the trillian leaves will increase ~100ms for every 10,000 additional leaves.

The politeiad backend is typically not going to be as performant as a standard web server backend because of the tradeoffs that we made in order to be able to return timestamp inclusion proofs for any individual piece of data. politeiad is going to be CPU bound on these expensive calls. The response times can be reduced by increasing the CPU resources of the politeiad server or by scaling out politeiad horizontally.

[lukebp]

We did some testing on production and found out that the request timeout needs to be set to 3 minutes in order to accommodate the vote results endpoint on slow 3g networks.