Limit the size and age of the ZIP-401 rejected transaction ID list

[conradoplg]

Motivation

ZIP-401 specifies that the list of TXIDs must have a maximum size and entries older than a specified threshold must be removed.

Specifications

https://zips.z.cash/zip-0401#specification

Designs

Some of the design was discussed in #2759

Solution

• Create a EvictedList that encapsulates both behaviours
• Use the EvictedList for both rejected and expired transactions
  • I'm using in both because that was easier with the current Storage design, and seems harmless (expired transactions will also have a timeout)

I tried creating a generic OrderedMap data structure, that EvictedList would use, in order to organize the code. However I noticed it would be more work since it's tricky to handle/document the behavior that removing a item from the list might not really remove a item from the list (when a value was refreshed and an stale entry is left behind). So I left the entire logic inside EvictedList (where the removal methods are not exposed).

Closes #2759
Closes #2958

Review

Anyone can review. @teor2345 discussed the design.

This may require adjustments after #2889 so it may be better to wait for that to be merged first.

Reviewer Checklist

• Does it make sense to use EvictList for the expired transaction, or should change Storage so that only rejected list uses it?
• Code implements Specs and Designs
• Tests for Expected Behaviour
• Tests for Errors

Follow Up Work

[teor2345]

This looks good, but I think contains_key can return transaction IDs that have expired since the last insert.

I also have a few minor suggestions, feel free to ignore them.

[teor2345]

Nice catch that we might need to remove multiple entries in insert()!

I think we might need to prune_old() in len() to meet our DoS performance goals.
(I didn't realise this until you fixed len().)

I also think we need to limit the length of ordered_entries, even if its entries haven't expired.

[teor2345]

We could merge this as-is, but we're waiting on #2889.

I have a few optional suggestions for comments and diagnostics.
(The assertions should never fail, so we might never see these messages.)

[teor2345]

There aren't any blockers for merging this PR.

It just needs to be rebased or merged with main after PR #2889 merges.

[conradoplg]

There aren't any blockers for merging this PR.

It just needs to be rebased or merged with main after PR #2889 merges.

Done

[teor2345]

Looks great, let's get it merged!