Speedup diff-states by over 50x (6+ days -> under 3 hours) by removing old bottleneck

[fxamacker]

Currently, diffCadenceValues() is used in diffCadenceDictionaryValue() to compare dict keys in old and new dict values. The complexity is O(n^2) where n is number of keys.

This commit removes explicit key comparisons in diffCadenceDictionaryValue() and detects not-found keys when retrieving elements for comparisons. The complexity is O(n) where n is number of keys.

Speedup for 1 account using 10 goroutines (from diff-states logs on gcp):

• before: 1 hour 9 minutes
• after: 1 minute 18 seconds

For best results, this PR should be used with:

• Optimize diff-states cmd by comparing data in parallel #6097
• Optimize diff-states to diff storage elems in parallel for large accounts #6105

When combined with PRs 6097 and 6105, this improvement allows diff-states to compare mainnet state:

• before: manually stopped after 6+ days
• after: finished in 2 hours 57 minutes

Speed comparisons used same vm on gcp, same mainnet files, nworkers=80, and used less than 2TB RAM.

Discussions on discord:

• current thread
• old thread

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 40.54054% with 22 lines in your changes missing coverage. Please review.

Project coverage is 41.75%. Comparing base (4def8fb) to head (bfb066d).

Files	Patch %	Lines
cmd/util/ledger/migrations/cadence_value_diff.go	40.54%	20 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6107      +/-   ##
==========================================
- Coverage   41.75%   41.75%   -0.01%     
==========================================
  Files        1971     1971              
  Lines      138753   138736      -17     
==========================================
- Hits        57937    57925      -12     
+ Misses      74807    74799       -8     
- Partials     6009     6012       +3     

Flag	Coverage Δ
unittests	41.75% <40.54%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[turbolent]

👏