[CO-432] Add foreign transaction to input and remove updateForeign

[paweljakubas]

Description

At this moment prefiltering comes down to filtering out the relevant inputs and so the regular call to updatePending fails to see that the foreign transaction that got removed. Up till now we were looking at the outputs instead of the inputs not because of any fundamental difference between regular and foreign pending transactions, but rather because prefiltering is removing some information that we need and so we look at the outputs as a (somewhat poor) substitute instead. As a consequence we have updatePending and updateForeign removing corresponding inputs and outputs, respectively.

Here, the following changes are proposed:

1. We adopt lookup seeking one of our currently tracked foreign transaction to appear in the block's inputs, and if so, make sure it appears in the prefiltered block
   (a) getters for foreign pending block are added
   (b) in prefilterTxForWallets we search inputs for transactions that have the same id as the foreign transation in the respective pending set
   (c) PrefilteredBlock is extended by pfbForeignInputs and populated if search of point 1b is successful
   (d) refactoring of updatePending in such a way that it works also with pfbForeignInputs and service both input and foreign input checkpoint
   (e) update of both (wallet-new/src/Cardano/Wallet/Kernel/DB/Spec/Update.hs) - applyBlock and applyBlockPartial is done in the context of the change of point 1d
2. updateForeign is removed
3. updateUtxo is changed in such a way that it includes rather (Set.union pfbInputs pfbForeignInputs) than pfbInputs

Linked issue

https://iohk.myjetbrains.com/youtrack/issue/CO-432

Type of change

• 🐞 Bug fix (non-breaking change which fixes an issue)
• 🛠 New feature (non-breaking change which adds functionality)
• ⚠️ Breaking change (fix or feature that would cause existing functionality to change)
• 🏭 Refactoring that does not change existing functionality but does improve things like code readability, structure etc
• 🔨 New or improved tests for existing code
• ⛑ git-flow chore (backport, hotfix, etc)

Developer checklist

• I have read the style guide document, and my code follows the code style of this project.
• If my code deals with exceptions, it follows the guidelines.
• I have updated any documentation accordingly, if needed. Documentation changes can be reflected in opening a PR on cardanodocs.com, amending the inline Haddock comments, any relevant README file or one of the document listed in the docs directory.
• CHANGELOG entry has been added and is linked to the correct PR on GitHub.

Testing checklist

• I have added tests to cover my changes.
• All new and existing tests passed.

QA Steps

$ ./scripts/build/cardano-sl.sh wallet-new
$ nix-build -A walletIntegrationTests -o launch_integration_tests --arg useStackBinaries true
$ ./launch_integration_tests

[paweljakubas]

As CI stucked on a number of jobs, I squashed the commits, rebased and force pushed

[paweljakubas]

concerning integration tests: PR affects how foreign transaction is dealt and we have:

  redeeming avvm key gives rise to the corresponding increase of balance of wallet'account - mnemonic not used
    +++ OK, passed 1 tests.

The failed test:

Failures:

  integration/TransactionSpecs.hs:102:22: 
  1) Transactions asset-locked wallets can receive funds and transactions are confirmed in index
       Falsifiable (after 1 test):
       
       not expected: 0

was present irrespective of my changes (I also checked this on my computer and corroborate this statement)

[paweljakubas]

At this moment it is double counting in an account balance - I tried also with

utxoMin   = utxoUnion `Core.utxoRestrictToInputs` (Set.union pfbInputs pfbForeignInputs)
utxo'     = utxoUnion `Core.utxoRemoveInputs`     (Set.union pfbInputs pfbForeignInputs)

in updateUtxo PrefilteredBlock{..} (utxo, balance) with the same result

[paweljakubas]

Added Ryan's suggestions, rebased and squashed into one commit

[paweljakubas]

rebased and adopted a number of simplifications : (a) get rid of foldr in favour of maps and (b) moving reindexedforeignInput upstream

[paweljakubas]

PR moved to cardano-wallet and hence closed here

[KtorZ]

Re-opening here, we still need to maintain both repository in sync during the transition (i.e. until cardano-wallet becomes a submodule of cardano-sl)