Andreabedini/plutus streaming #472

raduom · 2022-05-23T03:23:34Z

Merge plutus-streaming at the point where we can synchronise with the main chain.

Pre-submit checklist:

Branch
- Tests are provided (if possible)
- Commit sequence broadly makes sense
- Key commits have useful messages
- Relevant tickets are mentioned in commit messages
- Formatting, materialized Nix files, PNG optimization, etc. are updated
PR
- Self-reviewed the diff
- Useful pull request description
- Reviewer requested

It can be implemented separately, as done in Plutus.Streaming.LedgerState.

The Stream of blocks never ends, move out-of-band communication to be exception based.

* You need to dup a broadcast channel in order to read. * Update dependencies.

andreabedini

There are some changes I had made in my branch which you don't have here. Nothing too fundamental luckily.

I had written some functions for you, which are missing in your branch.

transactions ::
  Cardano.Api.BlockInMode Cardano.Api.CardanoMode ->
  [Ledger.CardanoTx]
transactions (Cardano.Api.BlockInMode (Cardano.Api.Block _ txs) eim) =
  map (\tx -> Ledger.CardanoApiTx (workaround (Ledger.SomeTx tx) eim)) txs

-- Not needed after https://github.com/input-output-hk/cardano-node/pull/3665
workaround ::
  (Cardano.Api.IsCardanoEra era => Cardano.Api.EraInMode era Cardano.Api.CardanoMode -> a) ->
  Cardano.Api.EraInMode era Cardano.Api.CardanoMode ->
  a
workaround k Cardano.Api.ByronEraInCardanoMode   = k Cardano.Api.ByronEraInCardanoMode
workaround k Cardano.Api.ShelleyEraInCardanoMode = k Cardano.Api.ShelleyEraInCardanoMode
workaround k Cardano.Api.AllegraEraInCardanoMode = k Cardano.Api.AllegraEraInCardanoMode
workaround k Cardano.Api.MaryEraInCardanoMode    = k Cardano.Api.MaryEraInCardanoMode
workaround k Cardano.Api.AlonzoEraInCardanoMode  = k Cardano.Api.AlonzoEraInCardanoMode

txInsAndOuts ::
  Ledger.CardanoTx ->
  (Set Ledger.TxIn, [(Ledger.TxOut, Ledger.TxOutRef)])
txInsAndOuts tx =
  (Ledger.getCardanoTxInputs tx, Ledger.getCardanoTxOutRefs tx)

txInsAndOuts' ::
  Ledger.CardanoTx ->
  (Ledger.TxId, [Ledger.TxOutRef], [Ledger.TxOutRef])
txInsAndOuts' tx = (txId, inRefs, outRefs)
  where
    txId = Ledger.getCardanoTxId tx
    inRefs = map Ledger.txInRef $ Set.toList $ Ledger.getCardanoTxInputs tx
    outRefs = map snd $ Ledger.getCardanoTxOutRefs tx

datums ::
  Ledger.CardanoTx ->
  [(Ledger.DatumHash, Ledger.Datum)]
datums tx = do
  let txIns = Set.toList $ Ledger.getCardanoTxInputs tx
  (Ledger.TxIn _ (Just (Ledger.ConsumeScriptAddress _validator _redeemer datum))) <- txIns
  pure (Ledger.datumHash datum, datum)

There is some tidying up to do and more importantly to figure out where some code belongs.

andreabedini · 2022-05-23T03:31:00Z

nix/pkgs/haskell/sha256map.nix

@@ -16,4 +16,5 @@
  "https://github.com/input-output-hk/servant-purescript"."44e7cacf109f84984cd99cd3faf185d161826963" = "10pb0yfp80jhb9ryn65a4rha2lxzsn2vlhcc6xphrrkf4x5lhzqc";
  "https://github.com/input-output-hk/Win32-network"."3825d3abf75f83f406c1f7161883c438dac7277d" = "19wahfv726fa3mqajpqdqhnl9ica3xmf68i254q45iyjcpj1psqx";
  "https://github.com/Quid2/flat"."ee59880f47ab835dbd73bea0847dab7869fc20d8" = "1lrzknw765pz2j97nvv9ip3l1mcpf2zr4n56hwlz0rk7wq7ls4cm";
+  "https://github.com/raduom/hysterical-screams"."e30c0f8e6e86e6f57249aa4437ac8cadc0d49420" = "1ayqxfk7d99mcir7syji2404vzz4s6yg56rmw9aj56djvd6804v2";


I think this code should be included in the main repo.

andreabedini · 2022-05-23T03:33:22Z

plutus-ledger/src/Cardano/Api/Improved.hs

+improveBlockInMode (BlockInMode bl eim@ShelleyEraInCardanoMode) = ImprovedBlockInMode bl eim
+improveBlockInMode (BlockInMode bl eim@AllegraEraInCardanoMode) = ImprovedBlockInMode bl eim
+improveBlockInMode (BlockInMode bl eim@MaryEraInCardanoMode)    = ImprovedBlockInMode bl eim
+improveBlockInMode (BlockInMode bl eim@AlonzoEraInCardanoMode)  = ImprovedBlockInMode bl eim


This should not be needed anymore (after IntersectMBO/cardano-node#3665)

andreabedini · 2022-05-23T03:38:19Z

plutus-ledger/src/Ledger/Tx/CardanoAPI.hs

+fromCardanoBlockInMode (improveBlockInMode -> ImprovedBlockInMode (C.Block _ txs) eim) = map (flip SomeTx eim) txs
+
+_txs :: Fold (C.BlockInMode C.CardanoMode) SomeCardanoApiTx
+_txs = folding fromCardanoBlockInMode


This is not used anywhere.

andreabedini · 2022-05-23T03:40:05Z

plutus-streaming/src/Cardano/Api/Extras.hs

+
+-- FIXME orphan instance
+-- https://github.com/input-output-hk/cardano-node/pull/3608
+instance IsString (Hash BlockHeader) where


This instance is in Cardano.Api now IntersectMBO/cardano-node#3619

andreabedini · 2022-05-23T03:41:19Z

plutus-streaming/src/Index/TxIdStatus.hs

+{-# LANGUAGE NamedFieldPuns    #-}
+{-# LANGUAGE OverloadedStrings #-}
+
+module Index.TxIdStatus where


You put this as part of plutus-streaming, should it be in a separate package? called marconi? @koslambrou what should we do here?

andreabedini · 2022-05-23T03:42:57Z

plutus-streaming/src/Index/TxIdStatus.hs

+-- blocks have settled
+getTxs :: BlockInMode CardanoMode -> (BlockNo, [TxId])
+getTxs (BlockInMode (Block header transactions) era) =
+  case era of


here you are reimplementing fromCardanoBlockInMode above

andreabedini · 2022-05-23T03:45:44Z

plutus-streaming/README.md

@@ -0,0 +1,29 @@
+# Plutus Streaming PoC


You need to cherrypick c9a67bb to fix the readme.

andreabedini · 2022-05-23T03:46:52Z

plutus-streaming/src/Index/TxIdStatus.hs

+  -- execute handle "UPDATE tx_state SET confirmations = confirmations + ?" (Only $ Map.size bufferedTxs)
+  -- execute_ handle "COMMIT"
+
+getBlocks :: SimpleChainSyncEvent -> BlockInMode CardanoMode


This is not used anywhere

andreabedini · 2022-05-23T03:51:00Z

plutus-streaming/src/Index/TxIdStatus.hs

+    go :: (Int, Map TxId TxConfirmedState)
+       -> (BlockNo, [TxId])
+       -> (Int, Map TxId TxConfirmedState)
+    go (confirmations, acc) (_, []) = (confirmations, acc)


Do we need to be careful about strictness here? Perhaps better to have a strict datatype

data GoArgs = GoArgs !Int !(Map TxId TxConfirmedState)`

so that go becomes go :: GoArgs -> (BlockNo, [TxId]) -> GoArgs and pattern match with go !args ... ?

This might not be a problem but I'd like to be sure we won't have a space leak here.

andreabedini · 2022-05-23T03:52:34Z

plutus-streaming/src/Plutus/Streaming/ChainIndex.hs

@@ -0,0 +1,52 @@
+module Plutus.Streaming.ChainIndex (


This module and the following (Plutus.Streaming.LedgerState) where basically proof-of-concept indexers. I don't think they belong to the streaming package.

raduom · 2022-05-23T04:31:20Z

On second thought, we don't need the TxIdState merged since it is only an experiment to validate my assumptions about the memory usage of Indexers.

andreabedini and others added 28 commits May 23, 2022 10:19

Plutus streaming PoC

0574796

Fix composition of two stream consumers

b8af860

ImprovedBlockInMode

9815cc4

PoC streaming wth ledger events

bcd9fea

WIP

244c68d

Renames, comments, exports

5e2bb95

WIP

0129be6

DIY LedgerState streaming PoC

9ac69dd

WIP

512c1ac

Enable -Wmissing-import-lists

e8a40d2

Remove some unused packages

48cfe7d

cli: require the use to specify the network id

d9eb57e

Fix more explicit imports

ab08787

connectToLocalNode can throw

1579c3e

Remove test executables

303b3e5

Remove the "WithLedgerState" stream.

48a8f9c

It can be implemented separately, as done in Plutus.Streaming.LedgerState.

Rework documentation

af1f055

Streams of no return

501bc73

The Stream of blocks never ends, move out-of-band communication to be exception based.

Replace Chan with TChan

414e079

You need to dup a broadcast channel in order to read. (IntersectMBO#422)

e78cad1

Andreabedini/plutus streaming (IntersectMBO#423)

6721053

* You need to dup a broadcast channel in order to read. * Update dependencies.

More tidy up, switch cli to JSON (as much as possible)

7273926

Updated sha map

1d7dc28

Test an inefficient TxStatus index

f6cbad9

Newer version of hysterical-streams.

c75ed47

Correct version of hysterical-screams

ead37da

Fix code to work with seq based hysterical-screams.

3e8c921

Newer version of indexes.

e4b8b9d

raduom self-assigned this May 23, 2022

raduom requested a review from koslambrou May 23, 2022 03:23

raduom requested a review from andreabedini May 23, 2022 03:23

Use a vector based sqlite index.

73e7339

raduom force-pushed the andreabedini/plutus-streaming branch from dc26cd4 to 73e7339 Compare May 23, 2022 03:31

raduom marked this pull request as ready for review May 23, 2022 03:32

andreabedini suggested changes May 23, 2022

View reviewed changes

raduom closed this May 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Andreabedini/plutus streaming #472

Andreabedini/plutus streaming #472

raduom commented May 23, 2022 •

edited

Loading

andreabedini left a comment

andreabedini May 23, 2022

andreabedini May 23, 2022

andreabedini May 23, 2022

andreabedini May 23, 2022

andreabedini May 23, 2022

andreabedini May 23, 2022

andreabedini May 23, 2022

andreabedini May 23, 2022

andreabedini May 23, 2022

andreabedini May 23, 2022

raduom commented May 23, 2022

Andreabedini/plutus streaming #472

Andreabedini/plutus streaming #472

Conversation

raduom commented May 23, 2022 • edited Loading

andreabedini left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raduom commented May 23, 2022

raduom commented May 23, 2022 •

edited

Loading