Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(recovery): add crash recovery implementation #3491

Draft
wants to merge 41 commits into
base: satp-dev
Choose a base branch
from

Conversation

Yogesh01000100
Copy link
Contributor

@Yogesh01000100 Yogesh01000100 commented Aug 20, 2024

This PR addresses issue #3114

Changes

  • Crash Recovery: Updated crash_recovery.proto and related ts files; added core recovery logic (created functions not yet implemented).
  • Knex Config: Improved knexfile.ts and knexfile-remote.ts; added Docker Compose for production.

@RafaelAPB
Copy link
Contributor

I will review this PR

@Yogesh01000100 Yogesh01000100 force-pushed the feature/crash-recovery-improvements branch from f9014b0 to 0de9744 Compare August 21, 2024 20:08
RafaelAPB and others added 28 commits August 23, 2024 14:14
Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(bungee): add skeleton for bungee

Signed-off-by: André Augusto <andre.augusto@tecnico.ulisboa.pt>

refactor(plugin-satp-hermes): update messages formats SATP core version 2.0.0 OpenAPI

* created new messages in openAPI definition

feat(bungee): fix bungee dependencies

Signed-off-by: André Augusto <andre.augusto@tecnico.ulisboa.pt>

feat(SATP-Hermes): gateway coordinator

feat(SATP-Hermes): gateway coordinator

feat(SATP-Hermes): gateway coordinator WIP

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

test(SATP-Hermes): enable decorators needed for gateway coordinator checks

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

docs(SATP-Hermes): update package.json

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

refactor(SATP-Hermes): re-estructure project

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

refactor(SATP-Hermes): re-estructure project

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

refactor(SATP-Hermes): re-estructure project

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

refactor(SATP-Hermes): remove kotlin sdk, add protobuffer

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(SATP-Hermes): protobuf structure

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(SATP-Hermes): protobuf structure

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

refactor(SATP-Hermes): refactor naming

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(SATP-Hermes): protobuf structure improvements

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(SATP-Hermes): update yarn.lock and clean up

Signed-off-by: André Augusto <andre.augusto@tecnico.ulisboa.pt>

refactor(test-tooling): fix types of streams: use NodeJS.ReadableStream

1. The container management library that we use in the test infrastructure
(called dockerode) is expecting streams that are defined in the global
namespace of the `@types/node` library, e.g. the standard library of NodeJS
itself.
2. Previously we were using the "streams" package to provide type information
to the streams that we were passing around to dockerode and it was working
fine, but after some changes that seem unrelated this has broken the
compilation process.
3. The mentioned changes are not yet on the main branch, but we expect
them to be there soon and so this change is laying the groundwork for that
by pre-emptively fixing the broken build's root cause which is that the
test-tooling package does not declare it's typings related dependencies
correctly: It implicitly uses the NodeJS standard library's types but
so far had not declared them on the package level.
4. This change is therefore to rectify the issue of the `@types/node`
dependency missing from the test-tooling package and also the refactoring
of some of the test ledger classes which were relying on the `streams`
builtin package instead of correctly using the NodeJS.ReadableStream global.
5. Earlier the reasoning for this was that we try to avoid pulling in
types from the global scope because we try to avoid any sort of dependency
on the global scope in general. Once we have proof though that this is
causing issues with the build, then we must give up the principle for
practical reasons (and only in the minimum viable scope, e.g. this does
not change the fact that everywhere else in the codebase we should still
do our best to avoid using the global scoped classes, types, functions,
etc..).

Thank you to @AndreAugusto11 and @RafaelAPB for pointing out this issue
through the pull request of his that is currently being worked on at the
time of this writing:
RafaelAPB#72

Related to but does not address hyperledger-cacti#2811

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>

refactor(SATP-Hermes): remove unused packages

* updated openapi version
* updated bungee plugin version to 2.0.0-alpha.2

Co-authored-by: Peter Somogyvari <peter.somogyvari@accenture.com>

Signed-off-by: André Augusto <andre.augusto@tecnico.ulisboa.pt>
Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
Co-authored-by: André Augusto <andre.augusto@tecnico.ulisboa.pt>
Co-authored-by: Peter Somogyvari <peter.somogyvari@accenture.com>
Co-authored-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

test(satp-hermes): fix hanging tests

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

refactor(SATP-Hermes): fix cspell and lint

Signed-off-by: André Augusto <andre.augusto@tecnico.ulisboa.pt>
Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(satp-hermes): BLO API specification

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(satp-hermes): add BLO layer

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

fix(satp-hermes): fix status endpoint and spec

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

fix(satp-hermes): fix lint

Signed-off-by: André Augusto <andre.augusto@tecnico.ulisboa.pt>

ci(satp-hermes): rebase and update ci

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(satp-hermes): update BLO specification

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

docs(satp-hermes): migrate BLO open api spec from json to yml

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(satp-hermes): update Go BLO SDK generation

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(satp-hermes): update BLO specification

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(satp-hermes): add health check endpoint to BLO spec

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

fix(satp-hermes): re-add open-api json files

The canonical source for the API definition are the yaml files located at packages/cactus-plugin-satp-hermes/src/main/yml/openapi.yml, due to its better readability compared to json. When yarn codegen is ran within the cactus-plugin-satp-hermes package, the json files are generated from the yaml.

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

 feat(satp-hermes): remove deprecated spec

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(SATP-Hermes): add gateway coordinator WIP

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(bungee): add skeleton for bungee

Signed-off-by: André Augusto <andre.augusto@tecnico.ulisboa.pt>

refactor(plugin-satp-hermes): update messages formats SATP core version 2.0.0 OpenAPI

* created new messages in openAPI definition

feat(bungee): fix bungee dependencies

Signed-off-by: André Augusto <andre.augusto@tecnico.ulisboa.pt>

feat(SATP-Hermes): gateway coordinator

feat(SATP-Hermes): gateway coordinator

feat(SATP-Hermes): gateway coordinator WIP

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

test(SATP-Hermes): enable decorators needed for gateway coordinator checks

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

docs(SATP-Hermes): update package.json

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

refactor(SATP-Hermes): re-estructure project

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

refactor(SATP-Hermes): re-estructure project

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

refactor(SATP-Hermes): re-estructure project

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

refactor(SATP-Hermes): remove kotlin sdk, add protobuffer

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(SATP-Hermes): protobuf structure

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(SATP-Hermes): protobuf structure

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

refactor(SATP-Hermes): refactor naming

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(SATP-Hermes): protobuf structure improvements

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(SATP-Hermes): update yarn.lock and clean up

Signed-off-by: André Augusto <andre.augusto@tecnico.ulisboa.pt>

refactor(test-tooling): fix types of streams: use NodeJS.ReadableStream

1. The container management library that we use in the test infrastructure
(called dockerode) is expecting streams that are defined in the global
namespace of the `@types/node` library, e.g. the standard library of NodeJS
itself.
2. Previously we were using the "streams" package to provide type information
to the streams that we were passing around to dockerode and it was working
fine, but after some changes that seem unrelated this has broken the
compilation process.
3. The mentioned changes are not yet on the main branch, but we expect
them to be there soon and so this change is laying the groundwork for that
by pre-emptively fixing the broken build's root cause which is that the
test-tooling package does not declare it's typings related dependencies
correctly: It implicitly uses the NodeJS standard library's types but
so far had not declared them on the package level.
4. This change is therefore to rectify the issue of the `@types/node`
dependency missing from the test-tooling package and also the refactoring
of some of the test ledger classes which were relying on the `streams`
builtin package instead of correctly using the NodeJS.ReadableStream global.
5. Earlier the reasoning for this was that we try to avoid pulling in
types from the global scope because we try to avoid any sort of dependency
on the global scope in general. Once we have proof though that this is
causing issues with the build, then we must give up the principle for
practical reasons (and only in the minimum viable scope, e.g. this does
not change the fact that everywhere else in the codebase we should still
do our best to avoid using the global scoped classes, types, functions,
etc..).

Thank you to @AndreAugusto11 and @RafaelAPB for pointing out this issue
through the pull request of his that is currently being worked on at the
time of this writing:
RafaelAPB#72

Related to but does not address hyperledger-cacti#2811

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>

refactor(SATP-Hermes): remove unused packages

* updated openapi version
* updated bungee plugin version to 2.0.0-alpha.2

Co-authored-by: Peter Somogyvari <peter.somogyvari@accenture.com>

Signed-off-by: André Augusto <andre.augusto@tecnico.ulisboa.pt>
Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>

fix(satp-hermes): re-add open-api json files

The canonical source for the API definition are the yaml files located at packages/cactus-plugin-satp-hermes/src/main/yml/openapi.yml, due to its better readability compared to json. When yarn codegen is ran within the cactus-plugin-satp-hermes package, the json files are generated from the yaml.

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

ci(satp-hermes): re-activate satp-hermes ci

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

fixup: update package json, ci, yarn lock

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(SATP-Hermes): add gateway coordinator WIP

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(bungee): add skeleton for bungee

Signed-off-by: André Augusto <andre.augusto@tecnico.ulisboa.pt>

refactor(plugin-satp-hermes): update messages formats SATP core version 2.0.0 OpenAPI

* created new messages in openAPI definition

feat(bungee): fix bungee dependencies

Signed-off-by: André Augusto <andre.augusto@tecnico.ulisboa.pt>

feat(SATP-Hermes): gateway coordinator

feat(SATP-Hermes): gateway coordinator

feat(SATP-Hermes): gateway coordinator WIP

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

test(SATP-Hermes): enable decorators needed for gateway coordinator checks

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

docs(SATP-Hermes): update package.json

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

refactor(SATP-Hermes): re-estructure project

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

refactor(SATP-Hermes): re-estructure project

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

refactor(SATP-Hermes): re-estructure project

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

refactor(SATP-Hermes): remove kotlin sdk, add protobuffer

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(SATP-Hermes): protobuf structure

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(SATP-Hermes): protobuf structure

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

refactor(SATP-Hermes): refactor naming

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(SATP-Hermes): protobuf structure improvements

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(SATP-Hermes): update yarn.lock and clean up

Signed-off-by: André Augusto <andre.augusto@tecnico.ulisboa.pt>

refactor(test-tooling): fix types of streams: use NodeJS.ReadableStream

1. The container management library that we use in the test infrastructure
(called dockerode) is expecting streams that are defined in the global
namespace of the `@types/node` library, e.g. the standard library of NodeJS
itself.
2. Previously we were using the "streams" package to provide type information
to the streams that we were passing around to dockerode and it was working
fine, but after some changes that seem unrelated this has broken the
compilation process.
3. The mentioned changes are not yet on the main branch, but we expect
them to be there soon and so this change is laying the groundwork for that
by pre-emptively fixing the broken build's root cause which is that the
test-tooling package does not declare it's typings related dependencies
correctly: It implicitly uses the NodeJS standard library's types but
so far had not declared them on the package level.
4. This change is therefore to rectify the issue of the `@types/node`
dependency missing from the test-tooling package and also the refactoring
of some of the test ledger classes which were relying on the `streams`
builtin package instead of correctly using the NodeJS.ReadableStream global.
5. Earlier the reasoning for this was that we try to avoid pulling in
types from the global scope because we try to avoid any sort of dependency
on the global scope in general. Once we have proof though that this is
causing issues with the build, then we must give up the principle for
practical reasons (and only in the minimum viable scope, e.g. this does
not change the fact that everywhere else in the codebase we should still
do our best to avoid using the global scoped classes, types, functions,
etc..).

Thank you to @AndreAugusto11 and @RafaelAPB for pointing out this issue
through the pull request of his that is currently being worked on at the
time of this writing:
RafaelAPB#72

Related to but does not address hyperledger-cacti#2811

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>

refactor(SATP-Hermes): remove unused packages

* updated openapi version
* updated bungee plugin version to 2.0.0-alpha.2

Co-authored-by: Peter Somogyvari <peter.somogyvari@accenture.com>

Signed-off-by: André Augusto <andre.augusto@tecnico.ulisboa.pt>
Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
Authored by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(satp-hermes): satp proto update (hyperledger-cacti#3147)

* feat(satp-hermes): update protos

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

* feat(satp-hermes): protos

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

* feat(satp-hermes): protos

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

---------

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

feat(satp-hermes): update protos

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

feat(satp-hermes): core stage services message names update

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

feat(satp-hermes): gRPC server start implementation and gateway_refactor in handlers

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

fix(satp-hermes): satp proto update

also updates yarn lock

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

fix(satp-hermes): update yarn, comment method

fix(satp-hermes): update yarn

ci(satp-hermes): fix bungee and satp ci

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>
Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

build(satp-hermes): add gateway client Go SDK

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(satp-hermes): re-add message proto

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>
Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>
Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

feat(satp-hermes): stage1-service and handler refactored

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

feat(satp-hermes):stage2-service and handler refactored server-side

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

feat(satp-hermes): service handlers server

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

feat(satp-hermes): client stage handlers refactor

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>
Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>
Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>
Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>
Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>
Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>
Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>
Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>
Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>
* Views have new attribute viewProof
* This attribute has the roots of all state proofs and transaction
proofs in the view
* Added new endpoint to verify integrity of view proofs
* Created strategy for bungee-hermes to support the ethereum-connector
* Created new test for the strategy-ethereum, and altered the API test
to test the new viewProof endpoint
* Updated documentation to reflect new changes.
* Fixes in Fabric Strategy, stateProofs and error handling

Co-authored-by: André Augusto <andre.augusto@tecnico.ulisboa.pt>
Co-authored-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

Signed-off-by: eduv09 <eduardovasques10@tecnico.ulisboa.pt>
Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>
Signed-off-by: Rafael Belchior <rbelchior@blockdaemon.com>
Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>
…atp protocol (hyperledger-cacti#3397)

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>
…tology (hyperledger-cacti#3382)

feat(satp-hermes): added forge to compile solidity contracts

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>
Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>
Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>
…i#3445)

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>
Signed-off-by: Iulia Mihaiu <maramih.2007@yahoo.com>
Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cc-tx-viz): add methods to fabric connector

Signed-off-by: Iulia Mihaiu <maramih.2007@yahoo.com>

fix(prometheus): metrics.ts leaks to global registry hyperledger-cacti#1202

1. Specified a `register` property of the gauges as an empty
array so that it does not pollute the global namespace. This
is how this is supposed to be done as per the docs of prom-client.

2. Once the change from 1) took place, the issue became that
the metrics gathering code was still trying to hit up the
global scope for the metrics, e.g. calling the get metrics
methods directly on the promClient object instead of the
registry that we create for each prmoetheus exporter object
separately. So a little additional refactor ensued to fix this
as well by making sure that we grab a reference of the registry
object at construction time and then re-use that wherever needed
instead of going through the global promClient object.

3. Added missing .startMetricsCollection calls in the plugin
constructors to ensure that the prometheus exporter object
gets initialized properly (this is where the registry gets
created as well so without this there are crashes happening
when one tries to access the metrics through the registry)

Why though?
The problem was that the metrics.ts file that we have for all the
plugin's metrics constructs a new Metric (Gauge) object at import
time which then defaults to registering the metric in the global
registry of prom-client by default.

The latter was causing crashes when separate versions of the same
metrics.ts file are imported in a scenario were the API server
imports plugins from a different directory (this issue is coming
from the branch where I'm working on plugin sandboxing via the
live-plugin-manager).

Fixes hyperledger-cacti#1202

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>

feat(cc-tx-viz): added test for multiple prometheus metrics on fabric

Signed-off-by: Iulia Mihaiu <maramih.2007@yahoo.com>

feat(cc-tx-viz): add metric model

Signed-off-by: Iulia Mihaiu <maramih.2007@yahoo.com>

feat(cc-tx-viz): added croschainEventLog models

Signed-off-by: Iulia Mihaiu <maramih.2007@yahoo.com>

feat(fabric): added endpoint for list of transaction receipts

Signed-off-by: Iulia Mihaiu <maramih.2007@yahoo.com>

feat(test-tooling): add RabbitMQ test server

feat(cc-tx-viz): add RabbitMQ support; update model

feat(cc-tx-viz): add preliminary support to Fabric and Besu

build(deps): upgrade axios to latest to fix CVE

Details
CVE-2021-3749
high severity
Vulnerable versions: <= 0.21.1
Patched version: 0.21.2
axios is vulnerable to Inefficient Regular Expression Complexity

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>

feat(cc-tx-viz): add methods to fabric connector

Signed-off-by: Iulia Mihaiu <maramih.2007@yahoo.com>

feat(cc-tx-viz): add metric model

Signed-off-by: Iulia Mihaiu <maramih.2007@yahoo.com>

xfix(cc-tx-viz): error implementation of IsVisualizable

feat(cc-tx-viz): add methods to fabric and besu; update models

Signed-off-by: Iulia Mihaiu <maramih.2007@yahoo.com>

feat(cc-tx-viz): update cross chain events log for fabric transactions

Signed-off-by: Iulia Mihaiu <maramih.2007@yahoo.com>

feat(cctxviz): fix fabric prometheus

feat(cctxviz): fix besu prometheus

feat(cctxviz): fix fabric test

feat(cctxviz): update tx receipt

feat(cctxviz): add test script

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cctxviz): add test receipts

feat(cctxviz): refactor CrossChainEventLog

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

test(cctxviz): refactor tests

feat(cc-tx-viz): update model, add tests

chore(cc-tx-viz): update dependencies

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cc-tx-viz): persist logs

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

test(cc-tx-viz): update test

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cc-tx-viz): update strinfify to csv

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

chore(cc-tx-viz): add csv folder

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

chore(cc-tx-viz): move csv folder

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cc-tx-viz): add test

feat(cctxviz): add csv file, process mining script

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cctxviz): proof of concept v1

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

chore(cctxviz): update csv

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

docs(cctxviz): update readme

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cctxviz): update tests

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cctxviz): add carbon footprint

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cctxviz): update tests

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cctxviz): update model

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(fabric-connector): update default queue

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

test(cctxviz): fabric test

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cctxviz): add parameters to fabric calls

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cctxviz): support besu

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cctxviz): update metrics

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

test(cctxvi): update fabric and besu tests

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

test(cctxviz): besu and fabric

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cctxviz): support call transactions on fabric

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cctxviz): add aggregation basis

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cctxviz): aggregate cctx

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

chore: update yarn and remove package.lock

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cctxviz): containerize draft

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cctxviz): add revenue

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cctxviz): update model playbook

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cctxviz): add dummy use case playbook

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

test(cctxviz): update persist test and dummy uc

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cctxviz): add timer

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cctxviz): workload test

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cctxviz): add more timers

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cctxviz): add timer to map on connectors

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cctxviz): update testing framework

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cctxviz): vis plot wip

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cctxviz): evaluate baseline

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cctxviz): update plots

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

test(cctxviz): add test files

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

feat(cctxviz): update testing framework

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

test(cctxviz): time-efficient polling receipts

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

chore(cctxviz): add test chaincode

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

test(cctxviz): update test

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

test(cctxviz): use case v1

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

test(cctxviz): update test framework

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

build(cc-tx-visualization): added missing fabric-contract-api

This will fix one compilation error.

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>

chore(deps): updated yarn.lock after rebase onto upstream/main

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>

build(cc-tx-visualization): deleted redundant test fixtures

Also added the test-tooling package as a dev dependency which
should fix a few of the compiler errors we were getting on account
of missing dependencies.

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>

fix: resolved build errors for cctxviz branch

- test-tooling package changed that needs to be reverted before commmit to main branch
- added import statement for test data generationt

xfix(cc-tx-viz): error implementation of IsVisualizable

chore: update yarn and remove package.lock

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

test(cctxviz): testing framework

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

test(cctxviz): test framework

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

test(cctxviz): update framework

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

test(cctxviz): add graphs

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

test(cctxviz): add test

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

fix(cctxviz): fix bug

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

test(cctxviz): add process conformance

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>
The plugin now utilizes RxJS instead of RabbitMQ in transaction monitoring.
ReplaySubjects store and emit observed transactional data to subscribers.

fix(cc-tx-visualization): rebase latest version

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

squash! - migrate a test case to Fabric v2.5.6 LTS AIO

This is just a snippet of a change from the pair programming
call with Bruno, it can be safely squashed, no worries.

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
Signed-off-by: Bruno Mateus <brumat315@gmail.com>

feat(cc-tx-visualization): ccmodel serialization with PM4PY

The plugin now creates the ccmodel for conformance checking with PM4PY.

Signed-off-by: Bruno Mateus <brumat315@gmail.com>
The plugin is now able to:
Create a Cross-Chain Model based on monitored transactions
Compare new transactions against the model and detect non-conformance

Signed-off-by: Bruno Mateus <brumat315@gmail.com>
Signed-off-by: eduv09 <eduardovasques10@tecnico.ulisboa.pt>
LordKubaya and others added 6 commits August 23, 2024 14:14
bug correction

session functionalities

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

feat(satp-hermes): errors file and other fixes

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

feat(satp-hermes): fix packages versions

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>
Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>
Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>
Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>
Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>
Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

feat(satp-hermes): satp rejection

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

feat(satp-hermes): handler bug correction and errors

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

fix(satp-hermes): bridge erros and log messages

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

fix(satp-hermes): some error fixes

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

fix(satp-hermes): some fixes requested

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>
Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>
@RafaelAPB
Copy link
Contributor

@Yogesh01000100 please rebase with satp-dev (should not have conflicts)

@Yogesh01000100 Yogesh01000100 force-pushed the feature/crash-recovery-improvements branch from 0de9744 to 4c0124d Compare August 23, 2024 17:19
@Yogesh01000100 Yogesh01000100 changed the title feat: add crash recovery and knex config for production feat(recovery): add crash recovery implementation Aug 25, 2024
@RafaelAPB
Copy link
Contributor

@Yogesh01000100 please include documentation and tests, and update the description, as discussed.

Co-authored-by: eduv09 <eduardovasques10@tecnico.ulisboa.pt>
Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

fix(satp-hermes): error messages in stage0

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

feat(satp-hermes): service tests added stage0 and error handling

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

feat(satp-hermes): test e2e

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

fix(satp-hermes): fix bugs and test e2e working transfer

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

fix(stap-hermes): bugs and 2 gateway test

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>
Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>
@Yogesh01000100 Yogesh01000100 force-pushed the feature/crash-recovery-improvements branch from ce9a179 to 24b8eaf Compare September 8, 2024 07:44
Signed-off-by: Eduardo Vasques <eduardovasques10@tecnico.ulisboa.pt>
Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

refactor(cbdc-example): update with new satp-plugin

Signed-off-by: eduv09 <eduardovasques10@tecnico.ulisboa.pt>

refactor(cbdc-example): fixes

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>
<
Signed-off-by: Eduardo Vasques <eduardovasques10@tecnico.ulisboa.pt>

fix(satp-hermes): bug fixing

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

fix(cbdc-example): bug fixing

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

feat(cbdc): final fixes

Signed-off-by: Eduardo Vasques <eduardovasques10@tecnico.ulisboa.pt>

fix(cbdc-example): remove unecessary variable

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

fix(satp-hermes): remove unessary logs

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>

fix(cbdc-example): docker ports

Signed-off-by: Carlos Amaro <carlosrscamaro@tecnico.ulisboa.pt>
@Yogesh01000100 Yogesh01000100 force-pushed the feature/crash-recovery-improvements branch from 24b8eaf to 728e7cb Compare September 16, 2024 18:56
@RafaelAPB
Copy link
Contributor

@Yogesh01000100 could you please squash the commits and rebase with latest version of satp-dev, prior to merge?

* Separated Transfer functionality into local blockchain transfers
and cross-chain transfers
* Added new table with the state of token approvals made to the bridge
* Highlight cross-chain functionality with red coloring
* add helper page with instructions on how to use app

Signed-off-by: André Augusto <andre.augusto@tecnico.ulisboa.pt>
@Yogesh01000100 Yogesh01000100 force-pushed the feature/crash-recovery-improvements branch from 728e7cb to 1a55673 Compare September 17, 2024 10:02
Signed-off-by: Yogesh01000100 <yogeshone678@gmail.com>

chore(satp-hermes): improve DB management

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

chore(satp-hermes): crash recovery architecture

Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

fix(recovery): enhance crash recovery and rollback implementation

Signed-off-by: Yogesh01000100 <yogeshone678@gmail.com>

refactor(recovery): consolidate logic and improve SATP message handling

Signed-off-by: Yogesh01000100 <yogeshone678@gmail.com>

feat(recovery): add rollback implementations

Signed-off-by: Yogesh01000100 <yogeshone678@gmail.com>

fix: correct return types and inits

Signed-off-by: Yogesh01000100 <yogeshone678@gmail.com>
@Yogesh01000100 Yogesh01000100 force-pushed the feature/crash-recovery-improvements branch from 1a55673 to 21ad772 Compare September 17, 2024 10:11
Copy link
Contributor

@RafaelAPB RafaelAPB left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks very good, but there are some changes to be done prior to merging.
Summarizing my comments:

  1. Add other authors to the commit
  2. Incorporate feedback from the logging process (namely un-hardcoding logs and adding more information)
  3. Implement RollbackState (for example, should state how many more steps are to be rolled-back, at any moment; what was rolledback already; estimated time to completion, etc)
  4. Please add tests that support the new feature
  5. Please add comprehensive documentation on this feature. Example: The readme of SATP should have a section on how to run the docker compose with several examples of configurations.

@@ -0,0 +1,17 @@
version: '3.8'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will need to do a few changes. We will have a satp runner image that encapsulates satp. The docker file is being developed by Bruno (please sync with him). In particular, please note what are the necessary env variables that need to be fed to docker compose.

See suggestion below:

version: '3.8'

services:
  db:
    image: postgres:13
    environment:
      POSTGRES_DB: ${DB_NAME}
      POSTGRES_USER: ${DB_USER}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_HOST: ${DB_HOST}
      PGPORT: ${DB_PORT}
    ports:
      - "${DB_PORT}:5432"
    volumes:
      - pgdata:/var/lib/postgresql/data

  satp-runner:
    image: satp-runner-gateway:latest  #to be published
    environment:
      # environment variables your SATP runner needs
    ports:
      - "3000:3000"  #  port mapping
    depends_on:
      - db

volumes:
  pgdata:

Additionally, we will need a load balancer, and other services for monitoring (@eduv09 will implement those). We would have something like this:

  nginx:
    image: nginx:latest
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "80:80"
    depends_on:
      - satp-runner

  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:latest
    volumes:
      - grafana_data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_USER=${GRAFANA_ADMIN_USER:-admin}
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD:-admin}
    ports:
      - "3000:3000"
    depends_on:
      - prometheus

@@ -78,7 +78,16 @@
"watch": "tsc --build --watch",
"forge": "forge build ./src/solidity/*.sol --out ./src/solidity/generated",
"forge:test": "forge build ./src/test/solidity/contracts/*.sol --out ./src/test/solidity/generated",
"forge:all": "run-s 'forge' 'forge:test'"
"forge:all": "run-s 'forge' 'forge:test'",
"db:setup": "bash -c 'npm run db:destroy || true && run-s db:start db:migrate db:seed'",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did many contributions to this code. Unfortunately the squash you did removed that history.

Please add me as a co-author to the squashed commit:
https://docs.github.com/en/pull-requests/committing-changes-to-your-project/creating-and-editing-commits/creating-a-commit-with-multiple-authors

}

// todo read from local log to get session data
/*private async getSessions(): Map<string, SessionData> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this commented?

const logEntry = {
sessionID: req.sessionId,
type: "RECOVER_SUCCESS",
key: "key", // generateKey(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this commented?

sessionID: req.sessionId,
type: "ROLLBACK_",
key: "key", //generateKey(),
operation: "ROLLBACK_",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same

throw new Error(`${fnTag}, session data is not correctly initialized`);
}
try {
// TODO record the rollback on the log. Implement RollbackLogEntry
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove todos when the task is completed. I see you are already recording the rollback log entry (despite being a dummy entry). Please un-hardcode it, and gather information from the current session

try {
// TODO: Implement Stage 1 specific rollback logic

// TODO: Record the rollback on the log. Implement RollbackLogEntry
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please un-hardcode the log creation where applicable

try {
// TODO: Implement Stage 2 specific rollback logic

// TODO: Record the rollback on the log. Implement RollbackLogEntry
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please un-hardcode the log creation where applicable.
For instance, details can be a subset of the execution trace to the moment. Remember, these logs should help developers finding bugs or understanding the state of the application at any moment

// TODO: Implement Stage 3 specific cleanup logic

state.currentStage = "Stage3";
// TODO: Update other state properties as needed
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see comments from other stages

@@ -1,4 +1,4 @@
// @generated by protoc-gen-es v1.7.2 with parameter "target=ts"
// @generated by protoc-gen-es v1.8.0 with parameter "target=ts"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let us please fix the latest working version of the generator in package.json (if not done yet)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants