feat(cacti-gui): cacti graphical interface proposal

[outSH]

Hyperledger Cacti GUI

We (Fujitsu) are currently implementing a universal GUI for Hyperledger Cacti project. I would like to present and describe our proposal, justify some decisions and start the discussion about the future improvements and development. Our goal is to create a common GUI that would allow exploring multiple ledgers states in single place and simplify visualization of custom solutions (BLPs) based on Hyperledger Cacti.

At the moment, we are working on delivering GUI for Ethereum ledgers only, but the overall architecture is extensible and applicable for additional ledgers. We plan to submit the PR with Ethereum GUI by the end of this year (2022) and continue working on this feature next year.

Summary

• Each ledger will have an additional (optional) plugin (cactus-plugin-persistence-X)
• The plugin will monitor ledger state, parse it, and store it to the data storage (for now - PostgreSQL)
• PostgreSQL will be wired up with Supabase, a backend solution that provides some useful features like automatic API generation, authentication, server side functions, etc...
• GUI app will query and display ledger data from PostgreSQL using Supabase APIs.

Tasks

• Add supabase-all-in-one image to be used for tests -> feat(supabase-all-in-one): add docker image for test supabase instance #2253
• Extend go-ethereum connector with methods required by persistence plugin -> connector-go-ethereum: getBlock and getTransactionReceipt methods #2255
• Add new package @hyperledger/cactus-plugin-persistence-ethereum -> feat(cactus-plugin-persistence-ethereum): add new persistence plugin #2258
• Add GUI package. -> feat(cactus-cmd-gui-app): add common cacti gui app #2264

Cacti Components

Ethereum Persistence Plugin

• New kind of ledger-specific plugin. Will be closely related to the same ledger connector plugin.
• Its main purpose is to observe a new ledger data, parse it accordingly and store in the external data storage.
• For now, we assume a single target data storage (PostgreSQL), but we want the actual data access layer to be separated from the main plugin logic. Thanks to lose coupling of both elements, it will be easy to replace the data storage in the future when needed
  (some users may prefer NoSQL or big data solutions, this plugin could even be used to push new data through queue solution for advanced, massive processing when needed).
• Component is meant to run alongside connector or, in general, inside internal Cacti network (i.e. not accessible for everyone).
• Main features:
  • Monitoring blocks and transactions, storing them in the DB.
  • Monitoring operations on specific tokens, storing details of token operations in the DB.
  • Synchronizing issued ERC721 tokens.
  • Keeping track of and updating user ERC20 token balances.
  • Synchronizing missing data from the past (in particular, filling block data missed because of connection or runtime issue). This will make this service recoverable.
• Disclaimers:
  • For now, we assume that only single persistence plugin per ledger can be running. Multiple instances can interfere with one another and there'll be no mechanism to prevent that at the moment.
  • There may be noticeable lag from transaction being accepted and appearing in the GUI. Our current goal is eventual consistency and not a real time block explorer.

Ledger Data Storage

• For now, we assume it's PostgreSQL. The database can be managed outside of the Cacti scope (i.e., own deployment, cloud instance - depends on user. Supabase ships one for development purposes).
• Single storage solution will support all the configured ledgers.
• Each ledger family will be kept in separate space (either different schema or using different table naming scheme - to be decided).
• For now we assume single ledger (chain) of one ledger family can exist in the database. For example, we assume only one Ethereum ledger is used at a time. This can be improved in the future if needed.
• Ledger database schema (tables, views, etc...) will be initialized by the persistence plugin (if it wasn't created yet). A mechanism for determining which ledgers are configured will be added in the future, once we add more ledgers.

Supabase API

• For GUI backend we decided to use Supabase.
• Supabase is an open source Firebase alternative - https://supabase.com/
• In short, it automatically generates API for accessing DB data, handles authentication, edge functions, subscriptions and storage.
• Issues:
  • Although Supabase is popular and actively developed (see https://github.com/supabase/supabase), it's still fairly young technology so we should proceed with caution and monitor any undesired behaviors and issues on the way.
  • Nevertheless I think it's safe to use for our case, since the actual data is being stored in PostgreSQL directly, so it will be easy to replace with custom service in the future if needed.
  • The great benefit of Supabase is that not only it speeds up the GUI development, but will also support rapid prototyping and development of any future PoC based on Cacti.

GUI Server

• This is an optional component that will simply serve a web app to the user.
• In the future it can handle user authentication and configuration (reading Cacti setup from consortium, etc..)
• Initially this package will contain the GUI to be run manually, in the future it can be plugin loaded by cmd-api-server.

GUI application

• This is a GUI app written in SolidJS (https://www.solidjs.com/)
• SolidJS builds upon popular ReactJS framework (https://reactjs.org/), so it will be easy to learn and use for many front-end developers and, when needed, can be quickly refactored back to React (the syntax is really similar).
• There'll be a separate view for each ledger, each will differ from each other but should maintain common visual theme.
• Each ledger app could ideally be implemented in an independent Micro-Frontend manner, but this is still in progress. For now we focus on delivering a single GUI app.

Ethereum Connector Plugin

• For now we'll use existing SocketIO based connector-go-ethereum for communicating with the Ethereum ledger.
• Connector plugin already support core required features, but the following will be implemented:
  • Flag to monitor all the blocks (currently it filters-out empty blocks)
  • Function to read specific block from the ledger (for backfill feature).

Deployment

• Each plugin should adhere to common cacti plugin behavior, i.e. any combination of plugins can be used in single cmd-api-server.
• For now we assume Persistence plugin is separate from the connector plugin (since go-ethereum-connector is not openapi compatible), so we assume following deployment:

• In case open-api connector plugin is used, persistence plugin can be deployed alongside it and I think this would be recommended setup.
• Persistence plugin can detect that connector is available on the same instance, and use internal API calls instead of expensive web calls.
• This will make persistence plugin a nice (optional) extension of connector from end-user perspective.

Glossary

• BLP: Business Logic Plugin - In Cacti, a component implementing custom ledger integration logic.
• Connector Plugin: In Cacti, a component responsible for sending communicating directly with ledger node. Currently we have open-api based ones (can be included in cmd-api-server), and SocketIO ones (that are stand-alone services).
• cmd-api-server: Deployable service containing multiple Cacti plugins.
• ERC721: A Non-Fungible Token (NFT) used to identify something or someone in a unique way.
• ERC20: Fungible Tokens - they have a property that makes each Token be exactly the same (in type and value) as another Token.
• PoC: Realization of a certain method or idea in order to demonstrate its feasibility.
• Micro-Frontend: Techniques, strategies and recipes for building a modern web app with multiple teams that can ship features independently.

[outSH]

@petermetz @jagpreetsinghsasan Please review and share your comments and opinion :)

[jagpreetsinghsasan]

I have several points regarding this.

1. How is this different from feat(scorpion): ledger explorer graphical user interface #1979 (specifically point 5 of it)?
2. Can we not use reactjs instead of solidjs (if there is no significant usage of solidjs, which reactjs doesnt provide)? This is just to reduce the learning overhead for easier external contribution.
3. Is it better to have a general GUI plugin (just like we have cactus-api-server, we can have cactus-gui-server) and the existing plugins to have functions to provide data to this GUI plugin? In this way we can leverage the existing prometheus exporter and build around those real-time metrics.
4. In the very first screenshot, there are 2 storages Ethereum Ledger and Ledger Data Storage. I dont think we should create the Ledger Data storage, because from the description of it, this itself seems like a Cactified variant of a group of ledgers, some sort of cached DB. Instead of this, we can develop read-only ledger functions within the original plugin and use them.

I would like to propose something like this for the cactus-gui

Grafana dashboard which dynamically loads all the connector plugins hosted over the gui-server.

Benefits :

1. Low maintenance of libraries, technologies, security vulnerabilities of dependencies compared to full-blown reactjs/solidjs based UI
2. Prometheus-grafana combination is one of the popular go-to options for dashboards with metrics.
3. Cacti stays more independent of the underlying ledgers (as all we will be doing is to update the prometheus exporter codes only). This will also come in handy when the underlying ledgers go significant changes.
4. If this is targeted for production grade environments, clients can use their patched/premium versions of grafana, as there can be multiple scenarios where some clients aren't comfortable with the technologies or dont have the right tools to patch them for security vulnerabilities.

I might be completely wrong in understanding this, so please review and suggest @outSH @petermetz

[outSH]

@jagpreetsinghsasan

1. How is this different from [feat(scorpion): ledger explorer graphical user interface #1979](https://github.com/hyperledger/cactus/issues/1979) (specifically point 5 of it)?

This is mostly in-line with peter task description, but aligned to our internal requirements.

2. Can we not use reactjs instead of solidjs (if there is no significant usage of solidjs, which reactjs doesnt provide)? This is just to reduce the learning overhead for easier external contribution.

As I've mentioned previously, SolidJS builds upon ReactJS, uses the same concepts and fairly similar syntax, so learning curve shouldn't be an issue. We can replace it with react if we need more mature libraries (possibly in a next step).

3. Is it better to have a general GUI plugin (just like we have cactus-api-server, we can have cactus-gui-server) and the existing plugins to have functions to provide data to this GUI plugin? In this way we can leverage the existing prometheus exporter and build around those real-time metrics.

I'm not sure if I understand you correctly, but I think this is similar to replacing supabase with a separate service that I've also described above as an option. But, as mentioned above, I'd like to stick with supabase for now because this gives us a lot of functionality and flexibility for free.

4. In the very first screenshot, there are 2 storages `Ethereum Ledger` and `Ledger Data Storage`. I dont think we should create the `Ledger Data storage`, because from the description of it, this itself seems like a `Cactified variant of a group of ledgers`, some sort of cached DB. Instead of this, we can develop `read-only ledger functions` within the original plugin and use them.

It's not really a cached DB, but an actual persistent DB (hence the name of the components that feeds data into it - persistence plugins). The backend storage layer is our internal core requirement for the GUI component. The whole concept is based on storing the parsed ledger data in the DB, and being able to make simple or complex cross-ledger queries and analytics on it (and display it when necessary).

Another key reason for storing the data in DB is performance. Even a simple operation like rendering the list of latest 100 blocks would cause 100 separate requests on ledgers that don't support a bulk read, if we were to use connector function calls ad hoc. Showing history of account token balance would be nearly impossible.

In general we don't want the GUI for the cacti itself (as the prometheus & grafana solution would suggest), but efficient way to visualize cross-ledger data.

[aruns05]

@outSH Is this being worked upon?

[outSH]

@aruns05 Yes, we will be pushing PR with GUI components next week :)

[aruns05]

@outSH can we speak about it . I tried to reachout to you in discord