Scalability: Artifact (Policy, VC, Schema) size and internal DB #1155
Assignees
Labels
Comments
prernaadev01
added
documentation
anvabr
changed the title
Merged
anvabr
pushed a commit
that referenced
this issue
Apr 25, 2023
### Headlines - Tagging of Policy Artifacts - Improving Policy Configurator Blocks using Colors and Shapes - Scalability of Artifact Size and DB - Secrets in Pipeline ### Enhancements - Tagging of Guardian policy artifacts [#1661](#1661) - Scalability: Artifact (Policy, VC, Schema) size and internal DB [#1155](#1155) - API contract reviewing [#1629](#1629) - Develop guardian-cli to manage project [#1636](#1636) - Dynamic Topic/Message configuration according to the environment [#1696](#1696) - Monitoring tools - Application events [#1842](#1842) - Multi configuration of persistance according to the environment [#1888](#1888) - Develop cypress test suite for contracts [#1816](#1816) - Secrets in pipelines [#1610](#1610) - Expose subscribed event or log messages to an external application [#1634](#1634) - API contract review - Phase one [#1805](#1805) - Direct service-to-service communication for large payloads [#1998](#1998) - Verra Reforestation Development [#2023](#2023) - Use colours and shapes to highlight blocks properties and/or context [#1795](#1795) ### Bugs Fixed - `policyRolesBlock` web interface allows me to choose role multiple times [#1980](#1980) - Invalid schema owner [#1972](#1972) - I cannot exit the code editor without saving the content of the expression [#1981](#1981) - Persistent data gets dropped between application restarts [#1950](#1950) - Show transaction id in log file of guardian-service [#1983](#1983) - Unable to publish policy [#1978](#1978) - Issue while setup, after new release: unable to Finish Setup [#1971](#1971) - Potential memory leak in worker service [#1926](#1926) - Wrong behaviour for minting token process for multipolicies [#2019](#2019) - Large amount of tokens being minted [#1976](#1976) - Clogging of queue due to high volume of tokens minted. [#2005](#2005) - Progress Bar when performing Schema operations are missing [#2046](#2046) - Unable to configure Guardian services [#1999](#1999) * fix docker compose Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * delete folder Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * update vc in tags * fixes Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * share tasks Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * service scripts Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * clean code Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * policy instance container Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * fix tests Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * corrected broken links in ReadMe * added versions to ReadMe * increase max hedera timeout Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * update Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * bump version Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * fix workers queue Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * test fix Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * wss updates Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * fix Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * fix Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * fix block errors Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * fix messageId Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * Add documentation for Metered Energy Cooking policy * Add policy workflow, registry and project proponent's responsibilities * Complete policy guide * Add TOC * Add futureproofing section * Add futureproofing section * Add policy comparison * Add futureproof flow diagram * Add policy import screenshot * Add policy file * update blocks * fix documents grid updates Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * fix Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * lint Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * fix loading for event blocks Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * GITBOOK-308: change request with no subject merged in GitBook * fix duplicate update block messages Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * added loading to token confirmation ui Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * update swagger Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * fix loading in event blocks Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * Add relevant demo video links * moved entities from guardian to common repo Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * remove entities from policy-service Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * chunking db entities Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * fix build Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * fix lint and tests Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * fix duplicate transaction logs Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * update Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * fix Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * update entities Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * added chunking property to send block Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * update common package json Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * fix contract tags * lint * Show transaction id in the log Signed-off-by: Giuseppe Bertone <giuseppe.bertone@swirldslabs.com> * fix update token Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * remove logs Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * moved to common * fix lint Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * - #1888 - #1696 integrate previous changes to Dokerfile docs and PREUSED GUARDIAN_ENV in .env Docs: Docker Compose V2 version lint Signed-off-by: otherNet <francesco.calderoni@intellecteu.com> * fix transaction logs Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * fix tag schemas Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * fix delete tag schemas Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * #1981 Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * automatically document cache Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * fix lint Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * reset send block properties * moved tests Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * set document cache to aggregate block Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * fix creation of vp document Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * fix publishing schemas * fix comments and blinking request dialog * themes Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * added defs to schemas Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * added document cache fields to external block Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * fix updating files Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * fix updating multi sign block Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * fix lint Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * update * merge new secrets to dev Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * merge new secrets to dev Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * merge new secrets to dev Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * improve minting tokens Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * fix build Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * update worker tasks priority Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * fix user keys Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * added loading tokens on creation retire request Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * fix user keys oldstyle Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * fix startup values Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * fix api tests Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * fix api tests Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * .env files Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * fix lint Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * fix common sorting in grids Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * fix publishing policies Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * update * added readme of modules * added docs link of modules * GITBOOK-309: Adding subpage for the Guardian CLI under "Frameworks/Libraries" * GITBOOK-310: added Guardian CLI * added BATCH_NFT_MINT_SIZE setting Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * feat: application-events module Signed-off-by: Felipe Neuhauss <felipe.neuhauss@gmail.com> * update Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * fix updating entities Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * fix exporting modules Signed-off-by: artembuslaev <buslaew.ar@mail.ru> * update Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * GITBOOK-311: Added Tymlez Policy demo guide * feat: api contract review Signed-off-by: Felipe Neuhauss <felipe.neuhauss@gmail.com> * update Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * filter by api Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * GITBOOK-312: Added env parameters for guardian-service * update Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * added handing non retryable errors Signed-off-by: artembuslaev <artem.buslaev@envisionblockchain.com> * added logs for mint Signed-off-by: artembuslaev <artem.buslaev@envisionblockchain.com> * [Cookstove Policy] Publish new policy version and readme with future TODOs * update * large payloads Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * docker-run Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * fix .env Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * add themes api Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * remove quotes Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * fix port expose Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * update Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * feature/user-driven-mrv * GITBOOK-313: adding instructions of changing operator ID and Key in settings * fix api test Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * fix 404 error Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * fix timeout Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * devide test Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * add auth header Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * add destructor Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * remove old test Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * update theme api Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * update swagger Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * feat: application-events additional fixes Signed-off-by: Felipe Neuhauss <felipe.neuhauss@gmail.com> * change get balances Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * rest balance Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * feat: application-events reduced events to avoid annoying notifications Signed-off-by: Felipe Neuhauss <felipe.neuhauss@gmail.com> * fix Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * GITBOOK-314: added parameters for services * GITBOOK-315: Adding Tag UI for Policies and Schemas * secret manager update Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * added duplicate transaction handler Signed-off-by: artembuslaev <artem.buslaev@envisionblockchain.com> * update Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * feat: application-events module.ts updated Signed-off-by: Felipe Neuhauss <felipe.neuhauss@gmail.com> * Fixed userInfo error * fix #2019 Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * GITBOOK-317: Tags APIs for Policies * GITBOOK-318: Schema Tag APIs added * GITBOOK-319: corrected errors in tags * Create API Guideline * Delete API Guideline * add settings validation Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * update Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * fix Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * fix Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * fix label Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * fix update settings Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * update Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * fix Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * fix legends Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * fix application events listener; add service to publish CI Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * Fixed conflict * Fixed conflict 2 * added new policies [skip ci] Signed-off-by: artembuslaev <artem.buslaev@envisionblockchain.com> * slice users Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * fix lint Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * Added iREC 9 in ReadMe * fix version Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * Added Verra ARR ReadMe * update gold standard ar policy Signed-off-by: artembuslaev <artem.buslaev@envisionblockchain.com> * fix filters in button block Signed-off-by: artembuslaev <artem.buslaev@envisionblockchain.com> * Deleted Branding related code * Adding Gold Standard AR ReadMe * GITBOOK-320: Added Themes UI * GITBOOK-321: Adding Theme APIs * Add files via upload * hotfix/progress-bar-schemas * API Guideline This documentation is to underline the different types of validation status code and any error status code, including Naming Conventions for RESTful APIs. This documentation is apart of the API contract review PR #1974 and issue #1629 * fix duplicate schema field label Signed-off-by: artembuslaev <artem.buslaev@envisionblockchain.com> * updated policy files [skip ci] Signed-off-by: artembuslaev <artem.buslaev@envisionblockchain.com> * fix merge Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> * bump version Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> * GITBOOK-322: Guidance for Open Source Policy * Updated Verra ARR Timestamp * Added Schema * Updated Gold Standard AR Timestamp --------- Signed-off-by: simvalery <valeriy.simonov@envisionblockchain.com> Signed-off-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> Signed-off-by: artembuslaev <buslaew.ar@mail.ru> Signed-off-by: Giuseppe Bertone <giuseppe.bertone@swirldslabs.com> Signed-off-by: otherNet <francesco.calderoni@intellecteu.com> Signed-off-by: Felipe Neuhauss <felipe.neuhauss@gmail.com> Signed-off-by: artembuslaev <artem.buslaev@envisionblockchain.com> Co-authored-by: Stepan Kiryakov <stepan.kiryakov@envisionblockchain.com> Co-authored-by: prernaadev01 <79293833+prernaadev01@users.noreply.github.com> Co-authored-by: Gautam Prajapati <gautam.prajapati@airbnb.com> Co-authored-by: Gautam Prajapati <gautamprajapati06@gmail.com> Co-authored-by: artembuslaev <buslaew.ar@mail.ru> Co-authored-by: prernaa.agarwal <prernaa.agarwal@envisionblockchain.com> Co-authored-by: artembuslaev <68849616+artembuslaev@users.noreply.github.com> Co-authored-by: Giuseppe Bertone <giuseppe.bertone@swirldslabs.com> Co-authored-by: Stepan Kirjakov <84465157+Stepan-Kirjakov@users.noreply.github.com> Co-authored-by: otherNet <francesco.calderoni@intellecteu.com> Co-authored-by: Serg Metelin <sergey.metelin@hedera.com> Co-authored-by: Felipe Neuhauss <felipe.neuhauss@gmail.com> Co-authored-by: artembuslaev <artem.buslaev@envisionblockchain.com> Co-authored-by: E. Santiago <eduardo.santiago@intellecteu.com> Co-authored-by: E. Santiago <eduardo.augusto.santiago@gmail.com> Co-authored-by: Vanessa Lopes <111521009+vaness453@users.noreply.github.com> Co-authored-by: Eduardo Santiago <91878554+PensoGlide@users.noreply.github.com>
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Problem description
In some situations where lots of MRV data (or a very big Policy/Schema document) is generated and the artifact sizes can be >100MB which would likely be a problem for both NATS and MongoDB.
Requirements
Design and execute tests which measure Guardian limits wrt the size of documents it can comfortably work with, translating these into the metrics for input parameters (e.g. MRVs) influencing these sizes.
Definition of done
Acceptance criteria
The text was updated successfully, but these errors were encountered: