Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: out of process plugin instances for language independence #475

Conversation

petermetz
Copy link
Contributor

How to Review

This PR depends on two others: #470 and #472 to ensure that the authors are not blocked in performing their work while waiting for the pull request to get reviewed. Please read the each point of the description below carefully!

  • It is strongly recommended that you review the parent PR(s) first before reviewing this one because once that one (parent PR) is merged, this can be rebased onto the main branch which will literally disappear the additional commits that are shown in this PR but actually belong to the parent PR
  • Before this one can be merged, fix(npm): clean script was missing folders #470 and feat(test-tooling): containers#getById and #waitForHealthCheck #472 have to be approved and merged first.
  • Reviewing this one can be done as of right now by looking at the changes made specifically by the commit(s) that this branch commits on top of the branch of the parent branch (parent PR)
  • Feel free to ask @petermetz via any of your preferred communication channels for help if you get stuck using the GitHub UI or any git features during the review.
  • You can use the 'commits' tab of the PR page of GitHub to filter down the diff to a specific commit which allows you to not be bothered by the other commits at all (which belong to the parent PR). To see which commit to filter the diffs down onto, see the section below titled as "commit to be reviewed".

Commit(s) to be reviewed:

commit 12389d040b596642795cab24572177616c9105d3
Author: Peter Somogyvari <peter.somogyvari@accenture.com>
Date:   Sun Jan 10 20:23:36 2021 -0800

    feat(keychain): add rust keychain plugin vault implementation
    
    This is partially a new feature, partially an example to
    demonstrate how one can develop a plugin written in
    in rust that then will be callable by other (non-rust) plugins
    that are hosted inside a Cactus node's API server(s).
    
    99% of the rust code here was generated by the command below:
    
    docker run --rm -v "${PWD}:/local" openapitools/openapi-generator-cli generate \
        --input-spec https://raw.githubusercontent.com/petermetz/cactus/feat/cmd-api-server/out-of-process-plugin-Instances-for-language-Independence-170/packages/cactus-plugin-keychain-vault/src/main/json/openapi.json \
        --generator-name rust-server \
        -o /local/gen/rust-server
    
    The only thing that was then added to the generated rust code
    is the logic that uses the vault client create to do the
    get/set operations in vault and parses the environment
    variables for the vault token, host.
    Still a to-do for the above to have a proper has/delete method
    in the rust code which was not implemented due to time constraints.
    
    Also added a Typescript implementation of the vault keychain plugin
    just so that we can have a full end to end test coverage of all the
    variants of configurations where the plugin is local vs. remote.
    
    Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
commit 137ee16cae64cd906c2542928888f1524c8a8f94
Author: Peter Somogyvari <peter.somogyvari@accenture.com>
Date:   Fri Jan 8 15:17:01 2021 -0800

    feat(keychain): implement OpenAPI endpoints
    
    WORK IN PROGRESS
    
    Specifically for the in-memory and the vault
    plugin implementations.
    
    Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
commit 2be5eedbd3fb229715cc3e313aed7c728be8a712
Author: Peter Somogyvari <peter.somogyvari@accenture.com>
Date:   Fri Jan 8 09:08:13 2021 -0800

    feat(core-api): plugin import types: LOCAL & REMOTE
    
    List of files with the interesting changes
    =================================
    
     2. For `REMOTE` type of plugin imports, the factory returned will create an
     API client object, configured to point to an arbitrary implementation of
     the plugin over the network (which is how we enable/unlock the possibility
     to have language independent plugin implementations since by specifying
     `REMOTE` when importing a plugin, you can provide a network host where a
     plugin is deployed that was implemented in your preferred programming
     langugage rather than Typescript/Javascript for example.)
    
     Important note:
     When specifying `REMOTE` as the plugin import type, you still need to also
     specify the `packageName` property pointing to an npm package that has the
     API client class definition and the corresponding factory in it. The local
     implementation however does not have to be present in that npm package,
     which is the whole point.
    
     To provide a specific example of the above case: Writing a ledger connector
     plugin in a non-NodeJS language (Rust, Go, Java, etc.) can be done in the
     following way:
     1. Define the API client for your plugin in Typescript (only needed if
     not already defined for the ledger of your choice)
     We recommend auto-generating this after having written a platform/language
     netural OpenAPI spec file first. This API client doesn't provide any
     actual implementation of your plugin, it just maps method names of your
     plugin to HTTP requests that your actual plugin implementation (in your
     language of choice) will have to be able to service.
     2. Write the actual code of your plugin in your language of choice.
     3. Profit.
    
     Also note: If you are re-implementing a connector plugin in a different
     language that already has an implementation in NodeJS/Typescript, then
     you can skip the step above that has you publish the npm package with the
     API client class and really only have to work in your language of choice.
    
    Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>

Fixes #170

Depends on #470
Depends on #472

@petermetz petermetz added enhancement New feature or request SPIKE Exploratory work to better scope additional effort API_Server dependencies Pull requests that update a dependency file Security Related to existing or potential security vulnerabilities Developer_Experience Significant_Change Applying this label triggers the more stringent review of the maintainers and the 2+1 PR rule. labels Jan 13, 2021
@petermetz petermetz added this to the v0.4.0 milestone Jan 13, 2021
@petermetz petermetz enabled auto-merge (rebase) January 13, 2021 08:23
Copy link
Contributor

@jonathan-m-hamilton jonathan-m-hamilton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

List of files with the interesting changes
=================================

The rest of the diff  is mostly just compiler worship.

./packages/cactus-core-api/src/main/typescript/openapi-spec.ts
./packages/cactus-core-api/src/main/typescript/plugin-factory-factory.ts
./packages/cactus-core-api/src/main/typescript/i-plugin-factory-options.ts
./packages/cactus-cmd-api-server/src/main/typescript/api-server.ts

Primary change
=============

 The original plugin import type that the API server received via configuration
 (via ENV, CLI or FILE).
 The `type` property of the plugin import can be used by the different
 `createPluginFactory()` implementations to determine the kind of factory
 they need to return. For example:

 1. For `LOCAL` type of plugin imports, the factory returned will construct
 the actual plugin implementation class (e.g. directly instantiate it)

 2. For `REMOTE` type of plugin imports, the factory returned will create an
 API client object, configured to point to an arbitrary implementation of
 the plugin over the network (which is how we enable/unlock the possibility
 to have language independent plugin implementations since by specifying
 `REMOTE` when importing a plugin, you can provide a network host where a
 plugin is deployed that was implemented in your preferred programming
 langugage rather than Typescript/Javascript for example.)

 Important note:
 When specifying `REMOTE` as the plugin import type, you still need to also
 specify the `packageName` property pointing to an npm package that has the
 API client class definition and the corresponding factory in it. The local
 implementation however does not have to be present in that npm package,
 which is the whole point.

 To provide a specific example of the above case: Writing a ledger connector
 plugin in a non-NodeJS language (Rust, Go, Java, etc.) can be done in the
 following way:
 1. Define the API client for your plugin in Typescript (only needed if
 not already defined for the ledger of your choice)
 We recommend auto-generating this after having written a platform/language
 netural OpenAPI spec file first. This API client doesn't provide any
 actual implementation of your plugin, it just maps method names of your
 plugin to HTTP requests that your actual plugin implementation (in your
 language of choice) will have to be able to service.
 2. Write the actual code of your plugin in your language of choice.
 3. Profit.

 Also note: If you are re-implementing a connector plugin in a different
 language that already has an implementation in NodeJS/Typescript, then
 you can skip the step above that has you publish the npm package with the
 API client class and really only have to work in your language of choice.

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
Specifically for the in-memory and the vault
plugin implementations.

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
This is partially a new feature, partially an example to
demonstrate how one can develop a plugin written in
in rust that then will be callable by other (non-rust) plugins
that are hosted inside a Cactus node's API server(s).

99% of the rust code here was generated by the command below:

docker run --rm -v "${PWD}:/local" openapitools/openapi-generator-cli generate \
    --input-spec https://raw.githubusercontent.com/petermetz/cactus/feat/cmd-api-server/out-of-process-plugin-Instances-for-language-Independence-170/packages/cactus-plugin-keychain-vault/src/main/json/openapi.json \
    --generator-name rust-server \
    -o /local/gen/rust-server

The only thing that was then added to the generated rust code
is the logic that uses the vault client create to do the
get/set operations in vault and parses the environment
variables for the vault token, host.
Still a to-do for the above to have a proper has/delete method
in the rust code which was not implemented due to time constraints.

Also added a Typescript implementation of the vault keychain plugin
just so that we can have a full end to end test coverage of all the
variants of configurations where the plugin is local vs. remote.

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
@petermetz petermetz force-pushed the feat/out-of-process-plugin-instances-for-language-independence-170 branch from 12389d0 to 7d83c60 Compare January 14, 2021 04:00
@github-actions
Copy link

🎉 Great news! Looks like all the dependencies have been resolved:

💡 To add or remove a dependency please update this issue/PR description.

Brought to you by Dependent Issues (:robot: ). Happy coding!

@petermetz petermetz merged commit 6dcdb8a into hyperledger-cacti:main Jan 14, 2021
@petermetz petermetz deleted the feat/out-of-process-plugin-instances-for-language-independence-170 branch January 14, 2021 05:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API_Server dependencies Pull requests that update a dependency file Developer_Experience enhancement New feature or request Security Related to existing or potential security vulnerabilities Significant_Change Applying this label triggers the more stringent review of the maintainers and the 2+1 PR rule. SPIKE Exploratory work to better scope additional effort
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Out of Process Plugin Instances for Sand-boxing and Language Independence
3 participants