From 991c16c073594e00edcd7b014eaa8554b8e3ea8e Mon Sep 17 00:00:00 2001 From: SKernchen Date: Mon, 6 May 2024 08:54:18 +0200 Subject: [PATCH 1/5] Start of tutorial for writing a plugin --- .../tutorials/writing-a-plugin-for-hermes.md | 105 ++++++++++++++++++ 1 file changed, 105 insertions(+) create mode 100644 docs/source/tutorials/writing-a-plugin-for-hermes.md diff --git a/docs/source/tutorials/writing-a-plugin-for-hermes.md b/docs/source/tutorials/writing-a-plugin-for-hermes.md new file mode 100644 index 00000000..8c20d7dd --- /dev/null +++ b/docs/source/tutorials/writing-a-plugin-for-hermes.md @@ -0,0 +1,105 @@ + + + + +# Write a plugin for HERMES + + +This tutorial will present the basic steps for writing an additional harvester. +At the moment only the harvest architecture is stable. +The full code and structure is available at [harvest-git](https://github.com/hermes-hmc/hermes-git). +This plugin extracts information from the local git history. +The harvest-git plugin will help to gather contributing and branch metadata. +```{note} +For this tutorial you should be familiar with HERMES. +If you never used HERMES before, you might want to check the tutorial: [Automated Publication with HERMES](https://docs.software-metadata.pub/en/latest/tutorials/automated-publication-with-ci.html). +``` + +## Plugin Architecture + +HERMES uses a plugin architecture. Therefore, users are invited to contribute own features. +The structure for every plugin follows the same schema. +There is a base class for every plugin. In this HermesPlugin class there is one abstract method __ call __ which needs to be overwritten. +Furthermore, the HermesCommand class provides all needs for writing a plugin used in a HERMES command. +So the HermesPlugins call method uses an Instance of the HermesCommand that triggered this plugin to run. +In our case this will be the HermesHarvestCommand which calls all harvest plugins. +The Plugin class also uses a derivative of HermesSettings to add parameters. +HermesSettings are the base class for command specific settings. +It uses pydantic settings to specify and validate the parameters. +The user can either set the parameters in the hermes.toml or overwrite them in the command line. +To overwrite the configuration, you use the -O operator with the dotted parameter name and the value. + +## Set Up Plugin +To write a new plugin, it is important to follow the given structure. +This means your plugins source code has a pydantic class with Settings and the plugin class which inherits from one base class. +For our specific case, we want to write a git harvest plugin. +Our class Structure should look like this: + + +```{code-block} python +from hermes.commands.harvest.base import HermesHarvestPlugin +from pydantic import BaseModel + + +class GitHarvestSettings(BaseModel): + from_branch: str = 'main' + + +class GitHarvestPlugin(HermesHarvestPlugin): + settings_class = GitHarvestSettings + + def __call__(self, command): + print("Hello World!") + + return {}, {} +``` + +The Code uses the HermesHarvestPlugin as base class and pydantics Basemodel for the Settings. In the GitHarvestSettings you +can see that one setting is made. The Parameter from_branch is specific for this plugin and can be reached through self.settings.harvest.git.git_branch as long as our plugin will be named git. +In the hermes.toml this would be achieved by [harvest.{plugin_name}]. +The GitHarvestSettings are assigned to the GitHarvestPlugin. In the plugin you need to overwrite the __ call __ method. +For now a simple Hello World will do. The method return two dictionaries. These will later depict the harvested data in codemeta (json-ld) and information for generating hermes metadata. +That is the basic structure for the plugins source code. + +To integrate this code, you have to register it as a plugin in the pyproject.toml. To learn more about the pyproject.toml check https://python-poetry.org/docs/pyproject/. +We will just look at the important places for this plugin. There are two ways to integrate this plugin. First we will show how to use the plugin environment as the running base with HERMES as a dependency. +Then we say how to integrate this plugin in HERMES itself. + +### Include HERMES as Dependency +```{code-block} toml +... +[tool.poetry.dependencies] +python = "^3.10" +hermes = "^0.8.0" +... +... +[tool.poetry.plugins."hermes.harvest"] +git = "hermes_git.harvest:GitHarvestPlugin" +... +``` +### Write Plugin to be included in HERMES +```{code-block} toml +... +[tool.poetry.dependencies] +... +pydantic-settings = "^2.1.0" +hermes-git = { git = "https://github.com/hermes-hmc/hermes-git.git", branch = "main" } +... +... +[tool.poetry.plugins."hermes.harvest"] +cff = "hermes.commands.harvest.cff:CffHarvestPlugin" +codemeta = "hermes.commands.harvest.codemeta:CodeMetaHarvestPlugin" +git = "hermes_git.harvest:GitHarvestPlugin" +... +``` + +```{admonition} Congratulations! +You can now write plugins for HERMES. +``` From 9e4baec5803dda82c3573fdd4b28854d27f4de66 Mon Sep 17 00:00:00 2001 From: SKernchen Date: Fri, 10 May 2024 05:16:48 +0200 Subject: [PATCH 2/5] First version of tutorial, needs correction --- .../tutorials/writing-a-plugin-for-hermes.md | 51 +++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/docs/source/tutorials/writing-a-plugin-for-hermes.md b/docs/source/tutorials/writing-a-plugin-for-hermes.md index 8c20d7dd..045e5d83 100644 --- a/docs/source/tutorials/writing-a-plugin-for-hermes.md +++ b/docs/source/tutorials/writing-a-plugin-for-hermes.md @@ -73,6 +73,11 @@ We will just look at the important places for this plugin. There are two ways to Then we say how to integrate this plugin in HERMES itself. ### Include HERMES as Dependency +This is probably the more common way, where you can see HERMES as a framework. +The idea is that your project is the main part. You create the pyproject.toml as usual. +In the dependencies block you need to include hermes. Then you just have to declare your plugin. +The HERMES software will look for plugins and install these. +In the code below you can see the parts of the pyproject.toml that are important. ```{code-block} toml ... [tool.poetry.dependencies] @@ -84,7 +89,18 @@ hermes = "^0.8.0" git = "hermes_git.harvest:GitHarvestPlugin" ... ``` +As you can see the plugin class from hermes_git is declared as git for hermes.harvest. +To use the plugin you have to adapt the harvest Settings in the hermes.toml. +We will discuss the exact step after showing the other pyproject.toml configuration. +```{note} +You have to run poetry install to add and install all entrypoints declared in the pyproject.toml. +``` + ### Write Plugin to be included in HERMES +This variant is used to contribute to the HERMES community or adapt the HERMES workflow for own purposes. +If you want to contribute, see the [Contribution Guidelines](https://docs.software-metadata.pub/en/latest/dev/contribute.html). +After cloning the HERMES workflow repository you can adapt the pyproject.toml. +In the code below you see the parts with the important lines. ```{code-block} toml ... [tool.poetry.dependencies] @@ -99,7 +115,42 @@ codemeta = "hermes.commands.harvest.codemeta:CodeMetaHarvestPlugin" git = "hermes_git.harvest:GitHarvestPlugin" ... ``` +In the dependencies you have to install your plugin. If your Plugin is pip installable than you can just give the name and the version. +If your plugin is in a buildable git repository, you can install it with the given expression. +Note that this differs with the accessibility and your wishes, check [Explicit Package Sources](https://python-poetry.org/docs/repositories/#explicit-package-sources). + +The second thing to adapt is to declare the access point for the plugin. +You can do that with `git = "hermes_git.harvest:GitHarvestPlugin"`. +This expression makes the GitHarvestPlugin from the hermes_git package, a hermes.harvest plugin named git. +So you need to configure this line with your plugin properties. + +Now you just need to add the plugin to the hermes.toml and reinstall the adapted poetry package. + +### Configure hermes.toml +To use the plugin, you have to set it in the hermes.toml. +The settings for the plugins are also set there. +For the harvest plugin the hermes.toml could look like this: +```{code-block} toml +[harvest] +sources = [ "cff", "git" ] # ordered priority (first one is most important) + +[harvest.cff] +enable_validation = false + +[harvest.git] +from_branch = "develop" +... +``` +In [harvest] you define that this plugin is used with less priority than the built-in cff plugin. +in [harvest.git] you set the configuration for the plugin. +In the Beginning of this tutorial we set the parameter `from_branch` in the git settings. Now we change the default from_branch to develop. +With this Configuration the plugin will be used. If you run hermes harvest, you should see the "Hello World" message. +Of course the hermes.toml is always changeable as you desire. ```{admonition} Congratulations! You can now write plugins for HERMES. ``` +To fill the plugin with code, you can check our [harvest_git](https://github.com/hermes-hmc/hermes-git) repository. +There is the code to check the local git history and extract contributors of the given branch. + +If you have any questions, wishes or requests, feel free to contact us. From 6289b945a22403b44ebc4a953a7bff702c3955aa Mon Sep 17 00:00:00 2001 From: Sophie <133236526+SKernchen@users.noreply.github.com> Date: Fri, 9 Aug 2024 09:54:18 +0200 Subject: [PATCH 3/5] Correct form and improve text Co-authored-by: Michael Meinel --- .../tutorials/writing-a-plugin-for-hermes.md | 66 ++++++++++--------- 1 file changed, 34 insertions(+), 32 deletions(-) diff --git a/docs/source/tutorials/writing-a-plugin-for-hermes.md b/docs/source/tutorials/writing-a-plugin-for-hermes.md index 045e5d83..e6781977 100644 --- a/docs/source/tutorials/writing-a-plugin-for-hermes.md +++ b/docs/source/tutorials/writing-a-plugin-for-hermes.md @@ -13,10 +13,10 @@ SPDX-FileContributor: Sophie Kernchen This tutorial will present the basic steps for writing an additional harvester. -At the moment only the harvest architecture is stable. -The full code and structure is available at [harvest-git](https://github.com/hermes-hmc/hermes-git). +At the moment only the architecture for harvester plugins is stable. +The full code and structure is available at [hermes-plugin-git](https://github.com/softwarepub/hermes-plugin-git). This plugin extracts information from the local git history. -The harvest-git plugin will help to gather contributing and branch metadata. +The hermes-plugin-git will help to gather contributing and branch metadata. ```{note} For this tutorial you should be familiar with HERMES. If you never used HERMES before, you might want to check the tutorial: [Automated Publication with HERMES](https://docs.software-metadata.pub/en/latest/tutorials/automated-publication-with-ci.html). @@ -26,15 +26,18 @@ If you never used HERMES before, you might want to check the tutorial: [Automate HERMES uses a plugin architecture. Therefore, users are invited to contribute own features. The structure for every plugin follows the same schema. -There is a base class for every plugin. In this HermesPlugin class there is one abstract method __ call __ which needs to be overwritten. -Furthermore, the HermesCommand class provides all needs for writing a plugin used in a HERMES command. -So the HermesPlugins call method uses an Instance of the HermesCommand that triggered this plugin to run. -In our case this will be the HermesHarvestCommand which calls all harvest plugins. -The Plugin class also uses a derivative of HermesSettings to add parameters. -HermesSettings are the base class for command specific settings. -It uses pydantic settings to specify and validate the parameters. -The user can either set the parameters in the hermes.toml or overwrite them in the command line. -To overwrite the configuration, you use the -O operator with the dotted parameter name and the value. +There is a top-level base class for every plugin. In this `HermesPlugin` class there is one abstract method `__ call __` which needs to be overwritten. +Furthermore, the `HermesCommand` class provides all needs for writing a plugin used in a HERMES command. +So the `HermesPlugin`s call method gets an instance of the `HermesCommand` that triggered this plugin to run. +In our case this will be the `HermesHarvestCommand` which calls all harvest plugins. +The plugin class also uses a derivative of `HermesSettings` to add parameters that can be adapted by the configuration file. +`HermesSettings` are the base class for command specific settings. +It uses [pydantic](https://docs.pydantic.dev/latest/) [settings](https://docs.pydantic.dev/latest/api/pydantic_settings/) to specify and validate the parameters. +The user can either set the parameters in the `hermes.toml` or overwrite them in the command line. +To overwrite a parameter from command line, use the `-O` command line option followed by the dotted parameter name and the value. +E.g., you can set your authentication token for InvenioRDM by adding the following options to your call to `hermes deposit`: +```shell +hermes deposit -O invenio_rdm.auth_token YourSecretAuthToken ## Set Up Plugin To write a new plugin, it is important to follow the given structure. @@ -61,23 +64,23 @@ class GitHarvestPlugin(HermesHarvestPlugin): return {}, {} ``` -The Code uses the HermesHarvestPlugin as base class and pydantics Basemodel for the Settings. In the GitHarvestSettings you -can see that one setting is made. The Parameter from_branch is specific for this plugin and can be reached through self.settings.harvest.git.git_branch as long as our plugin will be named git. -In the hermes.toml this would be achieved by [harvest.{plugin_name}]. -The GitHarvestSettings are assigned to the GitHarvestPlugin. In the plugin you need to overwrite the __ call __ method. +The code uses the `HermesHarvestPlugin` as base class and pydantics Basemodel for the settings. In the `GitHarvestSettings` you +can see that an additional parameter is defined. The Parameter `from_branch` is specific for this plugin and can be accessed inside the plugin using `self.settings.harvest.git.git_branch` as long as our plugin will be named git. +In the `hermes.toml` this would be achieved by [harvest.{plugin_name}]. +The `GitHarvestSettings` are associated with the `GitHarvestPlugin`. In the plugin you need to overwrite the `__ call __` method. For now a simple Hello World will do. The method return two dictionaries. These will later depict the harvested data in codemeta (json-ld) and information for generating hermes metadata. That is the basic structure for the plugins source code. -To integrate this code, you have to register it as a plugin in the pyproject.toml. To learn more about the pyproject.toml check https://python-poetry.org/docs/pyproject/. +To integrate this code, you have to register it as a plugin in the `pyproject.toml`. To learn more about the `pyproject.toml` check https://python-poetry.org/docs/pyproject/ or refer to [PEP621](https://peps.python.org/pep-0621/). We will just look at the important places for this plugin. There are two ways to integrate this plugin. First we will show how to use the plugin environment as the running base with HERMES as a dependency. Then we say how to integrate this plugin in HERMES itself. ### Include HERMES as Dependency This is probably the more common way, where you can see HERMES as a framework. -The idea is that your project is the main part. You create the pyproject.toml as usual. -In the dependencies block you need to include hermes. Then you just have to declare your plugin. -The HERMES software will look for plugins and install these. -In the code below you can see the parts of the pyproject.toml that are important. +The idea is that your project is the main part. You create the `pyproject.toml` as usual. +In the dependencies block you need to include `hermes`. Then you just have to declare your plugin. +The HERMES software will look for installed plugins and use them. +In the code below you can see the parts of the `pyproject.toml` that are important. ```{code-block} toml ... [tool.poetry.dependencies] @@ -89,9 +92,9 @@ hermes = "^0.8.0" git = "hermes_git.harvest:GitHarvestPlugin" ... ``` -As you can see the plugin class from hermes_git is declared as git for hermes.harvest. -To use the plugin you have to adapt the harvest Settings in the hermes.toml. -We will discuss the exact step after showing the other pyproject.toml configuration. +As you can see the plugin class from `hermes_git` is declared as `git` for the `hermes.harvest` entrypoint. +To use the plugin you have to adapt the harvest settings in the `hermes.toml`. +We will discuss the exact step after showing the other `pyproject.toml` configuration. ```{note} You have to run poetry install to add and install all entrypoints declared in the pyproject.toml. ``` @@ -127,9 +130,9 @@ So you need to configure this line with your plugin properties. Now you just need to add the plugin to the hermes.toml and reinstall the adapted poetry package. ### Configure hermes.toml -To use the plugin, you have to set it in the hermes.toml. +To use the plugin, you have to activate it in the `hermes.toml`. The settings for the plugins are also set there. -For the harvest plugin the hermes.toml could look like this: +For the harvest plugin the `hermes.toml` could look like this: ```{code-block} toml [harvest] sources = [ "cff", "git" ] # ordered priority (first one is most important) @@ -141,16 +144,15 @@ enable_validation = false from_branch = "develop" ... ``` -In [harvest] you define that this plugin is used with less priority than the built-in cff plugin. -in [harvest.git] you set the configuration for the plugin. -In the Beginning of this tutorial we set the parameter `from_branch` in the git settings. Now we change the default from_branch to develop. -With this Configuration the plugin will be used. If you run hermes harvest, you should see the "Hello World" message. -Of course the hermes.toml is always changeable as you desire. +In the `[harvest]` section you define that this plugin is used with less priority than the built-in `cff` plugin. +in the `[harvest.git]` section you set the configuration for the plugin. +In the beginning of this tutorial we set the parameter `from_branch` in the git settings. Now we change the default `from_branch` to `develop`. +With this configuration the plugin will be used. If you run `hermes harvest`, you should see the "Hello World" message. ```{admonition} Congratulations! You can now write plugins for HERMES. ``` -To fill the plugin with code, you can check our [harvest_git](https://github.com/hermes-hmc/hermes-git) repository. +To fill the plugin with code, you can check our [hermes-plugin-git](https://github.com/softwarepub/hermes-plugin-git) repository. There is the code to check the local git history and extract contributors of the given branch. If you have any questions, wishes or requests, feel free to contact us. From 62b9799dc8191b37d1043c962c02a8bf5ce09f17 Mon Sep 17 00:00:00 2001 From: Sophie <133236526+SKernchen@users.noreply.github.com> Date: Fri, 9 Aug 2024 09:56:18 +0200 Subject: [PATCH 4/5] Correct duplicated word --- docs/source/tutorials/writing-a-plugin-for-hermes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/tutorials/writing-a-plugin-for-hermes.md b/docs/source/tutorials/writing-a-plugin-for-hermes.md index e6781977..6eadb5f3 100644 --- a/docs/source/tutorials/writing-a-plugin-for-hermes.md +++ b/docs/source/tutorials/writing-a-plugin-for-hermes.md @@ -68,7 +68,7 @@ The code uses the `HermesHarvestPlugin` as base class and pydantics Basemodel fo can see that an additional parameter is defined. The Parameter `from_branch` is specific for this plugin and can be accessed inside the plugin using `self.settings.harvest.git.git_branch` as long as our plugin will be named git. In the `hermes.toml` this would be achieved by [harvest.{plugin_name}]. The `GitHarvestSettings` are associated with the `GitHarvestPlugin`. In the plugin you need to overwrite the `__ call __` method. -For now a simple Hello World will do. The method return two dictionaries. These will later depict the harvested data in codemeta (json-ld) and information for generating hermes metadata. +For now a simple Hello World will do. The method returns two dictionaries. These will contain the harvested data in CodeMeta (JSON-LD) and additional information, e.g., to provide provenance information. That is the basic structure for the plugins source code. To integrate this code, you have to register it as a plugin in the `pyproject.toml`. To learn more about the `pyproject.toml` check https://python-poetry.org/docs/pyproject/ or refer to [PEP621](https://peps.python.org/pep-0621/). From 46a7b9f2475865616cb07f1e88724bc0fe36ed28 Mon Sep 17 00:00:00 2001 From: Sophie <133236526+SKernchen@users.noreply.github.com> Date: Fri, 9 Aug 2024 10:42:09 +0200 Subject: [PATCH 5/5] Add name convention to hermes_plugin_git everywhere --- docs/source/tutorials/writing-a-plugin-for-hermes.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/source/tutorials/writing-a-plugin-for-hermes.md b/docs/source/tutorials/writing-a-plugin-for-hermes.md index 6eadb5f3..91f0c4bd 100644 --- a/docs/source/tutorials/writing-a-plugin-for-hermes.md +++ b/docs/source/tutorials/writing-a-plugin-for-hermes.md @@ -89,10 +89,10 @@ hermes = "^0.8.0" ... ... [tool.poetry.plugins."hermes.harvest"] -git = "hermes_git.harvest:GitHarvestPlugin" +git = "hermes_plugin_git.harvest:GitHarvestPlugin" ... ``` -As you can see the plugin class from `hermes_git` is declared as `git` for the `hermes.harvest` entrypoint. +As you can see the plugin class from `hermes_plugin_git` is declared as `git` for the `hermes.harvest` entrypoint. To use the plugin you have to adapt the harvest settings in the `hermes.toml`. We will discuss the exact step after showing the other `pyproject.toml` configuration. ```{note} @@ -109,13 +109,13 @@ In the code below you see the parts with the important lines. [tool.poetry.dependencies] ... pydantic-settings = "^2.1.0" -hermes-git = { git = "https://github.com/hermes-hmc/hermes-git.git", branch = "main" } +hermes-plugin-git = { git = "https://github.com/softwarepub/hermes-plugin-git.git", branch = "main" } ... ... [tool.poetry.plugins."hermes.harvest"] cff = "hermes.commands.harvest.cff:CffHarvestPlugin" codemeta = "hermes.commands.harvest.codemeta:CodeMetaHarvestPlugin" -git = "hermes_git.harvest:GitHarvestPlugin" +git = "hermes_plugin_git.harvest:GitHarvestPlugin" ... ``` In the dependencies you have to install your plugin. If your Plugin is pip installable than you can just give the name and the version. @@ -123,8 +123,8 @@ If your plugin is in a buildable git repository, you can install it with the giv Note that this differs with the accessibility and your wishes, check [Explicit Package Sources](https://python-poetry.org/docs/repositories/#explicit-package-sources). The second thing to adapt is to declare the access point for the plugin. -You can do that with `git = "hermes_git.harvest:GitHarvestPlugin"`. -This expression makes the GitHarvestPlugin from the hermes_git package, a hermes.harvest plugin named git. +You can do that with `git = "hermes_plugin_git.harvest:GitHarvestPlugin"`. +This expression makes the GitHarvestPlugin from the hermes_plugin_git package, a hermes.harvest plugin named git. So you need to configure this line with your plugin properties. Now you just need to add the plugin to the hermes.toml and reinstall the adapted poetry package.