diff --git a/.vale/styles/Infrahub/spelling.yml b/.vale/styles/Infrahub/spelling.yml
index 252b2269f8..5fbbbcddb9 100644
--- a/.vale/styles/Infrahub/spelling.yml
+++ b/.vale/styles/Infrahub/spelling.yml
@@ -4,5 +4,6 @@ message: "Did you really mean '%s'?"
level: error
filters:
- '[pP]y.*\b'
- - '\bimport_.*\b' # New filter to ignore variables starting with 'import_'
+ - '\bimport_.*\b' # New filter to ignore variables starting with 'import_'
+ - '\w+__value' # New filter to skip Infrahub filters in documentation (name__value)
ignore: spelling-exceptions.txt
diff --git a/docs/docs/guides/generator.mdx b/docs/docs/guides/generator.mdx
index 6e8b6ddae0..cc8c7342d8 100644
--- a/docs/docs/guides/generator.mdx
+++ b/docs/docs/guides/generator.mdx
@@ -10,12 +10,12 @@ Within Infrahub a generator is defined in an [external repository](/topics/repos
The goal of this guide is to develop a Generator and add it to Infrahub, we will achieve this by following these steps.
-1. Identify the relevant data you want to extract from the database using a [GraphQL query](/topics/graphql), that can take an input parameter to filter the data
-2. Write a Python script that uses the GraphQL query to read information from the system and generates new data based on the response
-3. Create an entry for the generator within an .infrahub.yml file.
-4. Create a Git repository
+1. Create a Git repository
+2. Create an entry for the generator within an .infrahub.yml file.
+3. Identify the relevant data you want to extract from the database using a [GraphQL query](/topics/graphql), that can take an input parameter to filter the data
+4. Write a Python script that uses the GraphQL query to read information from the system and generates new data based on the response
5. Test the generator with infrahubctl
-6. Add the repository to Infrahub as an external repository
+6. Push changes to GitHub and add the repository to Infrahub as an external repository
7. Validate that the generator works by triggering it through a proposed change
## Preparations
@@ -24,9 +24,11 @@ What your generator will look like will depend on what your schema looks like an
As the default Infrahub schema doesn't have a lot of object types to use as a test, we will illustrate how this could work by adding two nodes to the schema.
-Load the following schema using the [infrahubctl schema](/infrahubctl/infrahubctl-schema) command.
+### Create and load example schema
+
+Create a **widgets.yml** file with the following:
-```yaml
+```yaml title="widgets.yml"
# yaml-language-server: $schema=https://schema.infrahub.app/infrahub/schema/latest.json
---
version: '1.0'
@@ -56,90 +58,99 @@ nodes:
unique: true
```
-Perform these steps in the frontend.
-
-1. Create two new widget objects
- - One with the name `widget1` and count 1
- - One with the name `widget2` and count 2
-2. Create a Standard group called "widgets"
-3. Add both of the created objects to the new group
-
-## 1. Identify the relevant data
-
-Here we define a GraphQL query that we will use to gather information.
+Load the following schema using the [infrahubctl schema](/infrahubctl/infrahubctl-schema) command.
-```graphql
-query Widgets($name: String!) {
- TestWidget(name__value: $name) {
- edges {
- node {
- name {
- value
- }
- count {
- value
- }
- }
- }
- }
-}
+```shell
+infrahubctl schema load widgets.yml
```
-Create a local directory on your computer where we will store the generator files.
-
```shell
-mkdir example_generator
+schema 'widgets.yml' loaded successfully
+1 schema processed in 8.453 seconds.
```
-Within that directory store the above GraphQL query as widget_query.gql.
-
-## 2. Create a Python Generator
+### Add two new nodes
-The Generator class needs to implement a `generate` function that receives a `data` parameter that contains the response from the GraphQL query.
+Perform these steps in the [Infrahub UI](http://localhost:8000).
-The goal of this generator will be to create a number of resources that depends on the set count of the widgets.
+1. Create two new widget objects
+ - One with the name `widget1` and count 1
+ - One with the name `widget2` and count 2
+2. Create a [Standard group](http://localhost:8000/objects/CoreGroup) called "**widgets**
+3. Add both of the created objects to the new group
-```python
-from infrahub_sdk.generator import InfrahubGenerator
+:::info
+Any widget object must be added to or be part of the **widgets** group to be tied to the generator.
+:::
+## 1. Create GitHub repository
-class WidgetGenerator(InfrahubGenerator):
- async def generate(self, data: dict) -> None:
- widget = data["TestWidget"]["edges"][0]["node"]
- widget_name: str = widget["name"]["value"]
- widget_count: str = widget["count"]["value"]
+Follow GitHub's instructions on [creating a new repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-new-repository).
- for count in range(1, widget_count + 1):
+:::info Initialize with a README
+For ease, it is suggested to **Initialize this repository with: Add a README file**.
- payload = {
- "name": f"{widget_name.lower()}-{count}",
- }
- obj = await self.client.create(kind="TestResource", data=payload)
- await obj.save(allow_upsert=True)
-```
+This allows you to clone the repository down without any extra commands.
+:::
-Store this class within a new file called widget_generator.py.
+Once the repository is created, [clone the repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository) down and change directory into the cloned folder.
-## 3. Create an .infrahub.yml file
+## 2. Create an .infrahub.yml file
The [.infrahub.yml](/topics/infrahub-yml) file allows you to tie the different [components of a generator](/topics/generator) together into a working unit.
-```yaml
-# yaml-language-server: $schema=https://schema.infrahub.app/python-sdk/repository-config/latest.json
----
-generator_definitions:
- - name: widget_generator
- file_path: "widget_generator.py"
- targets: widgets
- query: widget_query
- class_name: WidgetGenerator
- parameters:
- name: "name__value"
-
-queries:
- - name: widget_query
- file_path: "widget_query.gql"
-```
+:::info Convert query to Infrahub SDK objects
+We provide `convert_query_response` option to be toggled to be able to access objects from the GraphQL query as Infrahub SDK objects rather than the raw dictionary response.
+
+This allows you to manage the returned data with helper methods on the SDK objects such as `save`, `fetch`, etc. on the returned data rather than having to build a payload to send back to Infrahub to manage the objects.
+
+Read more on the [Infrahub Python SDK](/python-sdk).
+:::
+
+
+
+
+
+
+ ```yaml title=".infrahub.yml"
+ # yaml-language-server: $schema=https://schema.infrahub.app/python-sdk/repository-config/latest.json
+ ---
+ generator_definitions:
+ - name: widget_generator
+ file_path: "example_generator/widget_generator.py"
+ targets: widgets
+ query: widget_query
+ class_name: WidgetGenerator
+ parameters:
+ name: "name__value"
+
+ queries:
+ - name: widget_query
+ file_path: "example_generator/widget_query.gql"
+ ```
+
+
+
+
+ ```yaml title=".infrahub.yml"
+ # yaml-language-server: $schema=https://schema.infrahub.app/python-sdk/repository-config/latest.json
+ ---
+ generator_definitions:
+ - name: widget_generator
+ file_path: "example_generator/widget_generator.py"
+ targets: widgets
+ query: widget_query
+ convert_query_response: true
+ class_name: WidgetGenerator
+ parameters:
+ name: "name__value"
+
+ queries:
+ - name: widget_query
+ file_path: "example_generator/widget_query.gql"
+ ```
+
+
@@ -148,8 +159,9 @@ queries:
- **file_path**: the relative file path to the file containing the generator as seen from within a Git repository
- **targets**: the name of a group of which the members will be a target for this generator
- **query**: the name of the GraphQL query used within this generator
+ - **convert_query_response**: convert the result of the GraphQL query to SDK InfrahubNode objects
+ - **class_name**: the name of the Python class in the generator file
- **parameters**: the parameter to pass to the generator GraphQL query, in this case this we will pass the name of the object (widget) as the name parameter
- - **query**: the name of the GraphQL query used within this generator
Here the `name` refers to the query's name and `file_path` should point to the GraphQL file within the repository.
@@ -158,32 +170,126 @@ queries:
See [this topic](/topics/infrahub-yml) for a full explanation of everything that can be defined in the `.infrahub.yml` file.
-## 4. Create a Git repository
+## 3. Identify the relevant data
-Within the `example_generator` folder you should now have 3 files:
+Here we define a GraphQL query that we will use to gather the information that will be passed into the generator.
-- `widget_query.gql`: Contains the GraphQL query
-- `generator.py`: Contains the Python code for the generator
-- `.infrahub.yml`: Contains the definition for the generator
+
+
+ ```graphql title="example_generator/widget_query.gql"
+ query Widgets($name: String!) {
+ TestWidget(name__value: $name) {
+ edges {
+ node {
+ name {
+ value
+ }
+ count {
+ value
+ }
+ }
+ }
+ }
+ }
+ ```
+
+
+ Here we must provide `__typename` and `id` within the query so the SDK can convert the query to the correct type and properly store within the SDK.
+
+ ```graphql title="example_generator/widget_query.gql"
+ query Widgets($name: String!) {
+ TestWidget(name__value: $name) {
+ edges {
+ node {
+ __typename
+ id
+ name {
+ value
+ }
+ count {
+ value
+ }
+ }
+ }
+ }
+ }
+ ```
-Before we can test our generator we must add the files to a local Git repository.
+
+
+
+Create a local directory on your computer where we will store the generator files.
```shell
-git init --initial-branch=main
-git add .
-git commit -m "First commit"
+mkdir example_generator
```
+## 4. Create a Python Generator
+
+The Generator class needs to implement a `generate` function that receives a `data` parameter that contains the response from the GraphQL query.
+
+The goal of this generator will be to create a number of resources that depends on the set count of the widgets.
+
+
+
+
+ ```python title="example_generator/widget_generator.py"
+ from infrahub_sdk.generator import InfrahubGenerator
+
+ class WidgetGenerator(InfrahubGenerator):
+ async def generate(self, data: dict) -> None:
+ # We can now access the nodes as typical SDK objects
+ widget = data["TestWidget"]["edges"][0]["node"]
+ widget_name: str = widget["name"]["value"]
+ widget_count: str = widget["count"]["value"]
+ for count in range(1, widget_count + 1):
+ payload = {
+ "name": f"{widget_name.lower()}-{count}",
+ }
+ obj = await self.client.create(kind="TestResource", data=payload)
+ await obj.save(allow_upsert=True)
+ ```
+
+
+
+ ```python title="example_generator/widget_generator.py"
+ from infrahub_sdk.generator import InfrahubGenerator
+
+ class WidgetGenerator(InfrahubGenerator):
+ async def generate(self, data: dict) -> None:
+ # We can now access the nodes as typical SDK objects
+ widget = self.nodes[0] # or self.store.get(data["TestWidget"]["edges"][0]["node"]["id"])
+ widget_name: str = widget.name.value
+ widget_count: str = widget.count.value
+
+ for count in range(1, widget_count + 1):
+
+ payload = {
+ "name": f"{widget_name.lower()}-{count}",
+ }
+ obj = await self.client.create(kind="TestResource", data=payload)
+ await obj.save(allow_upsert=True)
+ ```
+
+
+
+
+Store this class within a new file called **widget_generator.py** within the **example_generator** directory.
+
## 5. Test the generator using infrahubctl
-Using infrahubctl you can first verify that the `.infrahub.yml` file is formatted correctly by listing available generators.
+Using infrahubctl you can first verify that the **.infrahub.yml** file is formatted correctly by listing available generators.
-```shell title="❯ infrahubctl generator --list"
+```shell
+infrahubctl generator --list
+```
+
+```shell
Generators defined in repository: 1
widget_generator (widget_generator.py::Generator) Target: widgets
```
-:::note
+:::warning
When running a generator with `infrahubctl` the [SDK tracking](/python-sdk/topics/tracking) feature isn't used. The reason for this is that internally Infrahub uses the ID of the generator_definition to control the tracking, this isn't available from the outside. For this reason it is recommended to create test branches when developing generators and validating the results.
@@ -200,6 +306,46 @@ infrahubctl generator widget_generator --branch=test-branch1 name=widget1
infrahubctl generator widget_generator --branch=test-branch1 name=widget2
```
-Now you should see the tree TestResource objects within `test-branch1` one for the first widget and two for the second one.
+Now you should see the tree [TestResource](http://localhost:8000/objects/TestResource?branch=test-branch1) objects within `test-branch1` one for the first widget and two for the second one.
+
+
+
+Merge the changes into the **main** branch using [infrahubctl branch](/infrahubctl/infrahubctl-branch) command.
+
+```shell
+infrahubctl branch merge 'test-branch1'
+```
+
+## 6. Commit and add repository to Infrahub
+
+The root directory should include the **.infrahub.yml** file and **example_generator** folder.
+
+- `.infrahub.yml`: Contains the definition for the generator
+
+Within the **example_generator** folder you should now have 2 files:
+
+- `widget_query.gql`: Contains the GraphQL query
+- `widget_generator.py`: Contains the Python code for the generator
+
+Before we can run our generator in an Proposed Change (PC) pipeline we must push the files up to the remote Git repository we created in **Step 1**.
+
+```shell
+git add .
+git commit -m "Add generator"
+git push
+```
+
+Now that we have the latest changes, [add your repository to Infrahub](/guides/repository) and then the generators will be executed as part of the proposed change CI pipeline.
+
+## 7. Generator in the CI pipeline
-With this step completed you can [add your repository to Infrahub](/guides/repository) and then the generators will be executed as part of the proposed change pipeline.
+1. Navigate to [widgets](http://localhost:8000/objects/TestWidget) in the Infrahub UI
+2. Create a new branch named **widget-3**
+3. Create a new widget with the name of **widget3** and count of **3**
+4. Add **widget3** to the **widgets** group
+5. Create a [Proposed Change](http://localhost:8000/proposed-changes?branch=widget-3)
+6. Navigate to the **Checks** tab
+7. Once the **generator** CI check has completed, navigate to the **Data** tab
+
+8. Click the **Refresh diff** button to see the **widget3** resources created by the generator
+
diff --git a/docs/docs/media/guides/generator_pc_1.png b/docs/docs/media/guides/generator_pc_1.png
new file mode 100644
index 0000000000..4d80597c57
Binary files /dev/null and b/docs/docs/media/guides/generator_pc_1.png differ
diff --git a/docs/docs/media/guides/generator_pc_2.png b/docs/docs/media/guides/generator_pc_2.png
new file mode 100644
index 0000000000..dbc0c62320
Binary files /dev/null and b/docs/docs/media/guides/generator_pc_2.png differ
diff --git a/docs/docs/media/guides/generator_pc_3.png b/docs/docs/media/guides/generator_pc_3.png
new file mode 100644
index 0000000000..539da49cb2
Binary files /dev/null and b/docs/docs/media/guides/generator_pc_3.png differ