diff --git a/docs/cli-github-actions.md b/docs/How-To/cli-github-actions.md similarity index 94% rename from docs/cli-github-actions.md rename to docs/How-To/cli-github-actions.md index 21e505c..bd58f99 100644 --- a/docs/cli-github-actions.md +++ b/docs/How-To/cli-github-actions.md @@ -21,7 +21,7 @@ The basic procedure to use the Quix CLI in a GitHub Action is: 2. Authenticate the CLI with Quix Cloud using the command `quix login `, where `` is the [personal access token](../develop/authentication/personal-access-token.md) for the environment. 3. Run your CLI command or commands. -In this case you'll run the command to synchronize the repository with Quix using the command `quix env sync `. You can obtain the Workspace ID from your environment settings. +In this case you'll run the command to synchronize the repository with Quix using the command `quix cloud env sync `. You can obtain the Workspace ID from your environment settings. !!! tip @@ -64,7 +64,7 @@ jobs: curl -fsSL https://github.com/quixio/quix-cli/raw/main/install.sh | bash echo Logging into Quix Cloud quix login ${{ secrets.QUIX_PAT }} - quix env sync ${{ secrets.WORKSPACE_ID }} + quix cloud env sync ${{ secrets.WORKSPACE_ID }} ``` In this case the Action is run for `main` branch, but you could modify this or add other branches as required. diff --git a/docs/Reference/applications/convert.gen.md b/docs/Reference/apps/convert.gen.md similarity index 85% rename from docs/Reference/applications/convert.gen.md rename to docs/Reference/apps/convert.gen.md index e1ddc20..15f2987 100644 --- a/docs/Reference/applications/convert.gen.md +++ b/docs/Reference/apps/convert.gen.md @@ -1,4 +1,4 @@ -# applications convert +# apps convert ## Overview @@ -9,7 +9,7 @@ Converts applications from and to library item format **Usage:** ``` -quix applications convert [options] +quix apps convert [options] ``` **Arguments:** diff --git a/docs/Reference/applications/convert.md b/docs/Reference/apps/convert.md similarity index 100% rename from docs/Reference/applications/convert.md rename to docs/Reference/apps/convert.md diff --git a/docs/Reference/applications/create.gen.md b/docs/Reference/apps/create.gen.md similarity index 90% rename from docs/Reference/applications/create.gen.md rename to docs/Reference/apps/create.gen.md index 0018606..f1d229d 100644 --- a/docs/Reference/applications/create.gen.md +++ b/docs/Reference/apps/create.gen.md @@ -1,4 +1,4 @@ -# applications create +# apps create ## Overview @@ -9,7 +9,7 @@ Creates a new application locally based on a code sample of the library **Usage:** ``` -quix applications create [] [options] +quix apps create [] [options] ``` **Arguments:** diff --git a/docs/Reference/applications/create.md b/docs/Reference/apps/create.md similarity index 84% rename from docs/Reference/applications/create.md rename to docs/Reference/apps/create.md index 01a94b1..549b001 100644 --- a/docs/Reference/applications/create.md +++ b/docs/Reference/apps/create.md @@ -10,6 +10,7 @@ The command generates the application with a default structure relative to the c - The main Python file, usually `main.py` - `app.yaml` for application configuration +- `.env` for environment variable management, allowing you to securely configure sensitive information like API keys and connection strings without hardcoding them into your scripts. - `requirements.txt` for Python dependencies - `dockerfile` for containerization - `README.md` for documentation @@ -59,6 +60,24 @@ Leave the name empty so the default name is used: ``` ✓ Application name: Starter Source +``` + +You can then select the values of the library item variables, such as topics. In this case, a list of existing topics is presented, where you can pick an existing topic or create a new one: + +``` +? [Output Topic] output: + +> csv-data + grouped-transactions + transactions-output + + New topic +``` + +After selecting the output topic: + +``` +✓ [Output Topic] output: csv-data ✓ Application 'Starter Source' created successfully in '$\Starter Source' ``` + Then your new `Starter Source` application is created. \ No newline at end of file diff --git a/docs/Reference/apps/edit.gen.md b/docs/Reference/apps/edit.gen.md new file mode 100644 index 0000000..57c8913 --- /dev/null +++ b/docs/Reference/apps/edit.gen.md @@ -0,0 +1,18 @@ +# apps edit + +## Overview + +Edit the application values + +**Aliases:** `edit` + +**Usage:** + +``` +quix apps edit [options] +``` + +**Options:** + +- `-d, --directory ` : Base directory (defaults to current directory) + diff --git a/docs/Reference/apps/edit.md b/docs/Reference/apps/edit.md new file mode 100644 index 0000000..7fc98ca --- /dev/null +++ b/docs/Reference/apps/edit.md @@ -0,0 +1 @@ +{% include-markdown './edit.gen.md' %} \ No newline at end of file diff --git a/docs/Reference/applications/index.gen.md b/docs/Reference/apps/index.gen.md similarity index 63% rename from docs/Reference/applications/index.gen.md rename to docs/Reference/apps/index.gen.md index cc84995..2edecd3 100644 --- a/docs/Reference/applications/index.gen.md +++ b/docs/Reference/apps/index.gen.md @@ -1,21 +1,23 @@ -# applications +# apps ## Overview Manage your local applications and their variables -**Aliases:** `applications`, `application`, `apps`, `app` +**Aliases:** `apps`, `applications`, `application`, `app` **Usage:** ``` -quix applications [command] [options] +quix apps [command] [options] ``` **Commands:** - [convert](convert.md) : Converts applications from and to library item format - [create](create.md) : Creates a new application locally based on a code sample of the library +- [edit](edit.md) : Edit the application values - [list](list.md) : Lists all the local applications of the project +- [update](update.md) : Updates the application's '.env' file with the variables from 'quix.yaml' - [variables](variables/index.md) : Manage your local application's variables diff --git a/docs/Reference/applications/index.md b/docs/Reference/apps/index.md similarity index 100% rename from docs/Reference/applications/index.md rename to docs/Reference/apps/index.md diff --git a/docs/Reference/applications/list.gen.md b/docs/Reference/apps/list.gen.md similarity index 83% rename from docs/Reference/applications/list.gen.md rename to docs/Reference/apps/list.gen.md index be553fc..6f30556 100644 --- a/docs/Reference/applications/list.gen.md +++ b/docs/Reference/apps/list.gen.md @@ -1,4 +1,4 @@ -# applications list +# apps list ## Overview @@ -9,7 +9,7 @@ Lists all the local applications of the project **Usage:** ``` -quix applications list [options] +quix apps list [options] ``` **Options:** diff --git a/docs/Reference/applications/list.md b/docs/Reference/apps/list.md similarity index 100% rename from docs/Reference/applications/list.md rename to docs/Reference/apps/list.md diff --git a/docs/Reference/apps/update.gen.md b/docs/Reference/apps/update.gen.md new file mode 100644 index 0000000..9b81d8e --- /dev/null +++ b/docs/Reference/apps/update.gen.md @@ -0,0 +1,23 @@ +# apps update + +## Overview + +Updates the application's '.env' file with the variables from 'quix.yaml' + +**Aliases:** `update`, `upd` + +**Usage:** + +``` +quix apps update [options] +``` + +**Options:** + +- `-a, --all` : Forces the update of all applications regardless of the current directory +- `--workspace-id ` : The environment to use if your SDK broker configuration is set to Quix Cloud +- `--env-file-name ` : Set the name of the environment file [default: .env] +- `--directory ` : Base directory (defaults to current directory) +- `--reset` : Removes all the values of existing env variables and adds them from the 'quix.yaml' file [default: False] +- `--only-new` : Add only the missing variables in the '.env' file [default: False] + diff --git a/docs/Reference/apps/update.md b/docs/Reference/apps/update.md new file mode 100644 index 0000000..f93a35f --- /dev/null +++ b/docs/Reference/apps/update.md @@ -0,0 +1 @@ +{% include-markdown './update.gen.md' %} \ No newline at end of file diff --git a/docs/Reference/applications/variables/create.gen.md b/docs/Reference/apps/variables/create.gen.md similarity index 86% rename from docs/Reference/applications/variables/create.gen.md rename to docs/Reference/apps/variables/create.gen.md index 7d816fb..890112d 100644 --- a/docs/Reference/applications/variables/create.gen.md +++ b/docs/Reference/apps/variables/create.gen.md @@ -1,4 +1,4 @@ -# applications variables create +# apps variables create ## Overview @@ -9,7 +9,7 @@ Creates a variable of the current application **Usage:** ``` -quix applications variables create [options] +quix apps variables create [options] ``` **Options:** diff --git a/docs/Reference/applications/variables/create.md b/docs/Reference/apps/variables/create.md similarity index 100% rename from docs/Reference/applications/variables/create.md rename to docs/Reference/apps/variables/create.md diff --git a/docs/Reference/applications/variables/delete.gen.md b/docs/Reference/apps/variables/delete.gen.md similarity index 75% rename from docs/Reference/applications/variables/delete.gen.md rename to docs/Reference/apps/variables/delete.gen.md index 18bbde7..38af962 100644 --- a/docs/Reference/applications/variables/delete.gen.md +++ b/docs/Reference/apps/variables/delete.gen.md @@ -1,4 +1,4 @@ -# applications variables delete +# apps variables delete ## Overview @@ -9,7 +9,7 @@ Deletes a variable of the current application **Usage:** ``` -quix applications variables delete [] [options] +quix apps variables delete [] [options] ``` **Arguments:** diff --git a/docs/Reference/applications/variables/delete.md b/docs/Reference/apps/variables/delete.md similarity index 100% rename from docs/Reference/applications/variables/delete.md rename to docs/Reference/apps/variables/delete.md diff --git a/docs/Reference/applications/variables/edit.gen.md b/docs/Reference/apps/variables/edit.gen.md similarity index 87% rename from docs/Reference/applications/variables/edit.gen.md rename to docs/Reference/apps/variables/edit.gen.md index 3eac0cd..edae569 100644 --- a/docs/Reference/applications/variables/edit.gen.md +++ b/docs/Reference/apps/variables/edit.gen.md @@ -1,4 +1,4 @@ -# applications variables edit +# apps variables edit ## Overview @@ -9,7 +9,7 @@ Edits a new variable of the current application **Usage:** ``` -quix applications variables edit [options] +quix apps variables edit [options] ``` **Options:** diff --git a/docs/Reference/applications/variables/edit.md b/docs/Reference/apps/variables/edit.md similarity index 100% rename from docs/Reference/applications/variables/edit.md rename to docs/Reference/apps/variables/edit.md diff --git a/docs/Reference/applications/variables/export.gen.md b/docs/Reference/apps/variables/export.gen.md similarity index 87% rename from docs/Reference/applications/variables/export.gen.md rename to docs/Reference/apps/variables/export.gen.md index a1691d0..e4a64df 100644 --- a/docs/Reference/applications/variables/export.gen.md +++ b/docs/Reference/apps/variables/export.gen.md @@ -1,4 +1,4 @@ -# applications variables export +# apps variables export ## Overview @@ -9,7 +9,7 @@ Exports the variables from app.yaml to an .env file **Usage:** ``` -quix applications variables export [options] +quix apps variables export [options] ``` **Options:** diff --git a/docs/Reference/applications/variables/export.md b/docs/Reference/apps/variables/export.md similarity index 100% rename from docs/Reference/applications/variables/export.md rename to docs/Reference/apps/variables/export.md diff --git a/docs/Reference/applications/variables/import.gen.md b/docs/Reference/apps/variables/import.gen.md similarity index 86% rename from docs/Reference/applications/variables/import.gen.md rename to docs/Reference/apps/variables/import.gen.md index bf41c53..9416eb1 100644 --- a/docs/Reference/applications/variables/import.gen.md +++ b/docs/Reference/apps/variables/import.gen.md @@ -1,4 +1,4 @@ -# applications variables import +# apps variables import ## Overview @@ -9,7 +9,7 @@ Imports the .env file values to the app.yaml. The values from the .env file will **Usage:** ``` -quix applications variables import [options] +quix apps variables import [options] ``` **Options:** diff --git a/docs/Reference/applications/variables/import.md b/docs/Reference/apps/variables/import.md similarity index 100% rename from docs/Reference/applications/variables/import.md rename to docs/Reference/apps/variables/import.md diff --git a/docs/Reference/applications/variables/index.gen.md b/docs/Reference/apps/variables/index.gen.md similarity index 89% rename from docs/Reference/applications/variables/index.gen.md rename to docs/Reference/apps/variables/index.gen.md index 6ca5e46..effaefa 100644 --- a/docs/Reference/applications/variables/index.gen.md +++ b/docs/Reference/apps/variables/index.gen.md @@ -1,4 +1,4 @@ -# applications variables +# apps variables ## Overview @@ -9,7 +9,7 @@ Manage your local application's variables **Usage:** ``` -quix applications variables [command] [options] +quix apps variables [command] [options] ``` **Commands:** diff --git a/docs/Reference/applications/variables/index.md b/docs/Reference/apps/variables/index.md similarity index 100% rename from docs/Reference/applications/variables/index.md rename to docs/Reference/apps/variables/index.md diff --git a/docs/Reference/applications/variables/list.gen.md b/docs/Reference/apps/variables/list.gen.md similarity index 73% rename from docs/Reference/applications/variables/list.gen.md rename to docs/Reference/apps/variables/list.gen.md index 2ee3e6e..1efd5f1 100644 --- a/docs/Reference/applications/variables/list.gen.md +++ b/docs/Reference/apps/variables/list.gen.md @@ -1,4 +1,4 @@ -# applications variables list +# apps variables list ## Overview @@ -9,7 +9,7 @@ Lists the variables of the current application **Usage:** ``` -quix applications variables list [options] +quix apps variables list [options] ``` **Options:** diff --git a/docs/Reference/applications/variables/list.md b/docs/Reference/apps/variables/list.md similarity index 100% rename from docs/Reference/applications/variables/list.md rename to docs/Reference/apps/variables/list.md diff --git a/docs/Reference/pipeline/broker/down.gen.md b/docs/Reference/broker/down.gen.md similarity index 75% rename from docs/Reference/pipeline/broker/down.gen.md rename to docs/Reference/broker/down.gen.md index 1417953..9002da9 100644 --- a/docs/Reference/pipeline/broker/down.gen.md +++ b/docs/Reference/broker/down.gen.md @@ -1,15 +1,15 @@ -# pipeline broker down +# broker down ## Overview Stop the broker instance of the local pipeline and all the services depending on it -**Aliases:** `down`, `stop` +**Aliases:** `down` **Usage:** ``` -quix pipeline broker down [options] +quix broker down [options] ``` **Options:** diff --git a/docs/Reference/pipeline/broker/down.md b/docs/Reference/broker/down.md similarity index 84% rename from docs/Reference/pipeline/broker/down.md rename to docs/Reference/broker/down.md index 8f4d96a..6f50e43 100644 --- a/docs/Reference/pipeline/broker/down.md +++ b/docs/Reference/broker/down.md @@ -4,7 +4,7 @@ ## How It Works -The `quix pipeline broker down` command stops your local broker instance along with all the services that depend on it. When executed, it stops and removes the Docker containers defined in the `compose.yaml` file, effectively shutting down the local broker environment. +The `quix broker down` command stops your local broker instance along with all the services that depend on it. When executed, it stops and removes the Docker containers defined in the `compose.local.yaml` file, effectively shutting down the local broker environment. If the force option is used, the command bypasses all safety checks, allowing for an immediate shutdown without any prompts or checks. @@ -17,7 +17,7 @@ This ensures that your local environment is cleanly stopped and all resources ar To stop your local broker, use the following command: ```bash -$ quix pipeline broker down +$ quix broker down ``` This command stops and removes the Docker containers: diff --git a/docs/Reference/pipeline/broker/index.gen.md b/docs/Reference/broker/index.gen.md similarity index 74% rename from docs/Reference/pipeline/broker/index.gen.md rename to docs/Reference/broker/index.gen.md index 982e7b1..c7841b6 100644 --- a/docs/Reference/pipeline/broker/index.gen.md +++ b/docs/Reference/broker/index.gen.md @@ -1,4 +1,4 @@ -# pipeline broker +# broker ## Overview @@ -9,11 +9,12 @@ Manage the broker instance of the local pipeline **Usage:** ``` -quix pipeline broker [command] [options] +quix broker [command] [options] ``` **Commands:** - [down](down.md) : Stop the broker instance of the local pipeline and all the services depending on it +- [topics](topics/index.md) : Manage your pipeline topics - [up](up.md) : Start the broker instance of the local pipeline diff --git a/docs/Reference/cloud/applications/index.md b/docs/Reference/broker/index.md similarity index 100% rename from docs/Reference/cloud/applications/index.md rename to docs/Reference/broker/index.md diff --git a/docs/Reference/broker/topics/index.gen.md b/docs/Reference/broker/topics/index.gen.md new file mode 100644 index 0000000..8038b95 --- /dev/null +++ b/docs/Reference/broker/topics/index.gen.md @@ -0,0 +1,20 @@ +# broker topics + +## Overview + +Manage your pipeline topics + +**Aliases:** `topics`, `topic`, `tpc` + +**Usage:** + +``` +quix broker topics [command] [options] +``` + +**Commands:** + +- [list](list.md) : List your pipeline broker topics +- [read](read.md) : Read the messages from your pipeline topics +- [update](update.md) : Update or create your pipeline broker topics from your 'quix.yaml' configuration + diff --git a/docs/Reference/cloud/applications/library/index.md b/docs/Reference/broker/topics/index.md similarity index 100% rename from docs/Reference/cloud/applications/library/index.md rename to docs/Reference/broker/topics/index.md diff --git a/docs/Reference/broker/topics/list.gen.md b/docs/Reference/broker/topics/list.gen.md new file mode 100644 index 0000000..98a1097 --- /dev/null +++ b/docs/Reference/broker/topics/list.gen.md @@ -0,0 +1,20 @@ +# broker topics list + +## Overview + +List your pipeline broker topics + +**Aliases:** `list`, `ls` + +**Usage:** + +``` +quix broker topics list [options] +``` + +**Options:** + +- `-o, --output ` : Output type of the retrieved data +- `--directory ` : If specified, it will use that directory instead of the current one +- `-b, --broker-address ` : The broker address to connect to [default: localhost:19092] + diff --git a/docs/Reference/broker/topics/list.md b/docs/Reference/broker/topics/list.md new file mode 100644 index 0000000..184ca24 --- /dev/null +++ b/docs/Reference/broker/topics/list.md @@ -0,0 +1 @@ +{% include-markdown './list.gen.md' %} \ No newline at end of file diff --git a/docs/Reference/broker/topics/read.gen.md b/docs/Reference/broker/topics/read.gen.md new file mode 100644 index 0000000..f8b20f8 --- /dev/null +++ b/docs/Reference/broker/topics/read.gen.md @@ -0,0 +1,29 @@ +# broker topics read + +## Overview + +Read the messages from your pipeline topics + +**Aliases:** `read`, `tail` + +**Usage:** + +``` +quix broker topics read [] [options] +``` + +**Arguments:** + +- `` : The topic to read data from + +**Options:** + +- `-o, --output ` : Output type of the retrieved data +- `--directory ` : If specified, it will use that directory instead of the current one +- `-k, --key` : Include the key in the response +- `-O, --offset ` : The offset type +- `-l, --latest` : Selects the 'latest' offset type +- `-e, --earliest` : Selects the 'earliest' offset type +- `-b, --broker-address ` : The broker address to connect to [default: localhost:19092] +- `-g, --group ` : The consumer group ID + diff --git a/docs/Reference/broker/topics/read.md b/docs/Reference/broker/topics/read.md new file mode 100644 index 0000000..f051797 --- /dev/null +++ b/docs/Reference/broker/topics/read.md @@ -0,0 +1 @@ +{% include-markdown './read.gen.md' %} \ No newline at end of file diff --git a/docs/Reference/broker/topics/update.gen.md b/docs/Reference/broker/topics/update.gen.md new file mode 100644 index 0000000..ffb8060 --- /dev/null +++ b/docs/Reference/broker/topics/update.gen.md @@ -0,0 +1,19 @@ +# broker topics update + +## Overview + +Update or create your pipeline broker topics from your 'quix.yaml' configuration + +**Aliases:** `update`, `upd` + +**Usage:** + +``` +quix broker topics update [options] +``` + +**Options:** + +- `--directory ` : If specified, it will use that directory instead of the current one +- `-b, --broker-address ` : The broker address to connect to [default: localhost:19092] + diff --git a/docs/Reference/broker/topics/update.md b/docs/Reference/broker/topics/update.md new file mode 100644 index 0000000..f93a35f --- /dev/null +++ b/docs/Reference/broker/topics/update.md @@ -0,0 +1 @@ +{% include-markdown './update.gen.md' %} \ No newline at end of file diff --git a/docs/Reference/pipeline/broker/up.gen.md b/docs/Reference/broker/up.gen.md similarity index 67% rename from docs/Reference/pipeline/broker/up.gen.md rename to docs/Reference/broker/up.gen.md index 75481cc..9323f8b 100644 --- a/docs/Reference/pipeline/broker/up.gen.md +++ b/docs/Reference/broker/up.gen.md @@ -1,19 +1,20 @@ -# pipeline broker up +# broker up ## Overview Start the broker instance of the local pipeline -**Aliases:** `up`, `run` +**Aliases:** `up` **Usage:** ``` -quix pipeline broker up [options] +quix broker up [options] ``` **Options:** - `--directory ` : If specified, it will use that directory instead of the current one - `--dry-run` : Generate 'compose.yaml' without running it +- `-t, --create-topics` : Create topics specified in the quix.yaml diff --git a/docs/Reference/pipeline/broker/up.md b/docs/Reference/broker/up.md similarity index 55% rename from docs/Reference/pipeline/broker/up.md rename to docs/Reference/broker/up.md index 4570de3..5df4e60 100644 --- a/docs/Reference/pipeline/broker/up.md +++ b/docs/Reference/broker/up.md @@ -2,9 +2,9 @@ ## How It Works -The `quix pipeline broker up` command sets up and runs your local broker instance using Docker Compose. When executed, it generates the necessary `compose.yaml` file that defines the broker services required for your setup. It then builds and starts the Docker containers based on this configuration by running the equivalent of `docker compose up --build -d --remove-orphans`. +The `quix broker up` command sets up and runs your local broker instance using Docker Compose. When executed, it generates the necessary `compose.local.yaml` file that defines the broker services required for your setup. It then builds and starts the Docker containers based on this configuration by running the equivalent of `docker compose up --build -d --remove-orphans`. -If the dry-run option is used, the command allows you to generate the `compose.yaml` file without starting the containers, providing a preview of the configuration. This is particularly useful for validating your setup before making any changes to your running environment. +If the dry-run option is used, the command allows you to generate the `compose.local.yaml` file without starting the containers, providing a preview of the configuration. This is particularly useful for validating your setup before making any changes to your running environment. Deployments are connected using variables that ensure seamless data flow between different services. This process ensures that your local broker instance is correctly configured and running with all necessary services. For more details on the `docker compose up` command, refer to the [official Docker documentation](https://docs.docker.com/reference/cli/docker/compose/up/). @@ -22,20 +22,20 @@ Deployments are connected using variables that ensure seamless data flow between To start your local broker, use the following command: ```bash -$ quix pipeline broker up +$ quix broker up ``` -This command generates the necessary `compose.yaml` file and deployment configurations: +This command generates the necessary `compose.local.yaml` file and deployment configurations: ``` -Generating 'compose.yaml' -✓ Generated 'compose.yaml' +Generating 'compose.local.yaml' +✓ Generated 'compose.local.yaml' ``` -Next, it executes `docker compose up --build -d --remove-orphans` to build and run the Docker containers: +Next, it executes `docker compose compose.local.yaml up --build -d --remove-orphans` to build and run the Docker containers: ```text -Executing 'docker compose up --build -d --remove-orphans' +Executing 'docker compose compose.local.yaml up --build -d --remove-orphans' ``` ### Running the Containers @@ -53,18 +53,28 @@ Container githubrepo-console-1 Starting Container githubrepo-kafka-broker-1 Starting Container githubrepo-kafka-broker-1 Started Container githubrepo-console-1 Started + +✓ Open http://localhost:8080 to manage your pipeline broker ``` -## Generated `compose.yaml` File Overview +!!! tip + + Using the `--create-topics` or `-t"` options will create the topics from `quix.yaml` automatically. + +## Generated `compose.local.yaml` File Overview -The `compose.yaml` file configures the services in your local broker instance. Here's an overview of what will be generated: +The `compose.local.yaml` file configures the services in your local broker instance. Here's an overview of what will be generated: - **kafka-broker**: + - **build**: Specifies the context directory and Dockerfile for the Kafka broker. + - **environment**: Sets environment variables for the broker configuration, including the listener address and the Zookeeper instance address. - **console**: + - **build**: Specifies the context directory and Dockerfile for the Kafka management console. + - **environment**: Sets environment variables for the console configuration, including the Kafka broker address. -For more details on the `compose.yaml` file and its configurations, refer to the [official Docker Compose documentation](https://docs.docker.com/compose/compose-file/). \ No newline at end of file +For more details on the `compose.local.yaml` file and its configurations, refer to the [official Docker Compose documentation](https://docs.docker.com/compose/compose-file/). \ No newline at end of file diff --git a/docs/Reference/cloud/applications/get.gen.md b/docs/Reference/cloud/apps/get.gen.md similarity index 76% rename from docs/Reference/cloud/applications/get.gen.md rename to docs/Reference/cloud/apps/get.gen.md index c3e5255..dbf6fcc 100644 --- a/docs/Reference/cloud/applications/get.gen.md +++ b/docs/Reference/cloud/apps/get.gen.md @@ -1,4 +1,4 @@ -# cloud applications get +# cloud apps get ## Overview @@ -9,7 +9,7 @@ Get an application in the specified environment **Usage:** ``` -quix cloud applications get [ ] [options] +quix cloud apps get [ ] [options] ``` **Arguments:** diff --git a/docs/Reference/cloud/applications/get.md b/docs/Reference/cloud/apps/get.md similarity index 100% rename from docs/Reference/cloud/applications/get.md rename to docs/Reference/cloud/apps/get.md diff --git a/docs/Reference/cloud/applications/index.gen.md b/docs/Reference/cloud/apps/index.gen.md similarity index 71% rename from docs/Reference/cloud/applications/index.gen.md rename to docs/Reference/cloud/apps/index.gen.md index 793ed09..6f7c8fe 100644 --- a/docs/Reference/cloud/applications/index.gen.md +++ b/docs/Reference/cloud/apps/index.gen.md @@ -1,15 +1,15 @@ -# cloud applications +# cloud apps ## Overview Manage the applications of an environment -**Aliases:** `applications`, `application`, `apps`, `app` +**Aliases:** `apps`, `applications`, `application`, `app` **Usage:** ``` -quix cloud applications [command] [options] +quix cloud apps [command] [options] ``` **Commands:** diff --git a/docs/Reference/contexts/broker/index.md b/docs/Reference/cloud/apps/index.md similarity index 100% rename from docs/Reference/contexts/broker/index.md rename to docs/Reference/cloud/apps/index.md diff --git a/docs/Reference/cloud/applications/library/index.gen.md b/docs/Reference/cloud/apps/library/index.gen.md similarity index 57% rename from docs/Reference/cloud/applications/library/index.gen.md rename to docs/Reference/cloud/apps/library/index.gen.md index c5b550b..ad5017c 100644 --- a/docs/Reference/cloud/applications/library/index.gen.md +++ b/docs/Reference/cloud/apps/library/index.gen.md @@ -1,15 +1,15 @@ -# cloud applications library +# cloud apps library ## Overview Manage the library items of an environment -**Aliases:** `library` +**Aliases:** `library`, `lib` **Usage:** ``` -quix cloud applications library [command] [options] +quix cloud apps library [command] [options] ``` **Commands:** diff --git a/docs/Reference/pipeline/broker/index.md b/docs/Reference/cloud/apps/library/index.md similarity index 100% rename from docs/Reference/pipeline/broker/index.md rename to docs/Reference/cloud/apps/library/index.md diff --git a/docs/Reference/cloud/applications/library/list.gen.md b/docs/Reference/cloud/apps/library/list.gen.md similarity index 76% rename from docs/Reference/cloud/applications/library/list.gen.md rename to docs/Reference/cloud/apps/library/list.gen.md index e791a09..5b31591 100644 --- a/docs/Reference/cloud/applications/library/list.gen.md +++ b/docs/Reference/cloud/apps/library/list.gen.md @@ -1,4 +1,4 @@ -# cloud applications library list +# cloud apps library list ## Overview @@ -9,7 +9,7 @@ Get all library items **Usage:** ``` -quix cloud applications library list [options] +quix cloud apps library list [options] ``` **Options:** diff --git a/docs/Reference/cloud/applications/library/list.md b/docs/Reference/cloud/apps/library/list.md similarity index 100% rename from docs/Reference/cloud/applications/library/list.md rename to docs/Reference/cloud/apps/library/list.md diff --git a/docs/Reference/cloud/applications/list.gen.md b/docs/Reference/cloud/apps/list.gen.md similarity index 75% rename from docs/Reference/cloud/applications/list.gen.md rename to docs/Reference/cloud/apps/list.gen.md index def042b..5439bce 100644 --- a/docs/Reference/cloud/applications/list.gen.md +++ b/docs/Reference/cloud/apps/list.gen.md @@ -1,4 +1,4 @@ -# cloud applications list +# cloud apps list ## Overview @@ -9,7 +9,7 @@ Get all applications in the specified environment **Usage:** ``` -quix cloud applications list [] [options] +quix cloud apps list [] [options] ``` **Arguments:** diff --git a/docs/Reference/cloud/applications/list.md b/docs/Reference/cloud/apps/list.md similarity index 100% rename from docs/Reference/cloud/applications/list.md rename to docs/Reference/cloud/apps/list.md diff --git a/docs/Reference/cloud/environments/index.gen.md b/docs/Reference/cloud/environments/index.gen.md index 8936023..9961629 100644 --- a/docs/Reference/cloud/environments/index.gen.md +++ b/docs/Reference/cloud/environments/index.gen.md @@ -4,7 +4,7 @@ Manage the environments of your projects (e.g., Dev, Staging, Production) -**Aliases:** `environments`, `environment`, `envs`, `env`, `workspaces`, `workspace` +**Aliases:** `environments`, `environment`, `envs`, `env`, `workspaces`, `workspace`, `ws` **Usage:** @@ -17,5 +17,6 @@ quix cloud environments [command] [options] - [get](get.md) : Retrieves details of a specific environment accessible to the currently authenticated user - [list](list.md) : List all environments the current user has access to - [sync](sync.md) : Sync your environment based on the attached project +- [tokens](tokens/index.md) : Manage environment and SDK tokens - [use](use.md) : Select the default environment for the current context diff --git a/docs/Reference/cloud/environments/tokens/get.gen.md b/docs/Reference/cloud/environments/tokens/get.gen.md new file mode 100644 index 0000000..18a0f15 --- /dev/null +++ b/docs/Reference/cloud/environments/tokens/get.gen.md @@ -0,0 +1,18 @@ +# cloud environments tokens get + +## Overview + +Get the SDK/environment token + +**Aliases:** `get` + +**Usage:** + +``` +quix cloud environments tokens get [] [options] +``` + +**Arguments:** + +- `` + diff --git a/docs/Reference/cloud/environments/tokens/get.md b/docs/Reference/cloud/environments/tokens/get.md new file mode 100644 index 0000000..a41beb6 --- /dev/null +++ b/docs/Reference/cloud/environments/tokens/get.md @@ -0,0 +1 @@ +{% include-markdown './get.gen.md' %} \ No newline at end of file diff --git a/docs/Reference/cloud/environments/tokens/index.gen.md b/docs/Reference/cloud/environments/tokens/index.gen.md new file mode 100644 index 0000000..04d835f --- /dev/null +++ b/docs/Reference/cloud/environments/tokens/index.gen.md @@ -0,0 +1,19 @@ +# cloud environments tokens + +## Overview + +Manage environment and SDK tokens + +**Aliases:** `tokens`, `token` + +**Usage:** + +``` +quix cloud environments tokens [command] [options] +``` + +**Commands:** + +- [get](get.md) : Get the SDK/environment token +- [rotate](rotate.md) : Rotate and retrieve the SDK/environment token + diff --git a/docs/Reference/cloud/environments/tokens/index.md b/docs/Reference/cloud/environments/tokens/index.md new file mode 100644 index 0000000..2bd196e --- /dev/null +++ b/docs/Reference/cloud/environments/tokens/index.md @@ -0,0 +1 @@ +{% include-markdown './index.gen.md' %} \ No newline at end of file diff --git a/docs/Reference/cloud/environments/tokens/rotate.gen.md b/docs/Reference/cloud/environments/tokens/rotate.gen.md new file mode 100644 index 0000000..54c074a --- /dev/null +++ b/docs/Reference/cloud/environments/tokens/rotate.gen.md @@ -0,0 +1,18 @@ +# cloud environments tokens rotate + +## Overview + +Rotate and retrieve the SDK/environment token + +**Aliases:** `rotate` + +**Usage:** + +``` +quix cloud environments tokens rotate [] [options] +``` + +**Arguments:** + +- `` + diff --git a/docs/Reference/cloud/environments/tokens/rotate.md b/docs/Reference/cloud/environments/tokens/rotate.md new file mode 100644 index 0000000..c02f117 --- /dev/null +++ b/docs/Reference/cloud/environments/tokens/rotate.md @@ -0,0 +1 @@ +{% include-markdown './rotate.gen.md' %} \ No newline at end of file diff --git a/docs/Reference/cloud/index.gen.md b/docs/Reference/cloud/index.gen.md index 3b6425d..585e54f 100644 --- a/docs/Reference/cloud/index.gen.md +++ b/docs/Reference/cloud/index.gen.md @@ -17,7 +17,7 @@ quix cloud [command] [options] - [organisations](organisations/index.md) : Manage your organisation - [projects](projects/index.md) : Manage the projects of the current organisation - [environments](environments/index.md) : Manage the environments of your projects (e.g., Dev, Staging, Production) -- [applications](applications/index.md) : Manage the applications of an environment +- [apps](apps/index.md) : Manage the applications of an environment - [deployments](deployments/index.md) : Manage the deployments of an environment - [topics](topics/index.md) : Manage the topics of an environment - [users](users/index.md) : Manage the users of the organisation including audit, permissions and tokens diff --git a/docs/Reference/cloud/projects/index.gen.md b/docs/Reference/cloud/projects/index.gen.md index a910362..e1e6ad9 100644 --- a/docs/Reference/cloud/projects/index.gen.md +++ b/docs/Reference/cloud/projects/index.gen.md @@ -15,6 +15,6 @@ quix cloud projects [command] [options] **Commands:** - [get](get.md) : Get all projects of the current organisation -- [list](list.md) : Get all projects of the current organisation +- [list](list.md) : List all projects of the current organisation - [patch](patch.md) : Edit individual properties of a project diff --git a/docs/Reference/cloud/projects/list.gen.md b/docs/Reference/cloud/projects/list.gen.md index dce9ff9..ba2c8b8 100644 --- a/docs/Reference/cloud/projects/list.gen.md +++ b/docs/Reference/cloud/projects/list.gen.md @@ -2,7 +2,7 @@ ## Overview -Get all projects of the current organisation +List all projects of the current organisation **Aliases:** `list`, `ls` diff --git a/docs/Reference/cloud/topics/index.gen.md b/docs/Reference/cloud/topics/index.gen.md index fc52429..d520c22 100644 --- a/docs/Reference/cloud/topics/index.gen.md +++ b/docs/Reference/cloud/topics/index.gen.md @@ -4,7 +4,7 @@ Manage the topics of an environment -**Aliases:** `topics`, `topic` +**Aliases:** `topics`, `topic`, `tpc` **Usage:** diff --git a/docs/Reference/cloud/users/index.gen.md b/docs/Reference/cloud/users/index.gen.md index 624f52f..8987287 100644 --- a/docs/Reference/cloud/users/index.gen.md +++ b/docs/Reference/cloud/users/index.gen.md @@ -4,7 +4,7 @@ Manage the users of the organisation including audit, permissions and tokens -**Aliases:** `users`, `user` +**Aliases:** `users`, `user`, `usr` **Usage:** diff --git a/docs/Reference/cloud/users/permissions/copy.gen.md b/docs/Reference/cloud/users/permissions/copy.gen.md index f42be03..6f6b81e 100644 --- a/docs/Reference/cloud/users/permissions/copy.gen.md +++ b/docs/Reference/cloud/users/permissions/copy.gen.md @@ -4,7 +4,7 @@ Copy the user permissions from a user to other users -**Aliases:** `copy` +**Aliases:** `copy`, `cp` **Usage:** diff --git a/docs/Reference/cloud/users/permissions/index.gen.md b/docs/Reference/cloud/users/permissions/index.gen.md index 5d49287..d7e0cdc 100644 --- a/docs/Reference/cloud/users/permissions/index.gen.md +++ b/docs/Reference/cloud/users/permissions/index.gen.md @@ -4,7 +4,7 @@ Manage the user permissions -**Aliases:** `permissions`, `permission` +**Aliases:** `permissions`, `permission`, `perm`, `perms` **Usage:** diff --git a/docs/Reference/cloud/users/tokens/create.gen.md b/docs/Reference/cloud/users/tokens/create.gen.md index 779418a..a770498 100644 --- a/docs/Reference/cloud/users/tokens/create.gen.md +++ b/docs/Reference/cloud/users/tokens/create.gen.md @@ -19,6 +19,6 @@ quix cloud users tokens create [options] **Options:** - `-o, --output ` : Output type of the retrieved data -- `--expires-at ` : Define when the token will expire. Supports ISO 8601 dates: 'YYYY-MM-DD HH:MM:SS.sss'. All times UTC. [default: 2025-06-27] +- `--expires-at ` : Define when the token will expire. Supports ISO 8601 dates: 'YYYY-MM-DD HH:MM:SS.sss'. All times UTC. [default: 2025-08-21] - `-p, --permission-assignments (REQUIRED)` : Array of permission assignments in the format "[{Scope, Permission, Permission, ..}, {Scope, Permission, Permission, ..}, ...]" diff --git a/docs/Reference/contexts/broker/cloud.gen.md b/docs/Reference/contexts/broker/cloud.gen.md deleted file mode 100644 index 60930fc..0000000 --- a/docs/Reference/contexts/broker/cloud.gen.md +++ /dev/null @@ -1,21 +0,0 @@ -# contexts broker cloud - -## Overview - -Set the SDK broker configuration to point to a 'Quix Cloud' broker - -**Aliases:** `cloud`, `disable` - -**Usage:** - -``` -quix contexts broker cloud [options] -``` - -**Options:** - -- `--workspace-id ` : The environment to use if your SDK broker configuration is set to Quix Cloud -- `--env-file-name ` : Set the name of the environment file [default: .env] -- `--directory ` : Base directory (defaults to current directory) -- `--update` : Update '.env' files of all the applications with the current SDK broker configuration - diff --git a/docs/Reference/contexts/broker/cloud.md b/docs/Reference/contexts/broker/cloud.md deleted file mode 100644 index 1fc6188..0000000 --- a/docs/Reference/contexts/broker/cloud.md +++ /dev/null @@ -1,5 +0,0 @@ -{% include-markdown './cloud.gen.md' %} - -!!! info - - Used as an Alias for [this](../../sdk/broker/cloud.md) command \ No newline at end of file diff --git a/docs/Reference/contexts/broker/index.gen.md b/docs/Reference/contexts/broker/index.gen.md deleted file mode 100644 index 3c53adf..0000000 --- a/docs/Reference/contexts/broker/index.gen.md +++ /dev/null @@ -1,20 +0,0 @@ -# contexts broker - -## Overview - -Manage the SDK broker configuration that is injected to QuixStreams while developing locally - -**Aliases:** `broker` - -**Usage:** - -``` -quix contexts broker [command] [options] -``` - -**Commands:** - -- [cloud](cloud.md) : Set the SDK broker configuration to point to a 'Quix Cloud' broker -- [local](local.md) : Set the SDK broker configuration to point to a 'Local' broker (default: pipeline broker) -- [set](set.md) : Set a local broker address for the SDK broker configuration - diff --git a/docs/Reference/contexts/broker/local.gen.md b/docs/Reference/contexts/broker/local.gen.md deleted file mode 100644 index 8f77bf1..0000000 --- a/docs/Reference/contexts/broker/local.gen.md +++ /dev/null @@ -1,20 +0,0 @@ -# contexts broker local - -## Overview - -Set the SDK broker configuration to point to a 'Local' broker (default: pipeline broker) - -**Aliases:** `local`, `enable` - -**Usage:** - -``` -quix contexts broker local [options] -``` - -**Options:** - -- `--env-file-name ` : Set the name of the environment file [default: .env] -- `--directory ` : Base directory (defaults to current directory) -- `--update` : Update '.env' files of all the applications with the current SDK broker configuration - diff --git a/docs/Reference/contexts/broker/local.md b/docs/Reference/contexts/broker/local.md deleted file mode 100644 index b34c103..0000000 --- a/docs/Reference/contexts/broker/local.md +++ /dev/null @@ -1,5 +0,0 @@ -{% include-markdown './local.gen.md' %} - -!!! info - - Used as an Alias for [this](../../sdk/broker/local.md) command \ No newline at end of file diff --git a/docs/Reference/contexts/broker/set.gen.md b/docs/Reference/contexts/broker/set.gen.md deleted file mode 100644 index e0d2a45..0000000 --- a/docs/Reference/contexts/broker/set.gen.md +++ /dev/null @@ -1,26 +0,0 @@ -# contexts broker set - -## Overview - -Set a local broker address for the SDK broker configuration - -**Aliases:** `set` - -**Usage:** - -``` -quix contexts broker set [
] [options] -``` - -**Arguments:** - -- `
` : The broker address for the SDK broker configuration (ie: localhost:19092) - -**Options:** - -- `--enable` : Enables Local broker as SDK broker configuration -- `--workspace-id ` : The environment to use if your SDK broker configuration is set to Quix Cloud -- `--env-file-name ` : Set the name of the environment file [default: .env] -- `--directory ` : Base directory (defaults to current directory) -- `--update` : Update '.env' files of all the applications with the current SDK broker configuration [] - diff --git a/docs/Reference/contexts/broker/set.md b/docs/Reference/contexts/broker/set.md deleted file mode 100644 index 2faa547..0000000 --- a/docs/Reference/contexts/broker/set.md +++ /dev/null @@ -1,5 +0,0 @@ -{% include-markdown './set.gen.md' %} - -!!! info - - Used as an Alias for [this](../../sdk/broker/set.md) command \ No newline at end of file diff --git a/docs/Reference/contexts/environments/index.gen.md b/docs/Reference/contexts/environments/index.gen.md index 97e58ab..463d89f 100644 --- a/docs/Reference/contexts/environments/index.gen.md +++ b/docs/Reference/contexts/environments/index.gen.md @@ -4,7 +4,7 @@ Manage the default environment -**Aliases:** `environments`, `environment`, `envs`, `env`, `workspaces`, `workspace` +**Aliases:** `environments`, `environment`, `envs`, `env`, `workspaces`, `workspace`, `ws` **Usage:** diff --git a/docs/Reference/contexts/index.gen.md b/docs/Reference/contexts/index.gen.md index 7b14b97..e56d967 100644 --- a/docs/Reference/contexts/index.gen.md +++ b/docs/Reference/contexts/index.gen.md @@ -14,7 +14,6 @@ quix contexts [command] [options] **Commands:** -- [broker](broker/index.md) : Manage the SDK broker configuration that is injected to QuixStreams while developing locally - [create](create.md) : Add a new context to the CLI - [current](current.md) : Gets information about the current context - [delete](delete.md) : Remove a context from the CLI diff --git a/docs/Reference/ide.gen.md b/docs/Reference/ide.gen.md deleted file mode 100644 index 3067cf4..0000000 --- a/docs/Reference/ide.gen.md +++ /dev/null @@ -1,25 +0,0 @@ -# ide - -## Overview - -Launch your local IDE configuring all the settings for local application development - -**Aliases:** `ide` - -**Usage:** - -``` -quix ide [] [options] -``` - -**Arguments:** - -- `` : Base directory (defaults to current directory) - -**Options:** - -- `-dev, --devcontainer, --devcontainers` : If specified, generates devcontainers files -- `--vscode` : Selects VS Code -- `--pycharm` : Selects PyCharm -- `--use ` : Selects the IDE explicitly. (vscode, pycharm) - diff --git a/docs/Reference/ide.md b/docs/Reference/ide.md deleted file mode 100644 index d83b6e5..0000000 --- a/docs/Reference/ide.md +++ /dev/null @@ -1,127 +0,0 @@ -{% include-markdown './ide.gen.md' %} - -## How It Works - -The `quix ide` command sets up your development environment for a specified application by generating all necessary IDE-dependent files. This integration allows you to run and debug your application directly from your chosen IDE. - -This command is particularly useful the first time you edit an application, as it automates the setup process. While you can run it multiple times, its primary benefits are realized during the initial setup. - -!!! note - You need to have [VS Code](https://code.visualstudio.com/) or [PyCharm](https://www.jetbrains.com/pycharm/) installed for this command to be effective. - -### Automatic Environment Variable Export - -Each time you run or debug your code using the generated run configuration, the command exports variables from the `app.yaml` file to a `.env` file or a devcontainer configuration. This ensures that the environment is correctly set up before you begin editing and debugging. - -Specifically, this command executes [`quix application variables export`](./applications/variables/export.md), which adds the application values along with the broker or SDK token to a `.env` file. - -#### Debug Broker Configuration - -=== "Pipeline Broker" - - To use a SDK broker configuration, you can easily set one up by running [`quix pipeline broker up`](./pipeline/broker/up.md). This will start a local broker instance that listens on `localhost:19092`. This setting is enabled by default. If you need to change it, use `quix sdk broker local`. - -=== "Quix Cloud Broker" - - To use the broker address from your Quix Cloud environment, use `quix sdk broker cloud`. Ensure you are logged into Quix Cloud by running `quix login` if you are not already logged in. If you have never selected an environment, run `quix use` to select it. - -=== "Other Broker" - - To use a broker address other than `localhost:19092`, run `quix sdk broker set`. - - You can edit the *debug broker configuration* to suit your needs. The default configuration is designed to work out-of-the-box, but you can change it to point to any broker, including a Quix Cloud environment. - -In practice, this setup adds the following environment variable to your configuration: - -``` -Quix__Broker__Address={your_debug_broker_address} -``` - -If you set your SDK broker configuration configuration to point to Quix Cloud, it will add: - -``` -Quix__Sdk__Token={your_token} -``` - -!!! tip - Ensure you have a local broker configured or a Quix Cloud workspace set up. Without these, you won't have a Kafka instance to run your application against, and the command will not function properly. - -### Generated Files - -#### IDE Files - -=== "VS Code" - - For VS Code, the `quix ide` command generates the necessary configurations to ensure smooth debugging and development: - - - **launch.json**: Configures the Python debugger to use the integrated terminal. It includes a pre-launch task `quix-variables-export` to set environment variables correctly before debugging. - - **tasks.json**: Defines a custom task `quix-variables-export` that exports application variables from the `app.yaml` file to a `.env` file. This task is executed before running the debugger. - -=== "PyCharm" - - For PyCharm, the `quix ide` command generates configurations to facilitate debugging: - - - **Quix.xml**: Configures a run configuration that includes a pre-launch task to export environment variables using the `quix-variables-export` external tool. - - **External Tools.xml**: Defines an external tool to export application variables from the `app.yaml` file to a `.env` file. This tool runs before the main application execution starts. - -These configurations ensure that your development environment is correctly set up, enabling you to run and debug your application seamlessly. - -#### Dev Containers - -Dev Containers provide an isolated and consistent development environment. By using the `--devcontainer` option, the command sets up the necessary configurations for Dev Containers. This is particularly useful for ensuring your development environment matches your production environment, helping to avoid the common "it works on my machine" issues by creating a reproducible setup. - -The dev container configuration reuses your existing Dockerfile, installs Python and Docker extensions, and sets up the Quix CLI within the dev container. - -The configuration includes: - -- **Build**: Uses your existing Dockerfile to build the dev container. -- **Features**: Adds SSHD and Git features from Dev Containers, along with the Quix CLI. -- **Customizations**: Configures VS Code settings and extensions, ensuring Python and Docker tools are available. -- **Mounts**: Binds your local Quix configuration to the dev container, ensuring consistency. - -This setup ensures that your development environment is as close to the production environment as possible, providing a smooth and consistent development experience. - -For more information on Dev Containers, visit [Dev Containers](https://containers.dev/). - -## Example usage - -When you execute the local ide command without any options: - -```bash -$ quix ide -``` - -The interactive command starts. If your current directory is not an application directory, you are prompted to select the application you want to edit and debug in your IDE: - -```text -? Select application: - - Event Detection Transformation \Event Detection Transformation -> Starter Source \Starter Source - ✗ Cancel -``` - -After selecting the application, the command performs several checks to ensure that you can debug the application locally: - -```text -✓ Found application 'Starter Source' -✓ Using localhost:19092 -✓ VS Code is installed -✓ PyCharm is installed -``` - -If you have more than one supported IDE installed, you are asked to select which one to use: - -```text -? Select IDE: - -> VS Code - PyCharm - ✗ Cancel -``` - -Finally, your selected IDE starts: - -```text -✓ VS Code started -``` \ No newline at end of file diff --git a/docs/Reference/init.gen.md b/docs/Reference/init.gen.md index 1e02414..c1270ca 100644 --- a/docs/Reference/init.gen.md +++ b/docs/Reference/init.gen.md @@ -18,9 +18,8 @@ quix init [] [options] **Options:** -- `-dev, --devcontainers` : If specified, generates devcontainers files -- `--vscode` : If specified, generates VS Code launch files -- `--pycharm` : If specified, generates PyCharm launch files -- `--ide ` : Selects the IDE explicitly. (vscode, pycharm) -- `-d, --generate-dockerfiles` : If specified, generates dockerfiles based on the applications [default: True] +- `-d, --devcontainer, --devcontainers` : If specified, generates devcontainers files +- `-g, --generate-dockerfiles` : If specified, generates dockerfiles based on the applications [default: True] +- `-u, --update, --update-dotenv` : Update the '.env' files even if they already exist +- `-r, --reset, --reset-dotenv` : Reset the '.env' files and recreate them diff --git a/docs/Reference/init.md b/docs/Reference/init.md index 748ce99..f56594a 100644 --- a/docs/Reference/init.md +++ b/docs/Reference/init.md @@ -1,30 +1,68 @@ {% include-markdown './init.gen.md' %} + ## How It Works -The `quix init` command sets up the initial Quix files based on your existing repository. When executed, it generates the necessary configuration files such as `dockerfile` and `app.yaml` for your application. +The `quix init` command prepares your Quix project by generating essential configuration files based on the structure of your existing repository. These files include `dockerfile`, `app.yaml`, and `.env`, which are critical for development and deployment. + +When executed in the root folder of your project, `quix init` performs several key actions: -If run at the root level of your project, it will also create a `quix.yaml` file if one does not exist, and it attempts to generate any missing `app.yaml` and `dockerfile` files for all applications within the project. This ensures that your development environment is properly configured for seamless development and deployment. +- **Creates a `quix.yaml` File**: If one doesn't already exist, `quix init` generates an empty `quix.yaml` as the central configuration file for your project. +- **Generates Missing Files**: The command scans your project and creates any missing configuration files like `app.yaml`, `dockerfile`, and `.env` for all applications within the project. !!! tip - The `-dev, --devcontainers`, `--vscode`, `--pycharm`, and `--ide ` options work similarly to those in the [`quix ide`](./ide.md) command. + Running `quix init` ensures your project is fully initialized, filling in any missing pieces so you can focus on development. + +### Customization Based on Project Structure + +`quix init` customizes the generated files according to your project's language and dependencies: + +- **Language Detection**: Automatically detects the programming language and generates appropriate files, such as `requirements.txt` for Python projects. +- **Dependency Management**: Identifies and incorporates dependencies into the configuration files, simplifying the process of containerization and deployment. + +### Support for DevContainers + +The command can also generate DevContainer files, ensuring a consistent development environment across your team. This feature helps avoid issues where code works on one machine but not another. + +### Managing Environment Files -## Example usage +`quix init` offers flexibility in handling `.env` files: -### Creating an Application from Python Code +- **Update Existing Files**: Use the `-u` option to update existing `.env` files. +- **Reset Files**: Use the `-r` option to reset and recreate `.env` files from scratch. -If you have a Python file named `main.py` and you want to create a Quix application from it, you can use the following command: +### Integration with Git + +To safeguard sensitive data, `quix init` automatically updates your `.gitignore` file to exclude files like `.env`, ensuring they are not accidentally committed to your version control system. + +## Example Usage + +### Setting Up a Python Application + +If you have a Python file named `main.py` and want to create a Quix application from it, simply run: ```bash $ quix init ``` -When you run this command, it will automatically generate a `dockerfile` and an `app.yaml` file for your application. These files are essential for containerizing your application and defining its configuration, allowing you to easily deploy and manage your application in a consistent environment. +Here’s what happens when you run `quix init` in a directory containing a single `main.py` file: + +```bash +$ quix init +✓ Created dockerfile in +✓ Created requirements.txt in +✓ Created 'app.yaml' in '' +``` -Using `quix init` simplifies the process of setting up your application, ensuring that all required files are created and properly configured. +After running `quix init`, your directory will contain: -### Additional Functionality +- **`main.py`**: Your original Python file. +- **`dockerfile`**: Used for containerizing your application. +- **`app.yaml`**: Configures how your application runs within Quix. +- **`.env`**: Contains environment-specific variables. +- **`requirements.txt`**: Lists Python dependencies. -If you run the `quix init` command at the root level of your project directory, it generates an empty `quix.yaml` file if it's not there yet. Additionally, it will attempt to generate any missing `app.yaml` and `dockerfile` for all applications within the project. +### Additional Features +Running `quix init` in the root of your project also attempts to generate any missing `app.yaml`, `dockerfile`, and `.env` files for all applications, ensuring your project is fully prepared for development and deployment. It also supports DevContainers and integrates seamlessly with Git. -Running the command at the root level ensures that your entire project is properly initialized and configured, with all necessary files created for seamless development and deployment. +In summary, `quix init` is a powerful tool that simplifies the setup and management of your Quix projects, ensuring everything is correctly configured for efficient development, testing, and deployment. \ No newline at end of file diff --git a/docs/Reference/pipeline/down.gen.md b/docs/Reference/pipeline/down.gen.md index 87c44dd..660489e 100644 --- a/docs/Reference/pipeline/down.gen.md +++ b/docs/Reference/pipeline/down.gen.md @@ -4,7 +4,7 @@ Stop the execution of your local pipeline -**Aliases:** `down`, `stop` +**Aliases:** `down` **Usage:** diff --git a/docs/Reference/pipeline/down.md b/docs/Reference/pipeline/down.md index 3c60bcd..fb80607 100644 --- a/docs/Reference/pipeline/down.md +++ b/docs/Reference/pipeline/down.md @@ -2,7 +2,7 @@ ## How It Works -The `quix pipeline down` command stops and removes all the Docker containers defined in your `compose.yaml` file. It acts as a shortcut for the `docker compose down` command, ensuring that all running services are stopped and all associated containers, networks, and volumes are removed. +The `quix pipeline down` command stops and removes all the Docker containers defined in your `compose.local.yaml` file. It acts as a shortcut for the `docker compose down` command, ensuring that all running services are stopped and all associated containers, networks, and volumes are removed. When executed, this command performs the following steps: @@ -23,7 +23,7 @@ This process ensures that your environment is clean and ready for the next time ### Shutting Down the Pipeline with Docker Compose -To stop and remove all the containers defined in your `compose.yaml` file, use the following command: +To stop and remove all the containers defined in your `compose.local.yaml` file, use the following command: ```bash $ quix pipeline down diff --git a/docs/Reference/pipeline/index.gen.md b/docs/Reference/pipeline/index.gen.md index 6a34a11..d541d07 100644 --- a/docs/Reference/pipeline/index.gen.md +++ b/docs/Reference/pipeline/index.gen.md @@ -4,7 +4,7 @@ Manage your local pipeline operations and resources -**Aliases:** `pipeline`, `pipe` +**Aliases:** `pipeline`, `pipe`, `pl` **Usage:** @@ -15,11 +15,14 @@ quix pipeline [command] [options] **Commands:** - [view](view.md) : Create and preview a mermaid diagram of the pipeline -- [broker](broker/index.md) : Manage the broker instance of the local pipeline - [deployments](deployments/index.md) : Manage the deployments of the local pipeline - [status](status.md) : Display the current status of the local pipeline - [update](update.md) : Update 'quix.yaml' with the new local applications and update the variables of the existing deployments - [sync](sync.md) : Synchronize your local pipeline to Quix Cloud +- [topics](topics/index.md) : Manage the topics of your local pipeline +- [logs](logs.md) : View output from deployments of the local pipeline - [up](up.md) : Run your pipeline using docker compose - [down](down.md) : Stop the execution of your local pipeline +- [start](start.md) : Start a deployment of the local pipeline +- [stop](stop.md) : Stop a deployment of the local pipeline diff --git a/docs/Reference/pipeline/logs.gen.md b/docs/Reference/pipeline/logs.gen.md new file mode 100644 index 0000000..8c37e43 --- /dev/null +++ b/docs/Reference/pipeline/logs.gen.md @@ -0,0 +1,27 @@ +# pipeline logs + +## Overview + +View output from deployments of the local pipeline + +**Aliases:** `logs` + +**Usage:** + +``` +quix pipeline logs [] [options] +``` + +**Arguments:** + +- `` : The name of deployment + +**Options:** + +- `--directory ` : If specified, it will use that directory instead of the current one +- `--all` : If specified it will get all deployments logs +- `-f, --follow` : Follow log output +- `-n, --tail ` : Number of lines to show from the end of the logs for each container (default "all") +- `--since ` : Show logs since timestamp (e.g.2013-01-02T13:23:37Z) or relative (e.g. 42m for 42 minutes) +- `--until ` : Show logs before a timestamp (e.g.2013-01-02T13:23:37Z) or relative (e.g. 42m for 42 minutes) + diff --git a/docs/Reference/pipeline/logs.md b/docs/Reference/pipeline/logs.md new file mode 100644 index 0000000..7b993b5 --- /dev/null +++ b/docs/Reference/pipeline/logs.md @@ -0,0 +1 @@ +{% include-markdown './logs.gen.md' %} \ No newline at end of file diff --git a/docs/Reference/pipeline/start.gen.md b/docs/Reference/pipeline/start.gen.md new file mode 100644 index 0000000..2f94d11 --- /dev/null +++ b/docs/Reference/pipeline/start.gen.md @@ -0,0 +1,20 @@ +# pipeline start + +## Overview + +Start a deployment of the local pipeline + +**Aliases:** `start` + +**Usage:** + +``` +quix pipeline start [options] +``` + +**Options:** + +- `--directory ` : If specified, it will use that directory instead of the current one +- `-d, --deployment ` : The deployment you want to start +- `-a, --all` : Start all deployments + diff --git a/docs/Reference/pipeline/start.md b/docs/Reference/pipeline/start.md new file mode 100644 index 0000000..64c82f8 --- /dev/null +++ b/docs/Reference/pipeline/start.md @@ -0,0 +1 @@ +{% include-markdown './start.gen.md' %} \ No newline at end of file diff --git a/docs/Reference/pipeline/status.md b/docs/Reference/pipeline/status.md index 2cf72d4..e19ef6e 100644 --- a/docs/Reference/pipeline/status.md +++ b/docs/Reference/pipeline/status.md @@ -29,9 +29,9 @@ If all services in the pipeline are running, you will see output similar to the ``` Local Pipeline Status: ✓ Running (4/4) - console ✓ Up 1 second + console ✓ Up 1 second http://localhost:19092 + kafka-broker ✓ Up 1 second localhost:19092 demo-data-source ✓ Up 1 second - kafka-broker ✓ Up 1 second event-detection-transformation ✓ Up 1 second ``` @@ -41,10 +41,10 @@ If some services are not running, the output will indicate which services have e ``` Local Pipeline Status: ✗ Running (3/4) - console ✓ Up 1 second - demo-data-source ✓ Up 1 second - kafka-broker ✓ Up 1 second - event-detection-transformation ✗ Exited (1) 1 second ago + console ✓ Up 1 second http://localhost:19092 + kafka-broker ✓ Up 1 second localhost:19092 + demo-data-source ✓ Up 1 second + event-detection-transformation ✗ Exited (0) 1 second ago ``` #### No Services Running diff --git a/docs/Reference/pipeline/stop.gen.md b/docs/Reference/pipeline/stop.gen.md new file mode 100644 index 0000000..76f6831 --- /dev/null +++ b/docs/Reference/pipeline/stop.gen.md @@ -0,0 +1,20 @@ +# pipeline stop + +## Overview + +Stop a deployment of the local pipeline + +**Aliases:** `stop` + +**Usage:** + +``` +quix pipeline stop [options] +``` + +**Options:** + +- `--directory ` : If specified, it will use that directory instead of the current one +- `-d, --deployment ` : The deployment you want to stop +- `-a, --all` : Stop all deployments + diff --git a/docs/Reference/pipeline/stop.md b/docs/Reference/pipeline/stop.md new file mode 100644 index 0000000..5dcfd34 --- /dev/null +++ b/docs/Reference/pipeline/stop.md @@ -0,0 +1 @@ +{% include-markdown './stop.gen.md' %} \ No newline at end of file diff --git a/docs/Reference/pipeline/sync.gen.md b/docs/Reference/pipeline/sync.gen.md index 26c4c09..a281c4e 100644 --- a/docs/Reference/pipeline/sync.gen.md +++ b/docs/Reference/pipeline/sync.gen.md @@ -17,5 +17,7 @@ quix pipeline sync [options] - `--workspace-id ` : Select a concrete workspace-id to sync to - `--directory ` : Base directory (defaults to current directory) - `-u, --update` : Update 'quix.yaml' with the new local applications and update the variables of the existing deployments +- `-k, --keep-variables` : Keep the locally deleted variables +- `-p, --push` : Push the current changes to your repository - `-m, --commit-message ` : The message used for the commit if using '--push' diff --git a/docs/Reference/pipeline/topics/create.gen.md b/docs/Reference/pipeline/topics/create.gen.md new file mode 100644 index 0000000..270be2e --- /dev/null +++ b/docs/Reference/pipeline/topics/create.gen.md @@ -0,0 +1,23 @@ +# pipeline topics create + +## Overview + +Add a topic to your local pipeline + +**Aliases:** `create`, `add` + +**Usage:** + +``` +quix pipeline topics create [] [options] +``` + +**Arguments:** + +- `` : The name of the topic to edit + +**Options:** + +- `-p, --partitions ` : Topic partitions, if not specified it will use the default +- `--directory ` : Base directory (defaults to current directory) + diff --git a/docs/Reference/pipeline/topics/create.md b/docs/Reference/pipeline/topics/create.md new file mode 100644 index 0000000..f7828d6 --- /dev/null +++ b/docs/Reference/pipeline/topics/create.md @@ -0,0 +1 @@ +{% include-markdown './create.gen.md' %} \ No newline at end of file diff --git a/docs/Reference/pipeline/topics/delete.gen.md b/docs/Reference/pipeline/topics/delete.gen.md new file mode 100644 index 0000000..62d35be --- /dev/null +++ b/docs/Reference/pipeline/topics/delete.gen.md @@ -0,0 +1,23 @@ +# pipeline topics delete + +## Overview + +Delete a topic from your local pipeline + +**Aliases:** `delete`, `del`, `remove`, `rm` + +**Usage:** + +``` +quix pipeline topics delete [] [options] +``` + +**Arguments:** + +- `` : The name of topic + +**Options:** + +- `-o, --output ` : Output type of the retrieved data +- `--directory ` : Base directory (defaults to current directory) + diff --git a/docs/Reference/pipeline/topics/delete.md b/docs/Reference/pipeline/topics/delete.md new file mode 100644 index 0000000..7fba4ff --- /dev/null +++ b/docs/Reference/pipeline/topics/delete.md @@ -0,0 +1 @@ +{% include-markdown './delete.gen.md' %} \ No newline at end of file diff --git a/docs/Reference/pipeline/topics/edit.gen.md b/docs/Reference/pipeline/topics/edit.gen.md new file mode 100644 index 0000000..1b45d5d --- /dev/null +++ b/docs/Reference/pipeline/topics/edit.gen.md @@ -0,0 +1,23 @@ +# pipeline topics edit + +## Overview + +Edit a topic from your local pipeline + +**Aliases:** `edit` + +**Usage:** + +``` +quix pipeline topics edit [] [options] +``` + +**Arguments:** + +- `` : The name of the topic to edit + +**Options:** + +- `-p, --partitions ` : Topic partitions, if not specified it will use the default +- `--directory ` : Base directory (defaults to current directory) + diff --git a/docs/Reference/pipeline/topics/edit.md b/docs/Reference/pipeline/topics/edit.md new file mode 100644 index 0000000..7fc98ca --- /dev/null +++ b/docs/Reference/pipeline/topics/edit.md @@ -0,0 +1 @@ +{% include-markdown './edit.gen.md' %} \ No newline at end of file diff --git a/docs/Reference/pipeline/topics/get.gen.md b/docs/Reference/pipeline/topics/get.gen.md new file mode 100644 index 0000000..2f6c0ff --- /dev/null +++ b/docs/Reference/pipeline/topics/get.gen.md @@ -0,0 +1,23 @@ +# pipeline topics get + +## Overview + +Get a topic from your local pipeline + +**Aliases:** `get` + +**Usage:** + +``` +quix pipeline topics get [] [options] +``` + +**Arguments:** + +- `` : The name of topic + +**Options:** + +- `-o, --output ` : Output type of the retrieved data +- `--directory ` : Base directory (defaults to current directory) + diff --git a/docs/Reference/pipeline/topics/get.md b/docs/Reference/pipeline/topics/get.md new file mode 100644 index 0000000..a41beb6 --- /dev/null +++ b/docs/Reference/pipeline/topics/get.md @@ -0,0 +1 @@ +{% include-markdown './get.gen.md' %} \ No newline at end of file diff --git a/docs/Reference/pipeline/topics/index.gen.md b/docs/Reference/pipeline/topics/index.gen.md new file mode 100644 index 0000000..fa520d4 --- /dev/null +++ b/docs/Reference/pipeline/topics/index.gen.md @@ -0,0 +1,22 @@ +# pipeline topics + +## Overview + +Manage the topics of your local pipeline + +**Aliases:** `topics`, `topic`, `tpc` + +**Usage:** + +``` +quix pipeline topics [command] [options] +``` + +**Commands:** + +- [create](create.md) : Add a topic to your local pipeline +- [delete](delete.md) : Delete a topic from your local pipeline +- [edit](edit.md) : Edit a topic from your local pipeline +- [get](get.md) : Get a topic from your local pipeline +- [list](list.md) : List all the topics from your local + diff --git a/docs/Reference/pipeline/topics/index.md b/docs/Reference/pipeline/topics/index.md new file mode 100644 index 0000000..2bd196e --- /dev/null +++ b/docs/Reference/pipeline/topics/index.md @@ -0,0 +1 @@ +{% include-markdown './index.gen.md' %} \ No newline at end of file diff --git a/docs/Reference/pipeline/topics/list.gen.md b/docs/Reference/pipeline/topics/list.gen.md new file mode 100644 index 0000000..d1c28a4 --- /dev/null +++ b/docs/Reference/pipeline/topics/list.gen.md @@ -0,0 +1,19 @@ +# pipeline topics list + +## Overview + +List all the topics from your local + +**Aliases:** `list`, `ls` + +**Usage:** + +``` +quix pipeline topics list [options] +``` + +**Options:** + +- `--directory ` : Base directory (defaults to current directory) +- `-o, --output ` : Output type of the retrieved data + diff --git a/docs/Reference/pipeline/topics/list.md b/docs/Reference/pipeline/topics/list.md new file mode 100644 index 0000000..184ca24 --- /dev/null +++ b/docs/Reference/pipeline/topics/list.md @@ -0,0 +1 @@ +{% include-markdown './list.gen.md' %} \ No newline at end of file diff --git a/docs/Reference/pipeline/up.gen.md b/docs/Reference/pipeline/up.gen.md index 4a4f537..d136115 100644 --- a/docs/Reference/pipeline/up.gen.md +++ b/docs/Reference/pipeline/up.gen.md @@ -4,7 +4,7 @@ Run your pipeline using docker compose -**Aliases:** `up`, `run` +**Aliases:** `up` **Usage:** @@ -15,6 +15,8 @@ quix pipeline up [options] **Options:** - `--directory ` : If specified, it will use that directory instead of the current one -- `--dry-run` : Generate 'compose.yaml' without running it +- `-D, --dry-run` : Generate 'compose.yaml' without running it - `-u, --update` : Update 'quix.yaml' with the new local applications and update the variables of the existing deployments +- `-s, --skip-topics` : Skip topics creation, in this case Quix Streams might create the topics automatically +- `-k, --keep-variables` : Keep the locally deleted variables diff --git a/docs/Reference/pipeline/up.md b/docs/Reference/pipeline/up.md index c0b310a..bdaf8c8 100644 --- a/docs/Reference/pipeline/up.md +++ b/docs/Reference/pipeline/up.md @@ -2,9 +2,9 @@ ## How It Works -The `quix pipeline up` command sets up and runs your pipeline using Docker Compose. When executed, it generates the necessary `compose.yaml` file that defines the services in your pipeline. It then builds and starts the Docker containers based on this configuration by running the equivalent of `docker compose up --build -d`. +The `quix pipeline up` command sets up and runs your pipeline using Docker Compose. When executed, it generates the necessary `compose.local.yaml` file that defines the services in your pipeline. It then builds and starts the Docker containers based on this configuration by running the equivalent of `docker compose up --build -d`. -If the update option is used, the command updates the `quix.yaml` file with new local applications and variables of existing deployments. The dry-run option allows you to generate the `compose.yaml` file without starting the containers, providing a preview of the configuration. +If the update option is used, the command updates the `quix.yaml` file with new local applications and variables of existing deployments. The dry-run option allows you to generate the `compose.local.yaml` file without starting the containers, providing a preview of the configuration. Deployments are connected using variables with `VariableType` of `InputTopic` and `OutputTopic`, ensuring seamless data flow between different services. @@ -26,25 +26,55 @@ To start your pipeline, use the following command: $ quix pipeline up ``` -This command generates the necessary `compose.yaml` file and deployment configurations: +This command generates the necessary `compose.local.yaml` file and deployment configurations: ``` -Generating 'compose.yaml' -Generating deployment demo-data-source + Generating 'compose.local.yaml' + Generating deployment demo-data-source ✓ Generated deployment demo-data-source -Generating deployment event-detection-transformation + Generating deployment event-detection-transformation ✓ Generated deployment event-detection-transformation -✓ Generated 'compose.yaml' +✓ Generated 'compose.local.yaml' ``` !!! tip - You can also use the `--dry-run` option to generate the `compose.yaml` file without running it: + You can also use the `--dry-run` or `-D` option to generate the `compose.local.yaml` file without running it: ```bash $ quix pipeline up --dry-run ``` -Next, it executes `docker compose up --build -d --remove-orphans` to build and run the Docker containers: +When the compose file is generated, it will start the kafka broker first: + +``` +Starting the pipeline broker ... + +Executing 'docker compose -f compose.local.yaml up --build -d --remove-orphans kafka-broker' + + Container simplebroker-kafka-broker-1 Creating + Container simplebroker-kafka-broker-1 Created + Container simplebroker-kafka-broker-1 Starting + Container simplebroker-kafka-broker-1 Started +``` + +#### Updating the topics + +After the broker has started, the topics created automatically with the configuration provided in `quix.yaml`. + +``` +Updating topics ... + +✓ Created topic: f1-data +✓ Created topic: hard-braking +✓ No topics to update + +✓ Started the pipeline broker +``` + +!!! tip + Use the options `--skip-topics` or `-s` if you don't want to skip the topic creation. + +Next, it executes `docker compose compose.local.yaml up --build -d --remove-orphans` to build and run the your application Docker containers: ```text Executing 'docker compose up --build -d --remove-orphans' @@ -62,35 +92,43 @@ Containers Started ``` !!! tip - Using the `--update` option will perform the same actions as running [`quix pipeline update`](update.md) before generating the `compose.yaml` file. - + Using the `--update` option will perform the same actions as running [`quix pipeline update`](update.md) before generating the `compose.local.yaml` file. + +Finally you will see your applications running in docker and you will have visual way to manage and view your pipeline topics: + +``` +✓ Open http://localhost:8080 to manage your pipeline broker +``` -#### Generated `compose.yaml` File Overview +#### Generated `compose.local.yaml` File Overview -The `compose.yaml` file configures the services in your pipeline. Here's an overview of what will be generated: +The `compose.local.yaml` file configures the services in your pipeline. Here's an overview of what will be generated: - **demo-data-source** and **event-detection-transformation**: - - **volumes**: Mounts a null file to `.env`, effectively ignoring your local `.env` folder. - - **build**: Specifies the context directory and Dockerfile. - - **environment**: Injects environment variables for Kafka broker addresses and data inputs/outputs. These match the settings in `quix.yaml`. - - ```yaml - environment: - input: f1-data - output: hard-braking - Quix__Broker__Address: kafka-broker:9092 - ``` + - **volumes**: Mounts a null file to `.env`, effectively ignoring your local `.env` folder. + - **build**: Specifies the context directory and Dockerfile. + - **environment**: Injects environment variables for Kafka broker addresses and data inputs/outputs. These match the settings in `quix.yaml`. + + ```yaml + environment: + input: f1-data + output: hard-braking + Quix__Broker__Address: kafka-broker:9092 + ``` + + These environment variables are injected into the container at runtime: - These environment variables are injected into the container at runtime: + - **input**: Specifies the input data stream (`f1-data`). - - **input**: Specifies the input data stream (`f1-data`). + - **output**: Specifies the output data stream (`hard-braking`). - - **output**: Specifies the output data stream (`hard-braking`). + - **Quix__Broker__Address**: Specifies the address of the Kafka broker (`kafka-broker:9092`). - - **Quix__Broker__Address**: Specifies the address of the Kafka broker (`kafka-broker:9092`). - **kafka-broker and console**: - - **kafka-broker**: Installs and configures a Redpanda Kafka broker for you. - - **console**: Provides a management interface for interacting with the Kafka broker, including necessary environment configurations. -For more details on the `compose.yaml` file and its configurations, refer to the [official Docker Compose documentation](https://docs.docker.com/compose/compose-file/). \ No newline at end of file + - **kafka-broker**: Installs and configures a Redpanda Kafka broker for you. + + - **console**: Provides a management interface for interacting with the Kafka broker, including necessary environment configurations. + +For more details on the `compose.local.yaml` file and its configurations, refer to the [official Docker Compose documentation](https://docs.docker.com/compose/compose-file/). \ No newline at end of file diff --git a/docs/Reference/pipeline/update.gen.md b/docs/Reference/pipeline/update.gen.md index 48e6c58..52ed31b 100644 --- a/docs/Reference/pipeline/update.gen.md +++ b/docs/Reference/pipeline/update.gen.md @@ -4,7 +4,7 @@ Update 'quix.yaml' with the new local applications and update the variables of the existing deployments -**Aliases:** `update` +**Aliases:** `update`, `upd` **Usage:** @@ -17,6 +17,7 @@ quix pipeline update [options] - `--directory ` : Base directory (defaults to current directory) - `--existing, --only-existing` : Update only existing deployments - `--new, --only-new` : Add only the applications that are not currently deployed -- `--current, --only-current` : Update only the deployments of the current application folder. (Fails if not in an 'app.yaml' is not in the current directory) -- `--reset` : Removes all deployments from the 'quix.yaml' file. This operation is performed before any updates or additions to ensure a clean state +- `-a, --all` : Forces the update of all applications regardless of the current directory +- `--new-variables, --only-new-variables` : Only add new variables to the deployments without updating rest of the variables +- `--reset` : Removes all deployments and topics from the 'quix.yaml' file. This operation is performed before any updates or additions to ensure a clean state diff --git a/docs/Reference/pipeline/view.md b/docs/Reference/pipeline/view.md index 474ef1e..4c70c61 100644 --- a/docs/Reference/pipeline/view.md +++ b/docs/Reference/pipeline/view.md @@ -4,21 +4,20 @@ The `quix pipeline view` command generates a visual representation of your pipeline using a mermaid diagram. When executed, it creates or updates the `pipeline.md` and `quix-pipeline.html` files in your project directory. -These files contain mermaid code that visualizes your pipeline's structure and data flow. The command can open these files in VS Code or your default browser, providing an interactive and up-to-date view of your pipeline configuration. This helps you easily visualize and document your pipeline setup. +These files contain mermaid code that visualizes your pipeline's structure and data flow. The command can open these files in your default browser, providing an interactive and up-to-date view of your pipeline configuration. This helps you easily visualize and document your pipeline setup. ## Example Usage ``` $ quix pipeline view ``` +This command updates the pipeline.md and a temporary quix-pipeline.html file, which opens automatically in your browser. This allows you to view and interact with your pipeline documentation directly. This command will attempt to open the generated `quix-pipeline.html` file in your default browser. If it does not open automatically, you will see a message with a file path like this: -``` -Open file:///path/to/your/temp/quix-pipeline.html in your browser if the link didn't open in your browser. -``` +Visual Studio Code can display the pipeline diagram if you install the [Markdown Mermaid](https://marketplace.visualstudio.com/items?itemName=bierner.markdown-mermaid) extension. -Simply click the provided link to access the pipeline documentation. +Similarly, GitHub will automatically render the `pipeline.md` file with the included mermaid code, allowing you to visualize your pipeline directly on the platform. The `quix-pipeline.html` file will render a graphical representation of the pipeline in your default web browser: diff --git a/docs/Reference/run.gen.md b/docs/Reference/run.gen.md new file mode 100644 index 0000000..aa7c64d --- /dev/null +++ b/docs/Reference/run.gen.md @@ -0,0 +1,28 @@ +# run + +## Overview + +Run your applications locally + +**Aliases:** `run` + +**Usage:** + +``` +quix run [...] [options] +``` + +**Arguments:** + +- `` : Custom command to execute + +**Options:** + +- `--workspace-id ` : The environment to use if your SDK broker configuration is set to Quix Cloud +- `--directory ` : Base directory (defaults to current directory) +- `-u, --update` : Updates the application's '.env' file with the variables from 'quix.yaml' before running +- `-i, --intercept` : Stop the deployment while this command is running and then start it again +- `-s, --stop` : Stop the deployment while this command is running and without starting it again +- `-d, --deployment ` : Specify the deployment to intercept or update its variable values +- `-v, --verbose, --verbosity` : Display the verbose output + diff --git a/docs/Reference/run.md b/docs/Reference/run.md new file mode 100644 index 0000000..154d2e2 --- /dev/null +++ b/docs/Reference/run.md @@ -0,0 +1,75 @@ +{% include-markdown './run.gen.md' %} + +## How It Works + +The `quix run` command allows you to execute your Quix applications locally, providing a consistent environment for testing, debugging, and development. This command ensures that your application behaves as it would in production, but within the safety of your local environment. + +### Key Features: + +- **Environment Synchronization**: When using the `--update` option, the command ensures that your `.env` file is synchronized with your `quix.yaml` configuration. This guarantees that all environment variables are up-to-date, reflecting any recent changes. + +- **Deployment Control**: The `--intercept` option temporarily halts a live deployment while you run your application locally, ensuring that your tests are not affected by ongoing production activities. If necessary, the `--stop` option can fully stop the deployment, allowing for isolated testing until you're ready to restart. + +- **Custom Commands**: Beyond running the default application, `quix run` allows you to specify custom commands, offering flexibility in how you test and debug your code. + +This command is essential for maintaining a controlled and consistent environment across different stages of development, ensuring that your application functions correctly before being deployed to production. + +## Example Usage + +### Running Locally with Default Settings + +To run your application using the existing configurations in your `.env` file, simply execute: + +```bash +quix run +``` + +This will use the current directory and apply the environment variables defined in your `.env` file. + +!!! tip + Before running, you can ensure that your `.env` file is up-to-date by using the `--update` option. This will synchronize your environment variables with the latest `quix.yaml` configurations: + + ```bash + quix run --update + ``` + +### Targeting Specific Deployments + +In projects with multiple deployments, you can target a specific deployment using the `--deployment` option: + +```bash +quix run --deployment my-deployment +``` + +This command focuses on the specified deployment, allowing you to test or update it individually. + +### Controlling Deployments + +For situations where you need to test locally without interference from live deployments, the `--intercept` option can be used: + +```bash +quix run --intercept +``` + +This temporarily pauses the live deployment while your local version runs. Once testing is complete, the deployment resumes automatically. + +To completely stop a deployment and keep it down for as long as necessary: + +```bash +quix run --stop +``` + +This halts the deployment, ensuring it remains inactive until manually restarted. + +!!! note + These options are particularly useful during debugging when you need to ensure that the live environment does not interfere with your local tests. + +### Running a Custom Command + +To execute a specific command, such as running a different Python script: + +```bash +quix run python otherfile.py +``` + +This command runs the `otherfile.py` script, utilizing the environment variables defined in your `.env` file. diff --git a/docs/Reference/sdk/broker/current.gen.md b/docs/Reference/sdk/broker/current.gen.md new file mode 100644 index 0000000..d9e0642 --- /dev/null +++ b/docs/Reference/sdk/broker/current.gen.md @@ -0,0 +1,14 @@ +# sdk broker current + +## Overview + +Get the SDK broker configuration + +**Aliases:** `current` + +**Usage:** + +``` +quix sdk broker current [options] +``` + diff --git a/docs/Reference/sdk/broker/current.md b/docs/Reference/sdk/broker/current.md new file mode 100644 index 0000000..ef716c6 --- /dev/null +++ b/docs/Reference/sdk/broker/current.md @@ -0,0 +1 @@ +{% include-markdown './current.gen.md' %} \ No newline at end of file diff --git a/docs/Reference/sdk/broker/index.gen.md b/docs/Reference/sdk/broker/index.gen.md index bb78b55..d460b19 100644 --- a/docs/Reference/sdk/broker/index.gen.md +++ b/docs/Reference/sdk/broker/index.gen.md @@ -15,6 +15,7 @@ quix sdk broker [command] [options] **Commands:** - [cloud](cloud.md) : Set the SDK broker configuration to point to a 'Quix Cloud' broker +- [current](current.md) : Get the SDK broker configuration - [local](local.md) : Set the SDK broker configuration to point to a 'Local' broker (default: pipeline broker) - [set](set.md) : Set a local broker address for the SDK broker configuration - [update](update.md) : Update '.env' files of all the applications with the current SDK broker configuration diff --git a/docs/Reference/sdk/broker/set.md b/docs/Reference/sdk/broker/set.md index 2707789..fd39b19 100644 --- a/docs/Reference/sdk/broker/set.md +++ b/docs/Reference/sdk/broker/set.md @@ -3,7 +3,7 @@ ## How It Works -The `quix sdk broker set` command allows you to specify the address for the SDK broker configuration for the current context. By default, this sets the broker to `localhost:19092`, which is the default pipeline broker address that can be started with the [`quix pipeline broker up`](../../pipeline/broker/up.md) command. +The `quix sdk broker set` command allows you to specify the address for the SDK broker configuration for the current context. By default, this sets the broker to `localhost:19092`, which is the default pipeline broker address that can be started with the [`quix broker up`](../../broker/up.md) command. However, using this default broker address is not required. You can change it to suit your development setup. For instance, if you already have a Kafka broker running locally, you might set the address to `localhost:9092`. Alternatively, you can set it to any other Kafka broker address, such as `kafka.example.com:9092`. diff --git a/docs/Reference/sdk/broker/update.gen.md b/docs/Reference/sdk/broker/update.gen.md index 6a34f13..16fa27f 100644 --- a/docs/Reference/sdk/broker/update.gen.md +++ b/docs/Reference/sdk/broker/update.gen.md @@ -4,7 +4,7 @@ Update '.env' files of all the applications with the current SDK broker configuration -**Aliases:** `update` +**Aliases:** `update`, `upd` **Usage:** diff --git a/docs/Reference/sdk/broker/update.md b/docs/Reference/sdk/broker/update.md index f93a35f..e069cea 100644 --- a/docs/Reference/sdk/broker/update.md +++ b/docs/Reference/sdk/broker/update.md @@ -1 +1,45 @@ -{% include-markdown './update.gen.md' %} \ No newline at end of file +{% include-markdown './update.gen.md' %} + +## How It Works + +The `quix sdk broker update` command ensures that the `.env` files for all your applications are synchronized with the current SDK broker configuration. By updating these files, the command guarantees that every application is correctly configured to connect with the latest local broker settings. + +Upon execution, the command scans each application's `.env` file and updates the SDK configuration to reflect the current local broker. If an application's `.env` file is already up-to-date, it remains unchanged, ensuring stability and consistency across your development environment. + +!!! note + The `quix sdk broker update` command specifically updates the broker configuration within the `.env` files. Other environment variables remain unchanged. To update all variables according to your `quix.yaml` configuration, use the [`quix init --update`](../../init.md) command. + +## Example Usage + +### Running the Update + +To synchronize the broker configuration across all your applications, use: + +```bash +quix sdk broker update +``` + +After running this command, you will see which applications were updated: + +```bash +quix sdk broker update +! Application app1: 'path\to\app1\.env' was not updated +! Application app2: 'path\to\app2\.env' was not updated +✓ Application app3: 'path\to\app3\.env' was updated +``` + +### Before and After + +Consider the following example of an `.env` file before running the update: + +```dotenv title=".env before" +Quix__Broker__Address=localhost:9092 +``` + +After running `quix sdk broker update`, the file is updated to reflect the new broker address: + +```dotenv title=".env after" +Quix__Broker__Address=localhost:19092 +``` + +This update ensures that all your applications are correctly pointing to the current broker, maintaining consistent and reliable operation across your local development environment. diff --git a/docs/Reference/status.md b/docs/Reference/status.md index 26fc0d0..babe0f4 100644 --- a/docs/Reference/status.md +++ b/docs/Reference/status.md @@ -9,10 +9,9 @@ It performs the following checks: 1. **Authentication Status**: Verifies if you are logged into Quix Cloud and displays your user information if logged in. 2. **Current Context**: Shows details about the current context, including the name, URL, environment ID, and local broker settings. 3. **Git Repository**: Identifies and displays the root directory of your Git repository. -4. **Installed IDEs**: Checks for installed IDEs (VS Code and PyCharm) and reports any issues. -5. **Docker Installation**: Verifies if Docker is installed and reports the status. -6. **Git Installation**: Confirms if Git is installed and reports the status. -7. **Local Pipeline Status**: Checks and reports the status of your local pipeline. +4. **Docker Installation**: Verifies if Docker is installed and reports the status. +5. **Git Installation**: Confirms if Git is installed and reports the status. +6. **Local Pipeline Status**: Checks and reports the status of your local pipeline. This command is useful for quickly assessing the overall health and configuration of your development environment. @@ -33,13 +32,12 @@ If you are not logged into Quix Cloud, the status command will indicate this and User: ! Not logged in to Quix Cloud Current context: default (https://portal-api.platform.quix.io) Default environment: ! Not set - SDK broker configuration: Local (localhost:19092) -✓ Docker installed + SDK broker configuration: Local (localhost:19092) ! Local Pipeline Status: Not Running -✓ Git installed +! Local Broker Status: Not Running +✓ Docker detected +✓ Git detected Git Root: \path\to\your\repo -✓ VS Code installed -✓ PyCharm installed ``` ### Logged In with Default Environment @@ -56,12 +54,11 @@ If you are logged into Quix Cloud and have a default environment set, the status Current context: default (https://portal-api.platform.quix.io) Default environment: example-org-project-env SDK Broker configuration: Local (localhost:19092) -✓ Docker installed ! Local Pipeline Status: Not Running -✓ Git installed +! Local Broker Status: Not Running +✓ Docker detected +✓ Git detected Git Root: \path\to\your\repo -✓ VS Code installed -✓ PyCharm installed ``` !!! tip @@ -69,12 +66,21 @@ If you are logged into Quix Cloud and have a default environment set, the status ### Pipeline Status -The status of your local pipeline is also displayed. If the pipeline is running but not all services are up, it will be indicated: +The status of your local pipeline is also displayed: ```text -✗ Local Pipeline Status: Running (3/4) +✓ Local Pipeline Status: Running (2/2) +✓ Local Broker Status: Running (localhost:19092) +✓ Local Broker GUI: Running (http://localhost:8080) ``` +If the pipeline is running but not all services are up, it will be indicated: + +```text +✗ Local Pipeline Status: Running (1/2) +``` + + !!! tip Use the [`quix pipeline status`](pipeline/status.md) command to get detailed information about which services in your local pipeline are running and which are not. diff --git a/docs/Reference/update.gen.md b/docs/Reference/update.gen.md index a94aafe..bfac428 100644 --- a/docs/Reference/update.gen.md +++ b/docs/Reference/update.gen.md @@ -4,7 +4,7 @@ Update the version of the CLI -**Aliases:** `update` +**Aliases:** `update`, `upd` **Usage:** diff --git a/docs/cli-commands-summary.md b/docs/cli-commands-summary.md new file mode 100644 index 0000000..0524402 --- /dev/null +++ b/docs/cli-commands-summary.md @@ -0,0 +1,73 @@ +# Quix CLI Commands Summary + +This quick reference guide lists common use cases for the Quix CLI with the corresponding commands. Use this guide to quickly find and execute the commands you need while developing Quix data pipelines locally. + +## Core commands + +| Use Case | Command | +| ---------------------------------------------------------------------------------- | ---------------------- | +| [Update application `.env` files from `quix.yaml`](./Reference/init.md) | `quix init --update` | +| [Run applications with the variables from `.env`](./Reference/run.md) | `quix run` | +| [Update pipeline configuration](./Reference/pipeline/update.md) | `quix pipeline update` | +| [Bring pipeline up in Docker](./Reference/pipeline/up.md) | `quix pipeline up` | +| [Get a global status check of the CLI settings](./Reference/status.md) | `quix status` | +| [Display the current status of the local pipeline](./Reference/pipeline/status.md) | `quix pipeline status` | +| [Push local changes to Git and sync with Quix Cloud](./Reference/pipeline/sync.md) | `quix pipeline sync` | + +## Initialization + +| Use Case | Command | +| ---------------------------------------------------------------------------------- | -------------------------- | +| [Initialize a new Quix project](./Reference/init.md) | `quix init` | +| [Initialize with DevContainer support](./Reference/init.md) | `quix init --devcontainer` | +| [Initialize a project updating `.env` files from `quix.yaml`](./Reference/init.md) | `quix init --update` | + +## Application Management + +| Use Case | Command | +| -------------------------------------------------------------------------- | -------------------- | +| [Create a new application](./Reference/apps/create.md) | `quix app add` | +| [Edit the variables of an application](./Reference/apps/edit.md) | `quix app edit` | +| [List applications in the project](./Reference/apps/list.md) | `quix app list` | +| [Add a variable to an application](./Reference/apps/variables/create.md) | `quix app vars add` | +| [Edit the variables of an application](./Reference/apps/variables/edit.md) | `quix app vars edit` | + +## Pipeline Management + +| Use Case | Command | +| ------------------------------------------------------------------------------------------ | --------------------------- | +| [Update pipeline from current applications configurations](./Reference/pipeline/update.md) | `quix pipeline update` | +| [View pipeline graph](./Reference/pipeline/view.md) | `quix pipeline view` | +| [Bring pipeline up in Docker](./Reference/pipeline/up.md) | `quix pipeline up` | +| [Bring pipeline up in Docker and update the configuration](./Reference/pipeline/up.md) | `quix pipeline up --update` | +| [Bring pipeline down in Docker](./Reference/pipeline/down.md) | `quix pipeline down` | +| [Start a deployment in Docker](./Reference/pipeline/start.md) | `quix pipeline start` | +| [Stop a deployment in Docker](./Reference/pipeline/stop.md) | `quix pipeline stop` | +| [Read the logs from the local pipeline](./Reference/pipeline/logs.md) | `quix pipeline logs` | +| [Push local changes to Git and sync with Quix Cloud](./Reference/pipeline/sync.md) | `quix pipeline sync` | + +## Broker Management + +| Use Case | Command | +| ------------------------------------------------------------------------------------------ | ------------------------ | +| [Start local pipeline broker](./Reference/broker/up.md) | `quix broker up` | +| [Stop local pipeline broker](./Reference/broker/down.md) | `quix broker down` | +| [List all the topics from your pipeline broker](./Reference/broker/topics/list.md) | `quix broker topic list` | +| [Read the content of a topic from your pipeline broker](./Reference/broker/topics/read.md) | `quix broker topic read` | + +## SDK Broker Configuration + +| Use Case | Command | +| -------------------------------------------------------------------------------- | ----------------------------------------------- | +| [Set SDK broker to custom local address](./Reference/sdk/broker/set.md) | `quix sdk broker set --enable` | +| [Set SDK broker to Quix Cloud](./Reference/sdk/broker/cloud.md) | `quix sdk broker cloud` | +| [Set SDK broker to default local configuration](./Reference/sdk/broker/local.md) | `quix sdk broker local` | + +## Running Applications + +| Use Case | Command | +| --------------------------------------------------------------------------- | ---------------------- | +| [Run with the variables from `.env`](./Reference/run.md) | `quix run` | +| [Update the `.env` variables from the pipeline and run](./Reference/run.md) | `quix run --update` | +| [Run and stop deployed version of the application](./Reference/run.md) | `quix run --stop` | +| [Run and intercept deployed version of the application](./Reference/run.md) | `quix run --intercept` | \ No newline at end of file diff --git a/docs/cli-local-debug.md b/docs/cli-local-debug.md new file mode 100644 index 0000000..36999fb --- /dev/null +++ b/docs/cli-local-debug.md @@ -0,0 +1,425 @@ +# Developing QuixStreams Applications Locally with Quix CLI + +Debugging an application locally using `quix run` has some nuances that need to be properly explained. This tutorial will guide you through the process of debugging an application of an existing Quix data pipeline using a local broker and the Quix CLI. We'll cover prerequisites, best practices, and common issues. + +## Prerequisites + +This tutorial assumes that you have read the [Quickstart](./cli-quickstart.md) and installed the dependencies: + +- [Docker Desktop](https://docs.docker.com/engine/install/){target=_blank} +- [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git){target=_blank} +- [Python](https://www.python.org/){target=_blank} + +!!! tip + If you want to start developing after cloning a repository, run: + + ```bash + quix init + ``` + + This will generate all the necessary files to get you started. + +## Developing with Quix CLI + +### Step 1: Setting Up the Broker + +!!! note + If you followed all the steps from the [Quickstart](cli-quickstart.md) tutorial, your local pipeline broker is already up and running. + +Depending on your setup, you can choose to use your local pipeline broker, another local broker, or the Quix Cloud broker. Here are the steps to configure each: + +=== "Local Pipeline Broker" + + To use your local pipeline broker, start it using: + + ```bash + quix broker up + ``` + + This will spin up a local broker running on `localhost:19092`. + + !!! question "How to ensure that your local application is pointing to your local pipeline broker?" + + The SDK Broker configuration is the broker you use when you run the application locally outside the pipeline. + + Verify it by running: + + ```bash + quix status + ``` + + Look for this line: + + ``` + SDK Broker configuration: Local (localhost:19092) + ``` + + If you see something else and want a quick fix, run: + + ```bash + quix sdk broker set localhost:19092 --enable + ``` + +=== "Other Local Broker" + + If you want to use a different local broker, you can set the broker address manually. This is useful if you have another Kafka broker running on your local machine or elsewhere. To set a custom local broker address, run: + + ```bash + quix sdk broker set + ``` + + If you don't want to run the interactive version, you can specify the parameters and enable it by running: + + ```bash + quix sdk broker set localhost:9092 --enable + ``` + + !!! tip + If you had previously set your broker to Quix Cloud, you will need to switch it back to the local broker. Run: + + ```bash + quix sdk broker local + ``` + + For more details on the `set` command, refer to the [set broker documentation](./Reference/sdk/broker/set.md). + + !!! warning + This doesn't change your local pipeline broker, just the broker you use when you run the application locally outside the pipeline. + +=== "Quix Cloud Broker" + + To use the Quix Cloud broker, set the SDK broker configuration to Quix Cloud. This will configure your environment to use the Quix Cloud infrastructure for debugging: + + ```bash + quix sdk broker cloud + ``` + + This configuration is ideal if you want to test your application in an environment that closely resembles production. + + For more details on the `cloud` command, refer to the [cloud broker documentation](./Reference/sdk/broker/cloud.md). + + !!! warning + This doesn't change your local pipeline broker, just the broker you use when you run the application locally outside the pipeline. + +--- + +For a comprehensive overview of all SDK broker commands, refer to the [SDK broker documentation](./Reference/sdk/broker/index.md). + +### Step 2: Preparing Your Local Environment + +To ensure your application runs smoothly across different environments, it's essential to create a virtual environment for each application you develop. This isolates the dependencies for each application, avoiding conflicts between them. + +**Create a Virtual Environment** + + Navigate to your application's directory and run one of the following command: + + ```bash + python -m venv .venv + ``` + +!!! note + Depending on your system configuration, the command might differ slightly. Use one of these alternatives if needed: + + ```bash + python3 -m venv .venv + ``` + + or + + ```bash + py -m venv .venv + ``` + + This command sets up a virtual environment named `.venv` inside your application's directory. + +**Activate the Virtual Environment** + +After creating the virtual environment, you need to activate it to start using it for your application. The activation command varies by operating system: + +=== "Windows" + + ```powershell + .\.venv\Scripts\activate + ``` + +=== "Linux" + + ```bash + source .venv/bin/activate + ``` + +=== "macOS" + + ```bash + source .venv/bin/activate + ``` + +**Install Dependencies** + + Once the virtual environment is active, you can install your application's dependencies from the `requirements.txt` file. Run the following command: + + ```bash + pip install -r requirements.txt + ``` + + This ensures all necessary packages are installed specifically for this application. + +!!! tip "Using DevContainers" + If you prefer using DevContainers for an isolated development environment, you can set them up with the `--devcontainer` option: + + ```bash + quix init --devcontainer + ``` + + The `devcontainer.json` file is automatically generated. It reuses your existing Dockerfile, installs Python and Docker extensions, and sets up the Quix CLI within the dev container. This setup ensures that your development environment matches your production environment, reducing the "it works on my machine" issues. + + By using DevContainers, you ensure that your development environment is consistent, isolated, and reproducible. + + Your IDE will take care of the rest. + +### Step 3: Understanding Application Variables + +Let's take a look at the `.env` file generated for the `event-detection-transformation` application: + +!!! note + If you didn't follow the [Quickstart](cli-quickstart.md) you can get to this point by running: + + ```bash + quix app create event-detection-transformation + ``` + + or, you can get this `.env` file updated at any point for any Quix applications by running: + + ```bash + quix init --update + ``` + +```.env title=".env" +# ====================================================== +# 🚀 Quix CLI guidelines 🚀 +# ====================================================== +# To update the quix.yaml from this .env file, use: +# 🔄 quix pipeline update +# +# To update the .env file from the quix.yaml file, use: +# 🔄 quix init --update +# ====================================================== + +### Quix SDK Configuration ### +# Configuration settings for QuixStreams +Quix__Broker__Address=localhost:19092 + +### Input Topics ### +# Define the input topics used by the application +input=f1-data + +### Output Topics ### +# Define the output topics used by the application +output=hard-braking + +### Secrets ### +# Sensitive information such as API keys and passwords + +### Free Text ### +# Add any free text or comments here + +### Untracked Variables ### +# Variables that are not tracked by Quix CLI +``` +Each section of the .env file is dedicated to different types of variables used in Quix, such as those for rendering the data pipeline or managing secrets. + +You can create and remove variables for your local applications here at your discretion, and they will be integrated into the system as needed. + +### Step 4: Running Your Code with `quix run` + +By using the [`quix run`](./Reference/run.md) command, the `.env` variables are injected into your Python code as environment variables so you can read the values like this: + +```python title="main.py" +import os +... +input_topic = app.topic(os.environ["input"]) +output_topic = app.topic(os.environ["output"]) +``` + +!!! tip "Working with a local pipeline" + The `--stop` or `--intercept` options can be used to stop or pause the deployed version of your application during the execution of this command. + +!!! note + The sample provides support for the `load_dotenv` library, which is useful if you want to run Python directly or if your IDE doesn't automatically load `.env` files. + + ```python + from dotenv import load_dotenv + load_dotenv() + ``` + +### Step 5: Updating Your Pipeline from Your Local Variables + +Now, you can create new variables by simply adding them to the appropriate section of the `.env` file: + +```.env title=".env" hl_lines="28" +# ====================================================== +# 🚀 Quix CLI guidelines 🚀 +# ====================================================== +# To update the quix.yaml from this .env file, use: +# 🔄 quix pipeline update +# +# To update the .env file from the quix.yaml file, use: +# 🔄 quix init --update +# ====================================================== + +### Quix SDK Configuration ### +# Configuration settings for QuixStreams +Quix__Broker__Address=localhost:19092 + +### Input Topics ### +# Define the input topics used by the application +input=f1-data + +### Output Topics ### +# Define the output topics used by the application +output=hard-braking + +### Secrets ### +# Sensitive information such as API keys and passwords + +### Free Text ### +# Add any free text or comments here +variable=my-value + +### Untracked Variables ### +# Variables that are not tracked by Quix CLI +``` + +To move your local variables to your pipeline, run: + +```bash +quix pipeline update +``` + +Your `quix.yaml` file will now contain the new variable you created: + +```yaml title="quix.yaml" hl_lines="18-21" + - name: event-detection-transformation + application: event-detection-transformation + version: latest + deploymentType: Service + resources: + cpu: 200 + memory: 800 + replicas: 1 + variables: + - name: input + inputType: InputTopic + required: false + value: f1-data + - name: output + inputType: OutputTopic + required: false + value: hard-braking + - name: variable + inputType: FreeText + required: false + value: my-value +``` + +### Step 6: Updating Your Local Variables from Your Pipeline + +If your `quix.yaml` file contains values you want to use locally, run: + +```bash +quix init --update +``` + +!!! tip + You can also perform this operation when you run an application: + + ```bash + quix run --update + ``` + +This is helpful when you merge remote changes from git and the variable values have changed. + +```yaml title="quix.yaml" hl_lines="13 17 21" + - name: event-detection-transformation + application: event-detection-transformation + version: latest + deploymentType: Service + resources: + cpu: 200 + memory: 800 + replicas: 1 + variables: + - name: input + inputType: InputTopic + required: false + value: deployed-input-topic + - name: output + inputType: OutputTopic + required: false + value: deployed-output-topic + - name: variable + inputType: FreeText + required: false + value: a-pipeline-value +``` + +```.env title=".env" hl_lines="17 21 28" +# ====================================================== +# 🚀 Quix CLI guidelines 🚀 +# ====================================================== +# To update the quix.yaml from this .env file, use: +# 🔄 quix pipeline update +# +# To update the .env file from the quix.yaml file, use: +# 🔄 quix init --update +# ====================================================== + +### Quix SDK Configuration ### +# Configuration settings for QuixStreams +Quix__Broker__Address=localhost:19092 + +### Input Topics ### +# Define the input topics used by the application +input=deployed-input-topic + +### Output Topics ### +# Define the output topics used by the application +output=deployed-output-topic + +### Secrets ### +# Sensitive information such as API keys and passwords + +### Free Text ### +# Add any free text or comments here +variable=a-pipeline-value + +### Untracked Variables ### +# Variables that are not tracked by Quix CLI +``` + +## Other useful commands for Local Development + +There are several other CLI commands that you may find very useful during your local development. Here are some of the most relevant ones: + +- [quix pipeline view](./Reference/pipeline/view.md): Create and preview a mermaid diagram of the pipeline. +- [quix pipeline up](./Reference/pipeline/up.md): Run your pipeline locally using docker compose +- [quix pipeline status](./Reference/pipeline/status.md): Display the current status of the local pipeline +- [quix pipeline logs -f -n 10](./Reference/pipeline/logs.md): View output from deployments of the local pipeline in realtime +- [quix pipeline sync](./Reference/pipeline/sync.md): Synchronize your local pipeline to Quix Cloud +- [quix broker topics read](./Reference/broker/topics/read.md): Read the messages from your pipeline topics + +You should also check our [CLI Commands Summary](./cli-commands-summary.md), where you'll find the most useful commands for developing QuixStreams data pipelines locally. + +## Next steps + +
+ +- __Time to level up?__ + + --- + + Deploy your local pipeline to the Cloud, for scalability, observability, and even more Quix magic. + + [Deploy to Quix Cloud :octicons-arrow-right-24:](../quix-cloud/quickstart.md) + +
diff --git a/docs/cli-local-secrets.md b/docs/cli-local-secrets.md new file mode 100644 index 0000000..f58c43d --- /dev/null +++ b/docs/cli-local-secrets.md @@ -0,0 +1,274 @@ +# Managing Secrets in Quix CLI + +In any development environment, securing sensitive information like API tokens is essential. Quix CLI provides robust tools for managing these secrets within your data pipelines. This tutorial will walk you through the steps to securely add, update, and manage secrets in your Quix pipeline, ensuring that your sensitive data remains protected throughout the development process. + +Before diving into the details, make sure you've set up your local environment correctly. If you haven't already, complete the [Quickstart](./cli-quickstart.md) to get Quix CLI up and running. For detailed instructions on setting up your local development environment, refer to our [Local Development](./cli-local-debug.md) tutorial. + +## Step 1: Add the Secret to Your `.env` File + +Start by adding your secret as an environment variable in the `.env` file. Here’s how to securely store an API token: + +```dotenv title=".env" hl_lines="23" +# ====================================================== +# Quix CLI guidelines +# ====================================================== +# To update the quix.yaml from this .env file, use: +# quix pipeline update +# +# To update the .env file from the quix.yaml file, use: +# quix init --update +# ====================================================== + +### Quix SDK Configuration ### +# Configuration settings for QuixStreams +Quix__Broker__Address=localhost:19092 + +### Input Topics ### +# Define the input topics used by the application + +### Output Topics ### +# Define the output topics used by the application + +### Secrets ### +# Sensitive information such as API keys and passwords +api_secret_token=SECRET-API-TOKEN-VALUE + +### Free Text ### +# Add any free text or comments here + +### Untracked Variables ### +# Variables that are not tracked by Quix CLI +``` + +Adding the `api_secret_token` under the "Secrets" section ensures it is handled securely by the Quix CLI. + +## Step 2: Update Your Pipeline Configuration + +To integrate the new secret into your pipeline, run the following command: + +```bash +$ quix pipeline update +``` + +This command updates your pipeline configuration, incorporating the secret into the `quix.yaml` file and generating or updating the `.secrets` file: + +```text +Updating deployments of application starter-source +✓ starter-source → starter-source +✓ 'quix.yaml' was updated + +Updating secrets ... +✓ '.secrets' was updated + +✓ The pipeline view was updated +``` + +## Step 3: Verify the `quix.yaml` and `.secrets` Files + +After running the update command, verify that the `quix.yaml` file now includes the secret key. The key is linked to your secret and referenced by the relevant deployment: + +```yaml title="quix.yaml" hl_lines="21" +# Quix Project Descriptor +# This file describes the data pipeline and configuration of resources of a Quix Project. + +metadata: + version: 1.0 + +# This section describes the Deployments of the data pipeline +deployments: + - name: starter-source + application: starter-source + version: latest + deploymentType: Service + resources: + cpu: 200 + memory: 800 + replicas: 1 + variables: + - name: api_secret_token + inputType: Secret + required: false + secretKey: api_secret_token_key +``` + +Additionally, a `.secrets` file is generated or updated with the actual value associated with the secret key: + +```dotenv title=".secrets" hl_lines="7" +### Local Pipeline Secrets ### +# +# Stores secret values for local pipeline testing. +# Secrets here are referenced in quix.yaml using secret keys. +# +# Keep this file secure and out of version control. +api_secret_token_key=SECRET-API-TOKEN-VALUE +``` + +!!! tip + The `.secrets` file stores sensitive data securely for local development, keeping it separate from your application code. + +## Step 4: Reuse Secrets Across Multiple Deployments + +To use the same secret across multiple deployments, reference the same `secretKey` in different deployments within your `quix.yaml` file. For example: + +```yaml title="quix.yaml" hl_lines="21 35" +# Quix Project Descriptor +# This file describes the data pipeline and configuration of resources of a Quix Project. + +metadata: + version: 1.0 + +# This section describes the Deployments of the data pipeline +deployments: + - name: starter-source + application: starter-source + version: latest + deploymentType: Service + resources: + cpu: 200 + memory: 800 + replicas: 1 + variables: + - name: api_secret_token + inputType: Secret + required: false + secretKey: api_secret_token_key + + - name: other-deployment + application: other-application + version: latest + deploymentType: Service + resources: + cpu: 200 + memory: 800 + replicas: 1 + variables: + - name: a_different_variable_name + inputType: Secret + required: false + secretKey: api_secret_token_key +``` + +In this setup, both `starter-source` and `other-deployment` use the same secret key (`api_secret_token_key`), even though they reference it with different variable names (`api_secret_token` and `a_different_variable_name`). This ensures consistency across deployments and reduces redundancy. + +!!! tip + Reusing secret keys across deployments simplifies management and ensures all services have access to the necessary secrets. + +## Step 5: Deploying Secrets Locally + +When deploying your pipeline locally, it's crucial to verify that your secrets are correctly included in the configuration. The `quix pipeline up` command generates a `compose.local.yaml` file that outlines how your services are configured, including the injection of secrets. + +To ensure your secrets are properly handled, run the following command: + +```bash +$ quix pipeline up --dry-run +``` + +This generates the `compose.local.yaml` file without starting any containers, allowing you to inspect how secrets are managed: + +```text + Generating 'compose.local.yaml' + Generating deployment starter-source +✓ Generated deployment starter-source +✓ Generated 'compose.local.yaml' +``` + +In the `compose.local.yaml` file, your secrets and environment variables will be included in the environment section for each service: + +```yaml title="compose.local.yaml" hl_lines="9" +services: + starter-source: + volumes: + - /dev/null:/app/.env + build: + context: starter-source + dockerfile: dockerfile + environment: + api_secret_token: SECRET-API-TOKEN-VALUE + Quix__Broker__Address: kafka-broker:9092 + ... +``` + +By reviewing this file, you can confirm that your secrets are securely injected and your services are configured correctly before proceeding with a full deployment. This step ensures that your local deployment mirrors the intended configuration, including the correct handling of sensitive information. + +## Step 6: Ensure Secrets Are Excluded from Version Control + +It's crucial to ensure that all files containing secrets are excluded from version control. Running the `quix init` command will update your `.gitignore` file or create a new one if it doesn’t exist: + +```bash +$ quix init +✓ Created '.gitignore' with 5 entries: .env, compose.local.yaml, .secrets, .venv/, certificates/ +``` + +Your `.gitignore` file should now include entries to exclude sensitive files: + +```plaintext title=".gitignore" +.env +compose.local.yaml +.secrets +.venv/ +certificates/ +``` + +!!! warning + Always ensure that your `.secrets` and other sensitive files are excluded from version control. This is essential for maintaining the security of your pipeline. + +## Step 7: Update Local Secrets + +If you need to update the value of a secret locally, modify the `.secrets` file directly: + +```dotenv title=".secrets" hl_lines="7" +### Local Pipeline Secrets ### +# +# Stores secret values for local pipeline testing. +# Secrets here are referenced in quix.yaml using secret keys. +# +# Keep this file secure and out of version control. +api_secret_token_key=NEW-SECRET +``` + +To synchronize the updated secret with your `.env` file, run: + +```bash +quix init -u +``` + +The output will confirm the update: + +```text +Updating application starter-source +✓ starter-source → starter-source +``` + +Your `.env` file will now reflect the updated secret: + +```dotenv title=".env" hl_lines="23" +# ====================================================== +# Quix CLI guidelines +# ====================================================== +# To update the quix.yaml from this .env file, use: +# quix pipeline update +# +# To update the .env file from the quix.yaml file, use: +# quix init --update +# ====================================================== + +### Quix SDK Configuration ### +# Configuration settings for QuixStreams +Quix__Broker__Address=localhost:19092 + +### Input Topics ### +# Define the input topics used by the application + +### Output Topics ### +# Define the output topics used by the application + +### Secrets ### +# Sensitive information such as API keys and passwords +api_secret_token=NEW-SECRET + +### Free Text ### +# Add any free text or comments here + +### Untracked Variables ### +# Variables that are not tracked by Quix CLI +``` \ No newline at end of file diff --git a/docs/cli-quickstart.md b/docs/cli-quickstart.md index 12edd79..6e6a444 100644 --- a/docs/cli-quickstart.md +++ b/docs/cli-quickstart.md @@ -61,8 +61,8 @@ quix status View the output carefully to confirm you have Git and Docker installed: ``` -✓ Docker installed -✓ Git installed +✓ Docker detected +✓ Git detected ``` ## Step 3: Create a GitHub repository @@ -111,8 +111,6 @@ In your Git project directory, enter ```quix init```. This initializes your Quix quix init ``` - - If you look at the initial `quix.yaml` file you'll see the following: ``` yaml @@ -247,7 +245,7 @@ Run your pipeline in Docker: quix pipeline up ``` -This command generates a `compose.yaml` in the root of your project based on your `quix.yaml` file. +This command generates a `compose.local.yaml` in the root folder of your project based on your `quix.yaml` file. You'll see various console messages displayed in your terminal. When these have finished, then your deployed services are running in Docker. @@ -305,3 +303,17 @@ You're not connected to Quix Cloud [Deploy to Quix Cloud :octicons-arrow-right-24:](../quix-cloud/quickstart.md) + +Or + +
+ +- __Continue debugging your pipeline with the CLI__ + + --- + + Continue learning how to use Quix CLI to develop you pipeline locally. + + [Quix CLI local development :octicons-arrow-right-24:](./cli-local-debug.md) + +
diff --git a/docs/images/cli/pipeline-view-screenshot.png b/docs/images/cli/pipeline-view-screenshot.png new file mode 100644 index 0000000..d616e8d Binary files /dev/null and b/docs/images/cli/pipeline-view-screenshot.png differ diff --git a/mkdocs.yml b/mkdocs.yml index 14bcc83..6aef12a 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -54,19 +54,21 @@ plugins: nav: - 'Overview': 'overview.md' - 'Quickstart': 'cli-quickstart.md' -- 'Tutorials': - - 'Using the CLI with a template': 'cli-template.md' - - 'Using the CLI with GitHub Actions': 'cli-github-actions.md' +- 'Local Development': 'cli-local-debug.md' +- 'Local Secrets': 'cli-local-secrets.md' +- 'Commands Summary': 'cli-commands-summary.md' +- 'How-To guides': + - 'Using the CLI with GitHub Actions': 'How-To/cli-github-actions.md' - Reference: - Cloud Commands: - Reference/cloud/index.md - - cloud applications: - - Reference/cloud/applications/index.md - - cloud applications get: Reference/cloud/applications/get.md - - cloud applications list: Reference/cloud/applications/list.md - - cloud applications library: - - Reference/cloud/applications/library/index.md - - cloud applications library list: Reference/cloud/applications/library/list.md + - cloud apps: + - Reference/cloud/apps/index.md + - cloud apps get: Reference/cloud/apps/get.md + - cloud apps list: Reference/cloud/apps/list.md + - cloud apps library: + - Reference/cloud/apps/library/index.md + - cloud apps library list: Reference/cloud/apps/library/list.md - cloud deployments: - Reference/cloud/deployments/index.md - cloud deployments get: Reference/cloud/deployments/get.md @@ -81,6 +83,10 @@ nav: - cloud environments list: Reference/cloud/environments/list.md - cloud environments use: Reference/cloud/environments/use.md - cloud environments sync: Reference/cloud/environments/sync.md + - cloud environments tokens: + - Reference/cloud/environments/tokens/index.md + - cloud environments tokens get: Reference/cloud/environments/tokens/get.md + - cloud environments tokens rotate: Reference/cloud/environments/tokens/rotate.md - cloud organisations: - Reference/cloud/organisations/index.md - cloud organisations get: Reference/cloud/organisations/get.md @@ -113,27 +119,33 @@ nav: - cloud users current: Reference/cloud/users/current.md - cloud users list: Reference/cloud/users/list.md - Local Commands: - - applications: - - Reference/applications/index.md - - applications convert: Reference/applications/convert.md - - applications create: Reference/applications/create.md - - applications list: Reference/applications/list.md - - applications variables: - - Reference/applications/variables/index.md - - applications variables create: Reference/applications/variables/create.md - - applications variables delete: Reference/applications/variables/delete.md - - applications variables edit: Reference/applications/variables/edit.md - - applications variables export: Reference/applications/variables/export.md - - applications variables import: Reference/applications/variables/import.md - - applications variables list: Reference/applications/variables/list.md - - ide: Reference/ide.md + - apps: + - Reference/apps/index.md + - apps update: Reference/apps/update.md + - apps convert: Reference/apps/convert.md + - apps create: Reference/apps/create.md + - apps edit: Reference/apps/edit.md + - apps list: Reference/apps/list.md + - apps variables: + - Reference/apps/variables/index.md + - apps variables create: Reference/apps/variables/create.md + - apps variables delete: Reference/apps/variables/delete.md + - apps variables edit: Reference/apps/variables/edit.md + - apps variables export: Reference/apps/variables/export.md + - apps variables import: Reference/apps/variables/import.md + - apps variables list: Reference/apps/variables/list.md + - broker: + - Reference/broker/index.md + - broker down: Reference/broker/down.md + - broker up: Reference/broker/up.md + - broker topics: + - Reference/broker/topics/index.md + - broker topics list: Reference/broker/topics/list.md + - broker topics read: Reference/broker/topics/read.md + - broker topics update: Reference/broker/topics/update.md - init: Reference/init.md - pipeline: - Reference/pipeline/index.md - - pipeline broker: - - Reference/pipeline/broker/index.md - - pipeline broker down: Reference/pipeline/broker/down.md - - pipeline broker up: Reference/pipeline/broker/up.md - pipeline deployments: - Reference/pipeline/deployments/index.md - pipeline deployments create: Reference/pipeline/deployments/create.md @@ -142,16 +154,28 @@ nav: - pipeline deployments get: Reference/pipeline/deployments/get.md - pipeline deployments list: Reference/pipeline/deployments/list.md - pipeline down: Reference/pipeline/down.md + - pipeline logs: Reference/pipeline/logs.md + - pipeline start: Reference/pipeline/start.md - pipeline status: Reference/pipeline/status.md + - pipeline stop: Reference/pipeline/stop.md - pipeline sync: Reference/pipeline/sync.md - pipeline up: Reference/pipeline/up.md - pipeline update: Reference/pipeline/update.md - pipeline view: Reference/pipeline/view.md + - pipeline topics: + - Reference/pipeline/topics/index.md + - pipeline topics create: Reference/pipeline/topics/create.md + - pipeline topics delete: Reference/pipeline/topics/delete.md + - pipeline topics edit: Reference/pipeline/topics/edit.md + - pipeline topics get: Reference/pipeline/topics/get.md + - pipeline topics list: Reference/pipeline/topics/list.md + - run: Reference/run.md - sdk: - Reference/sdk/index.md - sdk broker: - Reference/sdk/broker/index.md - sdk broker cloud: Reference/sdk/broker/cloud.md + - sdk broker current: Reference/sdk/broker/current.md - sdk broker local: Reference/sdk/broker/local.md - sdk broker set: Reference/sdk/broker/set.md - sdk broker update: Reference/sdk/broker/update.md @@ -160,11 +184,6 @@ nav: - login: Reference/login.md - contexts: - Reference/contexts/index.md - - contexts broker: - - Reference/contexts/broker/index.md - - contexts broker cloud: Reference/contexts/broker/cloud.md - - contexts broker local: Reference/contexts/broker/local.md - - contexts broker set: Reference/contexts/broker/set.md - contexts create: Reference/contexts/create.md - contexts current: Reference/contexts/current.md - contexts list: Reference/contexts/list.md