Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chaos experiment user guide #230

Merged
merged 5 commits into from
Sep 28, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions website/docs/concepts/chaos-workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ spec:
name: pod-delete
app.kubernetes.io/part-of: litmus
app.kubernetes.io/component: chaosexperiment
app.kubernetes.io/version: 1.13.8
app.kubernetes.io/version: 3.0.0
spec:
definition:
scope: Namespaced
Expand Down Expand Up @@ -113,7 +113,7 @@ spec:
- update
- delete
- deletecollection
image: litmuschaos/go-runner:1.13.8
image: litmuschaos/go-runner:3.0.0
imagePullPolicy: Always
args:
- -c
Expand Down Expand Up @@ -141,7 +141,7 @@ spec:
name: pod-delete
app.kubernetes.io/part-of: litmus
app.kubernetes.io/component: experiment-job
app.kubernetes.io/version: 1.13.8
app.kubernetes.io/version: 3.0.0
container:
args:
- kubectl apply -f /tmp/pod-delete.yaml -n
Expand Down
6 changes: 3 additions & 3 deletions website/docs/user-guides/construct-workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ spec:
name: pod-delete
app.kubernetes.io/part-of: litmus
app.kubernetes.io/component: chaosexperiment
app.kubernetes.io/version: 1.13.8
app.kubernetes.io/version: 3.0.0
spec:
definition:
scope: Namespaced
Expand Down Expand Up @@ -162,7 +162,7 @@ spec:
- update
- delete
- deletecollection
image: litmuschaos/go-runner:1.13.8
image: litmuschaos/go-runner:3.0.0
imagePullPolicy: Always
args:
- -c
Expand Down Expand Up @@ -190,7 +190,7 @@ spec:
name: pod-delete
app.kubernetes.io/part-of: litmus
app.kubernetes.io/component: experiment-job
app.kubernetes.io/version: 1.13.8
app.kubernetes.io/version: 3.0.0
container:
args:
- kubectl apply -f /tmp/pod-delete.yaml -n
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ spec:
name: pod-delete
app.kubernetes.io/part-of: litmus
app.kubernetes.io/component: chaosexperiment
app.kubernetes.io/version: 1.13.8
app.kubernetes.io/version: 3.0.0
spec:
definition:
scope: Namespaced
Expand Down Expand Up @@ -118,7 +118,7 @@ spec:
- update
- delete
- deletecollection
image: litmuschaos/go-runner:1.13.8
image: litmuschaos/go-runner:3.0.0
imagePullPolicy: Always
args:
- -c
Expand Down Expand Up @@ -146,7 +146,7 @@ spec:
name: pod-delete
app.kubernetes.io/part-of: litmus
app.kubernetes.io/component: experiment-job
app.kubernetes.io/version: 1.13.8
app.kubernetes.io/version: 3.0.0
container:
args:
- kubectl apply -f /tmp/pod-delete.yaml -n
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,14 @@ sidebar_label: chaos-infrastructure-installation

## Prerequisites

- Before connecting a Chaos Delegate to the [ChaosCenter](../getting-started/resources.md#chaoscenter), learn about what is a [Chaos Delegate](../getting-started/resources.md#chaosagents) in Litmus.
- Before connecting a Chaos Infrastructure to the [ChaosCenter](../getting-started/resources.md#chaoscenter), learn about what is a [Chaos Infrastructure](../getting-started/resources.md#chaosagents) in Litmus.
- Make sure [litmusctl](../litmusctl/installation.md) is installed.

## Connecting Chaos Delegate
## Connecting Chaos Infrastructure

- Learn to [connect Chaos Delegate with non interactive mode using litmuctl](../litmusctl/litmusctl-usage.md)
- Learn to [connect Chaos Infrastructure with non interactive mode using litmuctl](../litmusctl/litmusctl-usage.md)

## Resource Requiremenets for Chaos Delegate plane components
## Resource Requiremenets for Chaos Infrastructure plane components

The Resource requests provided here have been estimated using data gathered manually through different methods -

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ spec:
name: pod-delete
app.kubernetes.io/part-of: litmus
app.kubernetes.io/component: chaosexperiment
app.kubernetes.io/version: 1.13.8
app.kubernetes.io/version: 3.0.0
spec:
definition:
scope: Namespaced
Expand Down Expand Up @@ -162,7 +162,7 @@ spec:
- update
- delete
- deletecollection
image: litmuschaos/go-runner:1.13.8
image: litmuschaos/go-runner:3.0.0
imagePullPolicy: Always
args:
- -c
Expand Down Expand Up @@ -190,7 +190,7 @@ spec:
name: pod-delete
app.kubernetes.io/part-of: litmus
app.kubernetes.io/component: experiment-job
app.kubernetes.io/version: 1.13.8
app.kubernetes.io/version: 3.0.0
container:
args:
- kubectl apply -f /tmp/pod-delete.yaml -n
Expand Down
Original file line number Diff line number Diff line change
@@ -1,43 +1,41 @@
---
id: delete-experiment
title: Delete a Chaos Scenario
sidebar_label: Delete Chaos Scenario
title: Delete a Chaos experiment
sidebar_label: Delete Chaos experiment
---

---

If required, it is possible to delete a chaos scenario schedule that you no longer wish to run against your application.
If required, it is possible to delete a chaos experiment schedule that you no longer wish to run against your application.

:::note
This also means that all the runs corresponding to that chaos scenario will also be deleted.
This also means that all the runs corresponding to that chaos experiment will also be deleted.
:::

## Before you begin

You can learn about the concept of chaos scenarios [here](../concepts/chaos-workflow.md) and how to schedule your first chaos scenario [here](schedule-experiment.md).
You can learn about the concept of chaos experiments [here](../concepts/chaos-workflow.md) and how to schedule your first chaos experiment [here](schedule-experiment.md).

## 1. Go to the chaos scenarios sections
## 1. Go to the chaos experiments sections

In the `Chaos Scenario` page, go to the `Schedules` tab and click on the options menu for the specific schedule you wish to delete:
In the `Chaos experiment` page, go to the specific experiment you wish to delete:

<img src={require('../assets/user-guides/injecting-fault/delete-workflow/step-1.png').default} width="1000" height="500" />

## 2. Click on the `Delete chaos scenario` option
## 2. Click on the `Delete experiment` option

After opening the options menu and clicking on the `Delete chaos scenario` option, you'll see a prompt in order to confirm your action. Please ensure that you want to delete the selected chaos scenario and click the `Delete` button:
After opening the options menu and clicking on the `Delete experiment` option, you'll see a prompt in order to confirm your action. Please ensure that you want to delete the selected chaos experiment and click the `Confirm` button:

<img src={require('../assets/user-guides/injecting-fault/delete-workflow/step-2.png').default} width="1000" height="500" />

## 3. The Chaos Scenario has been deleted
## 3. The Chaos experiment has been deleted

You will observe that the chaos scenario no longer appears in the list of schedules and has been removed.
You will observe that the chaos experiment no longer appears in the list of schedules and has been removed.

<img src={require('../assets/user-guides/injecting-fault/delete-workflow/step-3.png').default} width="1000" height="500" />

As stated above, we observe that the runs have been removed as well.

<img src={require('../assets/user-guides/injecting-fault/delete-workflow/step-4.png').default} width="1000" height="500" />
As stated above, the runs have been removed as well.

## Learn more

- [schedule a chaos scenario](schedule-experiment.md)
- [schedule a chaos experiment](schedule-experiment.md)
26 changes: 13 additions & 13 deletions website/versioned_docs/version-3.0.0/user-guides/edit-schedule.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,22 @@
---
id: edit-schedule
title: Edit Chaos Scenario Schedule
sidebar_label: Edit Chaos Scenario Schedule
title: Edit Chaos Experiment Schedule
sidebar_label: Edit Chaos Experiment Schedule
---

---

## Before you begin

You must schedule a chaos scenario. To know more about scheduling chaos scenarios click [here](schedule-experiment.md)
You must schedule a chaos experiment. To know more about scheduling chaos experiments click [here](schedule-experiment.md)

---

After you have scheduled a chaos scenario, you might have a need of changing the schedule of a recurring chaos scenario. To edit the schedule follow these steps:
After you have scheduled a chaos experiment, you might have a need of changing the schedule of a recurring chaos experiment. To edit the schedule follow these steps:

## 1. Select edit schedule from the menu

In the `Schedules` tab of `Litmus Chaos Scenarios` page you can click on the triple dots of the schedule to access more options for it. From the menu select the `Edit Schedule` option.
In the `Chaos experiments` page you can click on the triple dots of the experiment to access more options for it. From the menu select the `Edit Experiment` option.

<figure>
<img src={require('../assets/user-guides/injecting-fault/edit-schedule/schedule-menu.png').default} alt="Selecting Edit Schedule from the Menu" />
Expand All @@ -25,16 +25,16 @@ In the `Schedules` tab of `Litmus Chaos Scenarios` page you can click on the tri

## 2. Click on edit button

Now you'll be seeing the Summary of your chaos scenario and you can click on the `Edit` button to change the schedule.
Now you'll be seeing the pipeline diagram of your chaos experiment and you can click on the `Schedule` tab to change the schedule.

<figure>
<img src={require('../assets/user-guides/injecting-fault/edit-schedule/edit-schedule-page.png').default} alt="Summary of the Chaos Scenario with Edit button" />
<i>Summary of the Chaos Scenario with Edit button</i>
<img src={require('../assets/user-guides/injecting-fault/edit-schedule/edit-schedule-page.png').default} alt="Summary of the Chaos experiment with Edit button" />
<i>Summary of the Chaos experiment with Edit button</i>
</figure>

## 3. Change the schedule

Here you can change the schedule to the required interval and click on the `Verify` button.
Here you can change the schedule to the required interval and click on the `Set schedule` button.

<figure>
<img src={require('../assets/user-guides/injecting-fault/edit-schedule/edit-schedule.png').default} alt="Editing the Schedule" />
Expand All @@ -43,10 +43,10 @@ Here you can change the schedule to the required interval and click on the `Veri

## 4. Save the changes

Click on the `Save Changes` button to commit the changes to your chaos scenario.
Click the `Save` button to save the changes to the experiment chaos experiment.

## Learn more

- [Observe Chaos Scenario](observe-experiment.md)
- [Save Chaos Scenarios as a Template](save-as-template.md)
- [Re-run a Chaos Scenario](re-run-experiment.md)
- [Observe Chaos Experiment](observe-experiment.md)
- [Save Chaos Experiments as a Template](save-as-template.md)
- [Re-run a Chaos Experiment](re-run-experiment.md)
118 changes: 24 additions & 94 deletions website/versioned_docs/version-3.0.0/user-guides/observe-experiment.md
Original file line number Diff line number Diff line change
@@ -1,112 +1,42 @@
---
id: observe-experiment
title: Observe Chaos Scenario
sidebar_label: Observe Chaos Scenario
title: Observe Chaos experiment
sidebar_label: Observe Chaos experiment
---

---

## Before you begin
Visualization is an important aspect while doing chaos engineering. It allows the user to discover and inspect different changes that occur during a Chaos Experiment execution. <br/>
With ChaosCenter, the real-time data and status of the chaos experiments can be observed. Valuable information like pod logs, chaos experiment status, and chaos results can also be viewed.

You must schedule a chaos scenario. To know more about scheduling chaos scenarios click [here](schedule-experiment.md)
## Prerequisites

---

After scheduling a chaos scenario, you can track the status of the chaos scenario run from the `Runs` tab in the `Litmus Chaos Scenario`. The status that is currently displayed are:

- Failed
- Running
- Completed

<figure>
<img src={require('../assets/user-guides/injecting-fault/observe-workflow/runs-table.png').default} alt="Chaos Scenario Runs Table showing a Running Chaos Scenario" />
<i>Chaos Scenario Runs Table showing a Running Chaos Scenario</i>
</figure>

---

you can analyze a chaos scenario using two methods:

## Visualize the chaos scenario run graph

After scheduling a chaos scenario, you can click on the **Show the chaos scenario** option or click on the chaos scenario name to see the real-time graph of the chaos scenario.

<figure>
<img src={require('../assets/user-guides/injecting-fault/observe-workflow/running-workflow.png').default} alt="Chaos Scenario Runs Graph of Podtato Head chaos scenario" />
<i>Graph of Podtato Head chaos scenario</i>
</figure>

The graph consists of useful information such as :

- Phase of individual nodes.
- Total time taken for the nodes to execute.
- Structure of the experiments (Serial or Parallel experiments).

You can also visualize the non Chaos scenarios. The logs of individual nodes are also available here.

<figure>
<img src={require('../assets/argo-chaos-workflow.png').default} alt="Chaos Scenario run graph of a non chaos scenario" />
<i>Graph of a non Chaos Scenario</i>
</figure>

## View logs of individual nodes

you can click on the nodes to get the logs of that particular step. If the revert-chaos step is disabled, the complete logs are available which include the runner pod logs and the chaos logs.

<figure>
<img src={require('../assets/user-guides/injecting-fault/observe-workflow/running-workflow-with-logs.png').default} alt="Chaos Scenario Runs Podtato Head chaos scenario with Logs" />
<i>Podtato Head chaos scenario with Logs</i>
</figure>

## View chaos results

Once the experiment completes, the [Chaos Results](../glossary.md) are also available alongside the logs. The Chaos Results are directly fetched from the ChaosResult CRD.

<figure>
<img src={require('../assets/user-guides/injecting-fault/observe-workflow/completed-workflow-with-chaos-results.png').default} alt="Podtato Head chaos scenario with chaos logs and chaos result of pod-delete experiment" />
<i>Podtato Head chaos scenario with chaos logs and chaos result of generic/pod-delete experiment</i>
</figure>

## Resilience Score Calculation

A Resilience Score is the measure of how resilient your chaos scenario run is considering all the chaos experiments and their individual result points. This calculation takes into account the individual experiment weights (from a range of 1-10) which are relative to each other.

Once a weight has been assigned to the experiment, we look for the [Probe Success Percentage](../concepts/probes#probe-status--deriving-inferences) for that experiment itself (Post Chaos) and calculate the total resilience result for that experiment as a multiplication of the weight given and the probe success percentage returned after the Chaos Run.

```doc
Total Resilience for one single experiment = (Weight Given to that experiment * Probe Success Percentage)
```

> If an experiment doesn't have a probe in it, the probe success percentage returned can either be 0 or 100 based on the experiment verdict. If the experiment passed then it returns 100 else 0.

The Final Resilience Score is calculated by dividing the total test result by the sum of all the weights of all the experiments combined in a single chaos scenario.
The following should be required before creating a Chaos Experiment:

For example, if we consider two experiments in a chaos scenario, here is what the calculation would look like.
- ChaosCenter
- [Chaos Experiments](../concepts/chaos-workflow.md)

> Considering Probe Success Percentage is 100
## Litmus Chaos Experiment

| Experiment | Weight | Probe Success Percentage | Total Test Result |
| :--------- | :---------------------: | -----------------------: | -----------------------------------: |
| exp1 | 3 | 100 | (3 \* 100) = 300 |
| exp2 | 9 | 100 | (9 \* 100) = 900 |
| | Weight Sum = 3 + 9 = 12 | | Total Test Result = 300 + 900 = 1200 |
If the user chooses to 'Save' and 'Run' the experiment, they will be redirected directly to the experiment execution page where the experiment can be visualised else they will be taken Chaos Experiment Page.

```
Resilience Score = Total Test Result / Weight Sum
= 1200 / 12
= 100%
```
## Observe a Litmus Chaos Experiment

## Analytics from the runs tab
To observe a chaos experiment, user needs to select the highlighted experiment run box from the heatmap, it will redirect to experiment run execution page.<br/>
<img src={require('../assets/workflow-observe-select.png').default} width="800" />

Once the chaos scenario run execution completes, you can click the **Show the analytics** option in the `Runs` tab of `Litmus Chaos Scenarios` which opens up a [Chaos Scenario Dashboard](../user-guides/analyze-workflow.md) which can also be accessed from the Analytics section and is explained more [here](../user-guides/analyze-workflow.md). This analytics can be crucial to analyse the Cron Chaos Scenarios.
In the chaos experiment, a realtime graph of the chaos experiment is displayed. This graph contains valuable information regarding the status of individual steps of the chaos experiment.<br/><br/>
<img src={require('../assets/workflow-observe-running.png').default} width="800" /><br/><br/>
To view the details of each step, the user can click on the individual nodes and the right side pane will display the node details, results(once the execution is complete), and the logs related to it.
<br/><br/>
<img src={require('../assets/workflow-observe-log.png').default} width="800" />
<img src={require('../assets/workflow-observe-completed.png').default} width="800" />

## Resources
## Summary

<iframe width="560" height="315" src="https://www.youtube.com/embed/OuB3dS05DHU" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
After scheduling a chaos experiment, a user can view the details of the running chaos experiment from the ChaosCenter. ChaosCenter provides a real-time graph that is used to visualize the chaos experiment and get the details of individual steps of the chaos experiment. Important details like the logs and target applications can be viewed from ChaosCenter. Once the chaos experiment execution is completed, the resiliency score is calculated and the ChaosResult for the ChaosEngine pods is available.

## Learn more
## Learn More

- [Edit Schedule](edit-schedule.md)
- [Download Chaos Scenario Manifest](download-experiment-manifest.md)
- [Re-run a Chaos Scenario](re-run-experiment.md)
- [Explore Probes](../concepts/probes.md)
- [What is a Chaos Experiment](../concepts/chaos-workflow.md)
Loading