Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update README and Developer Guide #374

Merged
merged 4 commits into from
Sep 26, 2023
Merged

Update README and Developer Guide #374

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6333a40
Add Table of Contents
Sep 19, 2023
a9a9134
Shorten README and add more changes to Developer Guide
Sep 20, 2023
38847ed
Add fixes to PR
Sep 26, 2023
c0449b3
Lowercase repository name
Sep 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 29 additions & 13 deletions DEVELOPER_GUIDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,31 +2,47 @@

This document will walk you through on what's needed to start contributing code to OpenSearch Benchmark.

### Table of Contents
- [Installation](#installation)
- [Prerequisites](#prerequisites)
- [Setup](#setup)
- [Importing the project into an IDE](#importing-the-project-into-an-ide)
- [Executing tests](#executing-tests)
- [Unit tests](#unit-tests)
- [Integration tests](#integration-tests)
- [Submitting your changes for a pull request](#submitting-your-changes-for-a-pull-request)
- [Developing breaking changes](#developing-breaking-changes)
- [Miscellaneous](#miscellaneous)
- [git-secrets](#git-secrets)
- [Adding new major and minor Python versions to OpenSearch Benchmark](#adding-new-major-and-minor-python-versions-to-opensearch-benchmark)
- [Debugging unittests in Visual Studio Code](#debugging-unittests-in-visual-studio-code)

## Installation

### Prerequisites

- Pyenv : Install `pyenv` and follow the instructions in the output of `pyenv init` to set up your shell and restart it before proceeding.
- **Pyenv** : Install `pyenv` and follow the instructions in the output of `pyenv init` to set up your shell and restart it before proceeding.
For more details please refer to the [PyEnv installation instructions](https://github.com/pyenv/pyenv#installation).

- JDK: Although OSB is a Python application, it optionally builds and provisions OpenSearch clusters. JDK version 17 is used to build the current version of OpenSearch. Please refer to the [build setup requirements](https://github.com/opensearch-project/OpenSearch/blob/ca564fd04f5059cf9e3ce8aba442575afb3d99f1/DEVELOPER_GUIDE.md#install-prerequisites).
- **JDK**: Although OSB is a Python application, it optionally builds and provisions OpenSearch clusters. JDK version 17 is used to build the current version of OpenSearch. Please refer to the [build setup requirements](https://github.com/opensearch-project/OpenSearch/blob/ca564fd04f5059cf9e3ce8aba442575afb3d99f1/DEVELOPER_GUIDE.md#install-prerequisites).
Note that the `javadoc` executable should be available in the JDK installation. An earlier version of the JDK can be used, but not all the integration tests will pass.

```
export JAVA_HOME=/path/to/JDK17

```

- Install Docker and `docker-compose`. Start the Docker server. The user running the integration tests should have the permissions required to run docker commands. Test by running `docker ps`.
- **Docker**: Install Docker and `docker-compose`. Start the Docker server. The user running the integration tests should have the permissions required to run docker commands. Test by running `docker ps`.

- Git : git 1.9 or later.
- **Git** : supports versions 1.9+

### Setup

Use the following command-line instructions to set up OpenSearch Benchmark for development:
To develop OSB properly, it is recommended that you fork the official OpenSearch Benchmark repository.

After you git cloned the forked copy of OpenSearch Benchmark, use the following command-line instructions to set up OpenSearch Benchmark for development:
```
git clone https://github.com/opensearch-project/OpenSearch-Benchmark
cd OpenSearch-Benchmark
cd opensearch-benchmark
make prereq
make install
```
Expand All @@ -50,8 +66,6 @@ Depending on the platform and shell you have, use the following command to activ

For more information regarding activating virtual environments, please see https://docs.python.org/3/library/venv.html.

As an extra step, please consider configuring your JAVA_HOMES as mentioned in the `Important information related to integration tests`.

## Importing the project into an IDE

OpenSearch Benchmark builds using virtualenv. When importing into an IDE, such as PyCharm IDE, you will need to define an appropriate Python SDK, which is provided by virtualenv.
Expand All @@ -65,10 +79,10 @@ In order to run tests within the PyCharm IDE, ensure the `Python Integrated Tool
Once setup is complete, you may run the unit and integration tests.

### Unit Tests
Invoke unit tests by running the following command within the root directory of the repository:

```
make test

```

### Integration Tests
Expand All @@ -80,12 +94,14 @@ Integration tests can be run on the following operating systems:
* Amazon Linux 2
* MacOS

Invoke integration tests by running the following command within the root directory of the repository:
IanHoang marked this conversation as resolved.
Show resolved Hide resolved

```
make it

```

Integration tests are expected to run for approximately 20-30 mins.

## Submitting your changes for a pull request

Once your changes and tests are ready to submit for review:
Expand All @@ -109,7 +125,7 @@ Once your changes and tests are ready to submit for review:
## Developing Breaking Changes
Breaking changes should not be directly added to the `main` branch. These should be developed in their own feature branch. Prior to a new release this feature branch should be rebased onto the latest changes from `main`. `main` can then `pull` or `cherry-pick` the breaking changes from the feature branch.

## Misc
## Miscellaneous

### git-secrets
Security is our top priority. Avoid checking in credentials, install awslabs/git-secrets.
Expand All @@ -120,7 +136,7 @@ cd git-secrets
make install
```
### Adding New Major and Minor Python Versions to OpenSearch-Benchmark
To streamline the process, please refer to [this guide](./PYTHON_SUPPORT_GUIDE.md)
To streamline the process, please refer to [this guide](https://github.com/opensearch-project/opensearch-benchmark/blob/main/PYTHON_SUPPORT_GUIDE.md)

### Debugging Unittests in Visual Studio Code
To run and debug unittests in Visual Studio Code, add the following configuration to the Python Debugger `launch.json` file. See [the official Visual Studio Code documentation](https://code.visualstudio.com/docs/editor/debugging) for more information on setting up and accessing `launch.json` file.
Expand Down
91 changes: 3 additions & 88 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,103 +20,18 @@ We have also put considerable effort into OpenSearch Benchmark to ensure that be
Documentation
-------------

Documentation for OpenSearch Benchmark is [available online](https://opensearch.org/docs/latest/benchmark/index/).
Official documentation for OpenSearch Benchmark is [available online](https://opensearch.org/docs/latest/benchmark/index/).

Quick Start
-----------

OpenSearch Benchmark is developed for Unix and is actively tested on Linux, Ubuntu, and MacOS. OpenSearch Benchmark supports [benchmarking OpenSearch clusters running on Windows](<https://github.com/opensearch-project/OpenSearch-Benchmark/blob/main/DEVELOPER_GUIDE.md>) but OpenSearch Benchmark itself needs to be installed on machines running Unix.

Installing OpenSearch Benchmark
-------------------------------

**Note**: If you actively develop on OpenSearch, we recommend that you [install OpenSearch Benchmark in development mode](<https://github.com/opensearch-project/OpenSearch-Benchmark/blob/main/DEVELOPER_GUIDE.md>) instead as OpenSearch is fast moving and OpenSearch Benchmark always adapts accordingly to the latest main version.

Install Python 3.8+ including ``pip3``, git 1.9+ and an [appropriate JDK to run OpenSearch](<https://github.com/opensearch-project/OpenSearch-Benchmark/blob/main/DEVELOPER_GUIDE.md>) Be sure that ``JAVA_HOME`` points to that JDK and that the corresponding Java executable in ``$JAVA_HOME/bin`` is picked up via your PATH environment variable. Then run the following command, optionally prefixed by ``sudo`` if necessary:

python3 -m pip install opensearch-benchmark


If you have any trouble or need more detailed instructions, please look in the [detailed installation guide](<https://github.com/opensearch-project/OpenSearch-Benchmark/blob/main/DEVELOPER_GUIDE.md>).

Run your first test execution
-----------------------------

Now we're ready to run our first test execution:

opensearch-benchmark execute-test --distribution-version=1.0.0 --workload=geonames --test-mode

This will download OpenSearch 1.0.0 and run one of OpenSearch Benchmark's official workloads - the [geonames workload](<https://github.com/opensearch-project/opensearch-benchmark-workloads/tree/main/geonames>) - against it.
Note that this uses the `--test-mode` argument to only run a single instance of each operation in order to reduce the time needed for a test execution. This argument is used as a sanity check and should be removed in an actual benchmarking scenario.
After the test execution, a summary report is written to the command line:

------------------------------------------------------
_______ __ _____
/ ____(_)___ ____ _/ / / ___/_________ ________
/ /_ / / __ \/ __ `/ / \__ \/ ___/ __ \/ ___/ _ \
/ __/ / / / / / /_/ / / ___/ / /__/ /_/ / / / __/
/_/ /_/_/ /_/\__,_/_/ /____/\___/\____/_/ \___/
------------------------------------------------------

| Metric | Task | Value | Unit |
|-------------------------------:|---------------------:|----------:|-------:|
| Total indexing time | | 28.0997 | min |
| Total merge time | | 6.84378 | min |
| Total refresh time | | 3.06045 | min |
| Total flush time | | 0.106517 | min |
| Total merge throttle time | | 1.28193 | min |
| Median CPU usage | | 471.6 | % |
| Total Young Gen GC | | 16.237 | s |
| Total Old Gen GC | | 1.796 | s |
| Index size | | 2.60124 | GB |
| Total written | | 11.8144 | GB |
| Heap used for segments | | 14.7326 | MB |
| Heap used for doc values | | 0.115917 | MB |
| Heap used for terms | | 13.3203 | MB |
| Heap used for norms | | 0.0734253 | MB |
| Heap used for points | | 0.5793 | MB |
| Heap used for stored fields | | 0.643608 | MB |
| Segment count | | 97 | |
| Min Throughput | index-append | 31925.2 | docs/s |
| Median Throughput | index-append | 39137.5 | docs/s |
| Max Throughput | index-append | 39633.6 | docs/s |
| 50.0th percentile latency | index-append | 872.513 | ms |
| 90.0th percentile latency | index-append | 1457.13 | ms |
| 99.0th percentile latency | index-append | 1874.89 | ms |
| 100th percentile latency | index-append | 2711.71 | ms |
| 50.0th percentile service time | index-append | 872.513 | ms |
| 90.0th percentile service time | index-append | 1457.13 | ms |
| 99.0th percentile service time | index-append | 1874.89 | ms |
| 100th percentile service time | index-append | 2711.71 | ms |
| ... | ... | ... | ... |
| ... | ... | ... | ... |
| Min Throughput | painless_dynamic | 2.53292 | ops/s |
| Median Throughput | painless_dynamic | 2.53813 | ops/s |
| Max Throughput | painless_dynamic | 2.54401 | ops/s |
| 50.0th percentile latency | painless_dynamic | 172208 | ms |
| 90.0th percentile latency | painless_dynamic | 310401 | ms |
| 99.0th percentile latency | painless_dynamic | 341341 | ms |
| 99.9th percentile latency | painless_dynamic | 344404 | ms |
| 100th percentile latency | painless_dynamic | 344754 | ms |
| 50.0th percentile service time | painless_dynamic | 393.02 | ms |
| 90.0th percentile service time | painless_dynamic | 407.579 | ms |
| 99.0th percentile service time | painless_dynamic | 430.806 | ms |
| 99.9th percentile service time | painless_dynamic | 457.352 | ms |
| 100th percentile service time | painless_dynamic | 459.474 | ms |

----------------------------------
[INFO] SUCCESS (took 2634 seconds)
----------------------------------

Creating Your Own Workloads
---------------------------
For more information on how users can create their own workloads, see [the Create Workload Guide](./CREATE_WORKLOAD_GUIDE.md)
Want to get started with OpenSearch Benchmark quickly? See [OpenSearch Benchmark's Quick Start guide in the documentation](https://opensearch.org/docs/latest/benchmark/index/).

Getting help
------------

* Quick help: ``opensearch-benchmark --help``
* Look in [OpenSearch Benchmark's user guide](<https://github.com/opensearch-project/OpenSearch-Benchmark/blob/main/DEVELOPER_GUIDE.md>) for more information
* Want to contribute? Look at [OpenSearch Benchmark's Developer Guide](<https://github.com/opensearch-project/OpenSearch-Benchmark/blob/main/DEVELOPER_GUIDE.md>) for more information
* For any questions or answers, visit our [community forum](<https://discuss.opendistrocommunity.dev/>).
* File improvements or bug reports in our [Github repo](<https://github.com/opensearch-project/OpenSearch-Benchmark/issues>).

Expand Down
Loading