diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index ed6e387e4..7422aee3d 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -27,3 +27,14 @@ jobs: shell: bash - run: go version - run: make lint + + prettify: + name: Prettify + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + with: { fetch-depth: 1 } + - name: Install dependencies + run: npm i -g npm@8 && npm i -g prettier + - name: Check markdown files + run: prettier --prose-wrap never --check **/*.md diff --git a/ADMINS.md b/ADMINS.md index 1ec1f9e4b..f9220c93b 100644 --- a/ADMINS.md +++ b/ADMINS.md @@ -11,13 +11,13 @@ This document explains who the admins are (see below), what they do in this repo ## Current Admins -| Admin | GitHub ID | Affiliation | -| ----------------------- | ------------------------------------------- | ----------- | -| Charlotte | [CEHENKLE](https://github.com/CEHENKLE) | Amazon | -| Henri Yandell | [hyandell](https://github.com/hyandell) | Amazon | -| Jack Mazanec | [jmazanec15](https://github.com/jmazanec15) | Amazon | -| Vamshi Vijay Nakkirtha | [vamshin](https://github.com/vamshin) | Amazon | -| Vijayan Balasubramanian | [VijayanB](https://github.com/VijayanB) | Amazon | +| Admin | GitHub ID | Affiliation | +| --- | --- | --- | +| Charlotte | [CEHENKLE](https://github.com/CEHENKLE) | Amazon | +| Henri Yandell | [hyandell](https://github.com/hyandell) | Amazon | +| Jack Mazanec | [jmazanec15](https://github.com/jmazanec15) | Amazon | +| Vamshi Vijay Nakkirtha | [vamshin](https://github.com/vamshin) | Amazon | +| Vijayan Balasubramanian | [VijayanB](https://github.com/VijayanB) | Amazon | ## Admin Responsibilities diff --git a/CHANGELOG.md b/CHANGELOG.md index 4718a1889..5dcc0ad5b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,6 +20,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) - Adds `Err()` function to Response for detailed errors ([#246](https://github.com/opensearch-project/opensearch-go/pull/246)) - Adds Point In Time API ([#253](https://github.com/opensearch-project/opensearch-go/pull/253)) - Adds InfoResp type ([#253](https://github.com/opensearch-project/opensearch-go/pull/253)) +- Adds markdown linter ([#261](https://github.com/opensearch-project/opensearch-go/pull/261)) - Adds testcases to check upsert functionality ([#207](https://github.com/opensearch-project/opensearch-go/issues/207)) - Added @Jakob3xD to co-maintainers ([#270](https://github.com/opensearch-project/opensearch-go/pull/270)) @@ -44,4 +45,4 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) ### Security -[Unreleased]: https://github.com/opensearch-project/opensearch-go/compare/2.1...HEAD \ No newline at end of file +[Unreleased]: https://github.com/opensearch-project/opensearch-go/compare/2.1...HEAD diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index c4279a2f2..99eb3ac87 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -7,6 +7,8 @@ - [Unit Testing](#unit-testing) - [Integration Testing](#integration-testing) - [Execute integration tests from your terminal](#execute-integration-tests-from-your-terminal) + - [Lint](#lint) + - [Markdown lint](#markdown-lint) - [Use an Editor](#use-an-editor) - [GoLand](#goland) @@ -18,8 +20,7 @@ So you want to contribute code to the OpenSearch Go Client? Excellent! We're gla ### Git Clone OpenSearch Go Client Repository -Fork [opensearch-project/opensearch-go](https://github.com/opensearch-project/opensearch-go) and clone locally, -e.g. `git clone https://github.com/[your username]/opensearch-go.git`. +Fork [opensearch-project/opensearch-go](https://github.com/opensearch-project/opensearch-go) and clone locally, e.g. `git clone https://github.com/[your username]/opensearch-go.git`. ### Install Prerequisites @@ -48,13 +49,9 @@ go test -v -run TestName; ### Integration Testing -In order to test opensearch-go client, you need a running OpenSearch cluster. You can use Docker to accomplish this. -The [Docker Compose file](.ci/opensearch/docker-compose.yml) supports the ability to run integration tests for the project in local environments. -If you have not installed docker-compose, you can install it from this [link](https://docs.docker.com/compose/install/). +In order to test opensearch-go client, you need a running OpenSearch cluster. You can use Docker to accomplish this. The [Docker Compose file](.ci/opensearch/docker-compose.yml) supports the ability to run integration tests for the project in local environments. If you have not installed docker-compose, you can install it from this [link](https://docs.docker.com/compose/install/). -In order to differentiate unit tests from integration tests, Go has a built-in mechanism for allowing you to logically separate your tests -with [build tags](https://pkg.go.dev/cmd/go#hdr-Build_constraints). The build tag needs to be placed as close to the top of the file as possible, and must have a blank line beneath it. -Hence, create all integration tests with build tag 'integration'. +In order to differentiate unit tests from integration tests, Go has a built-in mechanism for allowing you to logically separate your tests with [build tags](https://pkg.go.dev/cmd/go#hdr-Build_constraints). The build tag needs to be placed as close to the top of the file as possible, and must have a blank line beneath it. Hence, create all integration tests with build tag 'integration'. #### Execute integration tests from your terminal @@ -71,6 +68,18 @@ Hence, create all integration tests with build tag 'integration'. make cluster.stop cluster.clean ``` +## Lint + +To keep all the code in a certain uniform format, it was decided to use some writing rules. If you wrote something wrong, it's okay, you can simply run the script to check the necessary files, and optionally format the content. But keep in mind that all these checks are repeated on the pipeline, so it's better to check locally. + +### Markdown lint + +To check the markdown files, run the following command: + +``` +make lint.markdown +``` + ## Use an Editor ### GoLand @@ -80,6 +89,4 @@ You can import the OpenSearch project into GoLand as follows: 1. Select **File | Open** 2. In the subsequent dialog navigate to the ~/go/src/opensearch-go and click **Open** -After you have opened your project, you need to specify the location of the Go SDK. -You can either specify a local path to the SDK or download it. To set the Go SDK, navigate to **Go | GOROOT** and -set accordingly. +After you have opened your project, you need to specify the location of the Go SDK. You can either specify a local path to the SDK or download it. To set the Go SDK, navigate to **Go | GOROOT** and set accordingly. diff --git a/MAINTAINERS.md b/MAINTAINERS.md index 8a62409d4..e68e93297 100644 --- a/MAINTAINERS.md +++ b/MAINTAINERS.md @@ -4,18 +4,18 @@ This document contains a list of maintainers in this repo. See [opensearch-proje ## Current Maintainers -| Maintainer | GitHub ID | Affiliation | -| ----------------------- | ----------------------------------------- | ------------------- | -| Daniel Doubrovkine | [dblock](https://github.com/dblock) | Amazon | -| Jakob Hahn | [Jakob3xD](https://github.com/Jakob3xD) | Hetzner Online GmbH | -| Vacha Shah | [VachaShah](https://github.com/VachaShah) | Amazon | -| Vijayan Balasubramanian | [VijayanB](https://github.com/VijayanB) | Amazon | +| Maintainer | GitHub ID | Affiliation | +| --- | --- | --- | +| Daniel Doubrovkine | [dblock](https://github.com/dblock) | Amazon | +| Jakob Hahn | [Jakob3xD](https://github.com/Jakob3xD) | Hetzner Online GmbH | +| Vacha Shah | [VachaShah](https://github.com/VachaShah) | Amazon | +| Vijayan Balasubramanian | [VijayanB](https://github.com/VijayanB) | Amazon | ## Emeritus -| Maintainer | GitHub ID | Affiliation | -| ---------------------- | ------------------------------------------- | ----------- | -| Jack Mazanec | [jmazanec15](https://github.com/jmazanec15) | Amazon | -| Rob Cowart | [robcowart](https://github.com/robcowart) | ElastiFlow | -| Sven Cowart | [svencowart](https://github.com/svencowart) | | -| Vamshi Vijay Nakkirtha | [vamshin](https://github.com/vamshin) | Amazon | +| Maintainer | GitHub ID | Affiliation | +| --- | --- | --- | +| Jack Mazanec | [jmazanec15](https://github.com/jmazanec15) | Amazon | +| Rob Cowart | [robcowart](https://github.com/robcowart) | ElastiFlow | +| Sven Cowart | [svencowart](https://github.com/svencowart) | | +| Vamshi Vijay Nakkirtha | [vamshin](https://github.com/vamshin) | Amazon | diff --git a/Makefile b/Makefile index 2548e4a95..ade9b9667 100644 --- a/Makefile +++ b/Makefile @@ -66,6 +66,42 @@ lint: ## Run lint on the package cd "internal/build/" && go mod tidy && go mod download && go vet ./...; \ } +package := "prettier" +lint.markdown: + @printf "\033[2m→ Checking node installed...\033[0m\n" + if type node > /dev/null 2>&1 && which node > /dev/null 2>&1 ; then \ + node -v; \ + echo -e "\033[33m Node is installed, continue...\033[0m\n"; \ + else \ + echo -e "\033[31m Please install node\033[0m\n"; \ + exit 1; \ + fi + @printf "\033[2m→ Checking npm installed...\033[0m\n" + if type npm > /dev/null 2>&1 && which npm > /dev/null 2>&1 ; then \ + npm -v; \ + echo -e "\033[33m NPM is installed, continue...\033[0m\n"; \ + else \ + echo -e "\033[31m Please install npm\033[0m\n"; \ + exit 1; \ + fi + @printf "\033[2m→ Checking $(package) installed...\033[0m\n" + if [ `npm list -g | grep -c $(package)` -eq 0 -o ! -d node_module ]; then \ + echo -e "\033[33m Installing $(package)...\033[0m"; \ + npm install -g $(package) --no-shrinkwrap; \ + fi + @printf "\033[2m→ Running markdown lint...\033[0m\n" + if npx $(package) --prose-wrap never --check **/*.md; [[ $$? -ne 0 ]]; then \ + echo -e "\033[32m→ Found invalid files. Want to auto-format invalid files? (y/n) \033[0m"; \ + read RESP; \ + if [[ $$RESP = "y" || $$RESP = "Y" ]]; then \ + echo -e "\033[33m Formatting...\033[0m"; \ + npx $(package) --prose-wrap never --write **/*.md; \ + echo -e "\033[34m \nAll invalid files are formatted\033[0m"; \ + else \ + echo -e "\033[33m Unfortunately you are cancelled auto fixing. But we will definitely fix it in the pipeline\033[0m"; \ + fi \ + fi + backport: ## Backport one or more commits from main into version branches ifeq ($(origin commits), undefined) @@ -196,3 +232,4 @@ help: ## Display help .DEFAULT_GOAL := help .PHONY: help backport cluster cluster.clean coverage godoc lint release test test-bench test-integ test-unit linters linters.install +.SILENT: lint.markdown diff --git a/README.md b/README.md index 8161dfd43..bd0260fc6 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,4 @@ -[![Go Reference](https://pkg.go.dev/badge/github.com/opensearch-project/opensearch-go.svg)](https://pkg.go.dev/github.com/opensearch-project/opensearch-go/v2) -[![Build](https://github.com/opensearch-project/opensearch-go/actions/workflows/lint.yml/badge.svg)](https://github.com/opensearch-project/opensearch-go/actions/workflows/lint.yml) -[![Unit](https://github.com/opensearch-project/opensearch-go/actions/workflows/test-unit.yml/badge.svg)](https://github.com/opensearch-project/opensearch-go/actions/workflows/test-unit.yml) -[![Integration](https://github.com/opensearch-project/opensearch-go/actions/workflows/test-integration.yml/badge.svg)](https://github.com/opensearch-project/opensearch-go/actions/workflows/test-integration.yml) -[![codecov](https://codecov.io/gh/opensearch-project/opensearch-go/branch/main/graph/badge.svg?token=MI9g3KYHVx)](https://codecov.io/gh/opensearch-project/opensearch-go) -[![Chat](https://img.shields.io/badge/chat-on%20forums-blue)](https://discuss.opendistrocommunity.dev/c/clients/) -![PRs welcome!](https://img.shields.io/badge/PRs-welcome!-success) +[![Go Reference](https://pkg.go.dev/badge/github.com/opensearch-project/opensearch-go.svg)](https://pkg.go.dev/github.com/opensearch-project/opensearch-go/v2) [![Build](https://github.com/opensearch-project/opensearch-go/actions/workflows/lint.yml/badge.svg)](https://github.com/opensearch-project/opensearch-go/actions/workflows/lint.yml) [![Unit](https://github.com/opensearch-project/opensearch-go/actions/workflows/test-unit.yml/badge.svg)](https://github.com/opensearch-project/opensearch-go/actions/workflows/test-unit.yml) [![Integration](https://github.com/opensearch-project/opensearch-go/actions/workflows/test-integration.yml/badge.svg)](https://github.com/opensearch-project/opensearch-go/actions/workflows/test-integration.yml) [![codecov](https://codecov.io/gh/opensearch-project/opensearch-go/branch/main/graph/badge.svg?token=MI9g3KYHVx)](https://codecov.io/gh/opensearch-project/opensearch-go) [![Chat](https://img.shields.io/badge/chat-on%20forums-blue)](https://discuss.opendistrocommunity.dev/c/clients/) ![PRs welcome!](https://img.shields.io/badge/PRs-welcome!-success) ![OpenSearch logo](OpenSearch.svg) diff --git a/UPGRADING.md b/UPGRADING.md index 5981282ff..d09583b4d 100644 --- a/UPGRADING.md +++ b/UPGRADING.md @@ -1,7 +1,7 @@ - [Upgrading Opensearch GO Client](#upgrading-opensearch-go-client) - - [Upgraading to >= 3.0.0](#upgrading-to->=-3.0.0) - - [opensearchapi](#opensearchapi-snapshot-delete) - - [opensearchapi](#opensearchapi-error-handling) + - [Upgraading to >= 3.0.0](#upgrading-to->=-3.0.0) + - [opensearchapi](#opensearchapi-snapshot-delete) + - [opensearchapi](#opensearchapi-error-handling) # Upgrading Opensearch GO Client @@ -50,11 +50,10 @@ reqSnapshots := &opensearchapi.SnapshotDeleteRequest{ With opensearch-go >= 3.0.0 opensearchapi responses are now checked for errors. Checking for errors twice is no longer needed. -Prior versions only returned an error if the request failed to execute. For example if the client can't reach the server or the TLS handshake failed. -With opensearch-go >= 3.0.0 each opensearchapi requests will return an error if the response http status code is > 299. -The error can be parsed into the new `opensearchapi.Error` type by using `errors.As` to match for exceptions and get a more detailed view. +Prior versions only returned an error if the request failed to execute. For example if the client can't reach the server or the TLS handshake failed. With opensearch-go >= 3.0.0 each opensearchapi requests will return an error if the response http status code is > 299. The error can be parsed into the new `opensearchapi.Error` type by using `errors.As` to match for exceptions and get a more detailed view. Before 3.0.0: + ```go createIndex := opensearchapi.IndicesCreateRequest{ Index: IndexName, @@ -94,4 +93,3 @@ if err != nil { } } ``` - diff --git a/USER_GUIDE.md b/USER_GUIDE.md index db4a128cd..da129b133 100644 --- a/USER_GUIDE.md +++ b/USER_GUIDE.md @@ -1,21 +1,18 @@ - [User Guide](#user-guide) - - [Example](#example) - - [Amazon OpenSearch Service](#amazon-opensearch-service) - - [AWS SDK V1](#aws-sdk-v1) - - [AWS SDK V2](#aws-sdk-v2) - - [Data Streams API](#data-streams-api) - - [Create Data Stream](#create-data-streams) - - [Delete Data Stream](#delete-data-streams) - - [Get All Data Streams](#get-data-streams) - - [Get Specific Data Stream](#get-specific-data-streams) - - [Get Specific Data Stream Stats](#get-specific-data-streams-stats) + - [Example](#example) + - [Amazon OpenSearch Service](#amazon-opensearch-service) - [AWS SDK V1](#aws-sdk-v1) - [AWS SDK V2](#aws-sdk-v2) + - [Data Streams API](#data-streams-api) + - [Create Data Stream](#create-data-streams) + - [Delete Data Stream](#delete-data-streams) + - [Get All Data Streams](#get-data-streams) + - [Get Specific Data Stream](#get-specific-data-streams) + - [Get Specific Data Stream Stats](#get-specific-data-streams-stats) # User Guide ## Example -In the example below, we create a client, an index with non-default settings, insert a document to the index, -search for the document, delete the document and finally delete the index. +In the example below, we create a client, an index with non-default settings, insert a document to the index, search for the document, delete the document and finally delete the index. ```go package main @@ -171,26 +168,17 @@ func example() error { ## Amazon OpenSearch Service -Before starting, we strongly recommend reading the full AWS documentation regarding using IAM credentials to sign -requests to OpenSearch APIs. -See [Identity and Access Management in Amazon OpenSearch Service.](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ac.html) +Before starting, we strongly recommend reading the full AWS documentation regarding using IAM credentials to sign requests to OpenSearch APIs. See [Identity and Access Management in Amazon OpenSearch Service.](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ac.html) -> Even if you configure a completely open resource-based access policy, all requests to the OpenSearch Service -> configuration API must be signed. If your policies specify IAM users or roles, requests to the OpenSearch APIs also -> must -> be signed using AWS Signature Version 4. +> Even if you configure a completely open resource-based access policy, all requests to the OpenSearch Service configuration API must be signed. If your policies specify IAM users or roles, requests to the OpenSearch APIs also must be signed using AWS Signature Version 4. > > See [Managed Domains signing-service requests.](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ac.html#managedomains-signing-service-requests) -Depending on the version of AWS SDK used, import the v1 or v2 request signer from `signer/aws` or `signer/awsv2` -respectively. -Both signers are equivalent in their functionality, they provide AWS Signature Version 4 (SigV4). +Depending on the version of AWS SDK used, import the v1 or v2 request signer from `signer/aws` or `signer/awsv2` respectively. Both signers are equivalent in their functionality, they provide AWS Signature Version 4 (SigV4). -To read more about SigV4 -see [Signature Version 4 signing process](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html) +To read more about SigV4 see [Signature Version 4 signing process](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html) -Here are some Go samples that show how to sign each OpenSearch request and automatically search for AWS credentials from -the ~/.aws folder or environment variables: +Here are some Go samples that show how to sign each OpenSearch request and automatically search for AWS credentials from the ~/.aws folder or environment variables: #### AWS SDK V1 @@ -344,7 +332,8 @@ func getCredentialProvider(accessKey, secretAccessKey, token string) aws.Credent ### Create Data Streams - - Create new client +- Create new client + ``` client, err := opensearch.NewDefaultClient() if err != nil { @@ -352,7 +341,7 @@ if err != nil { } ``` - - Create template index +- Create template index ``` iPut := opensearchapi.IndicesPutIndexTemplateRequest{ @@ -365,7 +354,8 @@ iPut := opensearchapi.IndicesPutIndexTemplateRequest{ iPutResponse, err := iPut.Do(context.Background(), client) ``` - - Prepare request object +- Prepare request object + ``` es := opensearchapi.IndicesCreateDataStreamRequest{ Name: "demo-name", @@ -378,7 +368,8 @@ es := opensearchapi.IndicesCreateDataStreamRequest{ } ``` - - Execute request +- Execute request + ``` res, err := es.Do(context.TODO(), client) if err != nil { @@ -387,7 +378,8 @@ if err != nil { } ``` - - Try to read response +- Try to read response + ``` defer res.Body.Close() body, err := ioutil.ReadAll(res.Body) @@ -401,7 +393,8 @@ fmt.Println("Response Headers: ", res.Header) fmt.Println("Response Body: ", string(body)) ``` - - Successfully created data stream +- Successfully created data stream + ``` Response Status Code: 200 Response Headers: map[Content-Length:[28] Content-Type:[application/json; charset=UTF-8]] @@ -410,8 +403,9 @@ Response Body: {"acknowledged" : true} ### Delete Data Streams - - Create new client as previous example - - Prepare request object +- Create new client as previous example +- Prepare request object + ``` opensearchapi.IndicesDeleteDataStreamRequest{ Name: "demo-name", @@ -423,9 +417,11 @@ opensearchapi.IndicesDeleteDataStreamRequest{ }, } ``` + - Execute request as previous example - Try to read response as previous example -- Successfully deleted data stream +- Successfully deleted data stream + ``` Response Status Code: 200 Response Headers: map[Content-Length:[28] Content-Type:[application/json; charset=UTF-8]] @@ -436,6 +432,7 @@ Response Body: {"acknowledged" : true} - Create new client as previous example - Prepare request object + ``` r := opensearchapi.IndicesGetDataStreamRequest{ Pretty: true, @@ -446,9 +443,11 @@ r := opensearchapi.IndicesGetDataStreamRequest{ }, } ``` + - Execute request as previous example - Try to read response as previous example - Successfully retrieved data streams + ``` Response Status Code: 200 Response Headers: map[Content-Length:[28] Content-Type:[application/json; charset=UTF-8]] @@ -459,6 +458,7 @@ Response Body: {"data_streams":[{"name":"demo-name","timestamp_field":{"n - Create new client as previous example - Prepare request object + ``` r := opensearchapi.IndicesGetDataStreamRequest{ Name: "demo-name", @@ -470,9 +470,11 @@ r := opensearchapi.IndicesGetDataStreamRequest{ }, } ``` + - Execute request as previous example - Try to read response as previous example - Successfully retrieved data stream + ``` Response Status Code: 200 Response Headers: map[Content-Length:[28] Content-Type:[application/json; charset=UTF-8]] @@ -483,6 +485,7 @@ Response Body: {"data_streams":[{"name":"demo-name","timestamp_field":{"n - Create new client as as previous example - Prepare request object + ``` r := opensearchapi.IndicesGetDataStreamStatsRequest{ Name: "demo-name", @@ -494,11 +497,13 @@ r := opensearchapi.IndicesGetDataStreamStatsRequest{ }, } ``` + - Execute request as previous example - Try to read response as previous example - Successfully retrieved data stream stats + ``` Response Status Code: 200 Response Headers: map[Content-Length:[28] Content-Type:[application/json; charset=UTF-8]] Response Body: {"_shards":{"total":2,"successful":1,"failed":0},"data_stream_count":1,"backing_indices":1,"total_store_size":"208b","total_store_size_bytes":208,"data_streams":[{"data_stream":"demo-name","backing_indices":1,"store_size":"208b","store_size_bytes":208,"maximum_timestamp":0}]} -``` \ No newline at end of file +```