GH 1451 #1454

yaniveliash · 2023-03-07T07:38:43Z

Description

A CLI Tool to generate a stub for new observability integrations

Issues Resolved

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

ps48 · 2023-03-07T16:30:55Z

@yaniveliash Thank you so much for your contribution. I see the DCO is failing on the pull request. Can you please provide sign-offs to your commits? You can checkout more details here: https://github.com/opensearch-project/observability/pull/1454/checks?check_run_id=11814492848

Signed-off-by: Yaniv Eliash <yaniv@sre.engineer>

yaniveliash · 2023-03-07T17:26:10Z

@yaniveliash Thank you so much for your contribution. I see the DCO is failing on the pull request. Can you please provide sign-offs to your commits? You can checkout more details here: https://github.com/opensearch-project/observability/pull/1454/checks?check_run_id=11814492848

Done
I'm pretty sure there will be more to fix ;-)

YANG-DB · 2023-03-07T17:36:02Z

Hi @yaniveliash
Thanks again for your contribution .

can you plz add categories & collection section for user interaction ?

Swiddis

Thanks for the contribution! I left some comments for minor tweaks regarding cross-platform stability, but otherwise this is a good start.

scripts/integrations/main.py

derek-ho · 2023-03-07T17:55:27Z

scripts/integrations/config.ini

@@ -0,0 +1,2 @@
+[DEFAULT]
+LICENSE = MIT


should this be apache 2.0?

Maybe?
I'm less familiar with opensource licenses and regulations,
I've placed it just for default value concept and will change it to whatever needed :)

Apache-2.0 for compatibility with SPDX, but I agree.

derek-ho · 2023-03-07T17:57:15Z

scripts/integrations/main.py

+ }}
+
+ with open(directory + '/config.json', 'w', encoding='utf-8') as f:
+ json.dump(data, f, ensure_ascii=False, indent=4)


This would only be for local data dump - should we think about how can we add this globally, not just on this cluster? Automated way to create a PR, etc.?

@derek-ho
Is that a design question for the team or something for me to rethink about?
ain't sure :)

see design RFC

the script's path * Changed file and directories path concatanation to os.path.join * Linter kicked in and formatted the file. Signed-off-by: Yaniv Eliash <yaniv@sre.engineer>

YANG-DB · 2023-03-07T18:36:39Z

Can you plz add a README.md describing the flow of this CLI ?
thanks 🙏

YANG-DB · 2023-03-07T18:16:40Z

scripts/integrations/main.py

+ schema: str = typer.Option(
+ "", "--schema", "-s", help="Which schema, and at which version, to use. The data will be ingested with this schema."),
+ catalog: str = typer.Option(
+ "", "--catalog", "-c", help="The specific catalog to target, such as `observability` or `security`."),


can we have an functionality that will scan the /schema folder and present the results for the customer to select from ?
see https://github.com/opensearch-project/observability/tree/2.6/schema

Which part to scan in particular?
folder names? (e.g. metrics and traces) or something else?

Do you need the option to provide a checkout tag?

the next PR has additional clarification on the catalog folder structure and information
see catalog.json

YANG-DB · 2023-03-07T18:30:27Z

scripts/integrations/main.py

+ sys.exit(1)
+
+
+def main(


the general flow for this CLI would be as follows:

General: select the schema from a closed list
For each collections:

"collection":[ { "logs": [{ "info": "access logs", "input_type":"logfile", "dataset":"nginx.access", "labels" :["nginx","access"], "schema": "./schema/logs/access.json" }, ...

Collections: select number of collections

For Each :

Select schema category for collection (sub folder in the schema )

input info, dataset , labels, schema

select input_type from a list given from file names (without suffixes ) under the schema folder

From where can I source the list of collections and their respective schemas?

see above comment ...

YANG-DB · 2023-03-09T23:10:22Z

IMO u can mark this PR as draft for now ...

Swiddis · 2023-03-14T19:55:57Z

Hi,

The requirements for the tool have been considerably fleshed out. The updated spec is available on issue #1451. Are you available to keep working on this?

For an initial PR, it would be sufficient to stub out the 3 described commands, create, package, and check. These can just be routed to different no-op methods with arguments. If you want to add more functionality on top of that, by all means, feel free to do so.

YANG-DB · 2023-04-02T02:29:49Z

scripts/integrations/main.py

+ }}
+
+ with open(directory + '/config.json', 'w', encoding='utf-8') as f:
+ json.dump(data, f, ensure_ascii=False, indent=4)


see design RFC

Swiddis · 2023-08-08T17:32:12Z

Closed for inactivity. The design of the CLI has changed considerably since the integrations project has taken off, so the modern form of it fills a different purpose. The current version of the CLI is available at the OpenSearch Catalog.

yaniveliash requested a review from a team as a code owner March 7, 2023 07:38

YANG-DB added enhancement New feature or request feature labels Mar 7, 2023

yaniveliash added 4 commits March 7, 2023 19:24

ignore python venv

4577589

Signed-off-by: Yaniv Eliash <yaniv@sre.engineer>

main cli tool

64f6c43

Signed-off-by: Yaniv Eliash <yaniv@sre.engineer>

python requirements file

af41a52

Signed-off-by: Yaniv Eliash <yaniv@sre.engineer>

default cli variables

79939e2

Signed-off-by: Yaniv Eliash <yaniv@sre.engineer>

yaniveliash force-pushed the gh-1451 branch from 410c780 to 79939e2 Compare March 7, 2023 17:24

Swiddis requested changes Mar 7, 2023

View reviewed changes

scripts/integrations/main.py Outdated Show resolved Hide resolved

scripts/integrations/main.py Outdated Show resolved Hide resolved

derek-ho reviewed Mar 7, 2023

View reviewed changes

* Removed base_integration_path, now new integrations will be created at

8e5c3e1

the script's path * Changed file and directories path concatanation to os.path.join * Linter kicked in and formatted the file. Signed-off-by: Yaniv Eliash <yaniv@sre.engineer>

YANG-DB requested changes Mar 7, 2023

View reviewed changes

YANG-DB added the integration Integration project label Mar 7, 2023

YANG-DB requested review from YANG-DB, Swiddis and derek-ho and removed request for Swiddis March 8, 2023 20:01

YANG-DB requested changes Apr 2, 2023

View reviewed changes

Swiddis mentioned this pull request Apr 3, 2023

Add Initial Codebase for Integrations CLI #1488

Closed

9 tasks

Swiddis closed this Aug 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH 1451 #1454

GH 1451 #1454

yaniveliash commented Mar 7, 2023

ps48 commented Mar 7, 2023

yaniveliash commented Mar 7, 2023

YANG-DB commented Mar 7, 2023

Swiddis left a comment

derek-ho Mar 7, 2023

yaniveliash Mar 7, 2023

Swiddis Mar 7, 2023

derek-ho Mar 7, 2023

yaniveliash Mar 7, 2023

YANG-DB Apr 2, 2023

YANG-DB commented Mar 7, 2023

YANG-DB Mar 7, 2023

yaniveliash Mar 8, 2023

yaniveliash Mar 8, 2023

YANG-DB Mar 8, 2023

YANG-DB Mar 7, 2023

yaniveliash Mar 8, 2023

YANG-DB Mar 8, 2023

YANG-DB commented Mar 9, 2023

Swiddis commented Mar 14, 2023

YANG-DB Apr 2, 2023

Swiddis commented Aug 8, 2023

GH 1451 #1454

GH 1451 #1454

Conversation

yaniveliash commented Mar 7, 2023

Description

Issues Resolved

Check List

ps48 commented Mar 7, 2023

yaniveliash commented Mar 7, 2023

YANG-DB commented Mar 7, 2023

Swiddis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YANG-DB commented Mar 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YANG-DB commented Mar 9, 2023

Swiddis commented Mar 14, 2023

Choose a reason for hiding this comment

Swiddis commented Aug 8, 2023