add Nginx getting-started catalog component for setting up fluent-bit… #166

YANG-DB · 2024-06-26T21:17:16Z

Description

add Nginx getting-started catalog component for setting up fluent-bit agent for:

collecting
transforming
shipping Nginx

Also includes live docker compose sample to run for showcasing the instructions

The updated schema for the integration will contain an optional getting-started field containing:

    "workflows": [
      {
        "name": "fluent-bit",
        "description": "This is a fluent-bit based getting started instructions tutorial",
        "steps": [
          {
            "name": "Fluent-Bit Parser",
            "type": "console-cmd",
            "label": "Log Parsing",
            "info": "<info URL>",
            "description": "Setup Fluent-Bit parser config file parsing Nginx access log fields",
            "content": "[PARSER]\n    Name   apache\n    Format regex\n    Regex  ...."
             ... 
       
          },

Schema Fields :

Workflows contains a list of getting-started named workflows
Each workflow contains a list of named steps
Each step has a type, content, input-params and description, info, input-params
Type can be on of the following: console-cmd, file-url , image, video
Content holds the command to be run
Description describes the actual step task
Info links to an external file (in the file-url case) or to additional documentation in other cases
input-params is a set of user input that has to be replaced before running the command

Issues Resolved

#165

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

… agent for collecting, transforming and shipping Nginx access logs into opensearch Signed-off-by: YANGDB <yang.db.dev@gmail.com>

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

Swiddis · 2024-07-02T23:48:10Z

integrations/observability/nginx/getting-started/nginx-getting-started-1.0.0.json

+ {
+ "name": "Fluent-Bit Log Converter",
+ "label": "Log Parsing",
+ "info": "<info URL>",


Are these supposed to still be placeholders?

thanks - moved into draft util all the comments / questions are resolved

Swiddis · 2024-07-02T23:48:20Z

integrations/observability/nginx/getting-started/nginx-getting-started-1.0.0.json

+ ],
+ "liveDemo": {
+ "command": "docker-compose up -d",
+ "path": "<docker-compose-url>"


Another placeholder

thanks - moved into draft util all the comments / questions are resolved

integrations/observability/nginx/getting-started/opensearch_dashboards.yml

Swiddis · 2024-07-02T23:52:36Z

integrations/observability/nginx/nginx-1.0.0.json

+ "label": "Log Parsing",
+ "info": "<info URL>",
+ "description": "Setup Fluent-Bit logs converter lua script config file converting Nginx access log into Simple schema format",
+ "content": "local hexCharset = \"0123456789abcdef\"\nlocal function randHex(length)\n if length > 0 then\n local index = math.random(1, #hexCharset)\n return randHex(length - 1) .. hexCharset:sub(index, index)\n else\n return \"\"\n end\nend\n\nlocal function format_apache(c)\n return string.format(\n \"%s - %s [%s] \\\"%s %s HTTP/1.1\\\" %s %s\",\n c.host,\n c.user,\n os.date(\"%d/%b/%Y:%H:%M:%S %z\"),\n c.method,\n c.path,\n c.code,\n c.size\n )\nend\n\nlocal function format_nginx(c)\n return string.format(\n \"%s %s %s [%s] \\\"%s %s HTTP/1.1\\\" %s %s \\\"%s\\\" \\\"%s\\\"\",\n c.remote,\n c.host,\n c.user,\n os.date(\"%d/%b/%Y:%H:%M:%S %z\"),\n c.method,\n c.path,\n c.code,\n c.size,\n c.referer,\n c.agent\n )\nend\n\nlocal formats = {\n [\"apache.access\"] = format_apache,\n [\"nginx.access\"] = format_nginx\n}\n\nfunction convert_to_otel(tag, timestamp, record)\n local data = {\n traceId=randHex(32),\n spanId=randHex(16),\n [\"@timestamp\"]=os.date(\"!%Y-%m-%dT%H:%M:%S.000Z\"),\n observedTimestamp=os.date(\"!%Y-%m-%dT%H:%M:%S.000Z\"),\n body=formats[tag](record),\n attributes={\n data_stream={\n dataset=tag,\n namespace=\"production\",\n type=\"logs\"\n }\n },\n event={\n category=\"web\",\n name=\"access\",\n domain=tag,\n kind=\"event\",\n result=\"success\",\n type=\"access\"\n },\n http={\n request={\n method=record.method\n },\n response={\n bytes=tonumber(record.size),\n status_code=tonumber(record.code)\n },\n flavor=\"1.1\",\n url=record.path\n },\n communication={\n source={\n address=\"127.0.0.1\",\n ip=record.remote\n }\n }\n }\n return 1, timestamp, data\nend\n"


I think it's better for content to be a file ref like what we have for the other long-form content in integrations. Makes the config file smaller and easier to read, and makes an easier editing experience.

@Swiddis can u plz share an example for the above content ?

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

add getting started dashboard asset + markdown Signed-off-by: YANGDB <yang.db.dev@gmail.com>

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

- quickstart - connect to an existing collector Signed-off-by: YANGDB <yang.db.dev@gmail.com>

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

…on resource - getting started - getting_started.json - metadata - Getting-Started.md - markdown - sample.json - sample data Signed-off-by: YANGDB <yang.db.dev@gmail.com>

…on resource - getting started - getting_started.json - metadata - Getting-Started.md - markdown - data - sample data Signed-off-by: YANGDB <yang.db.dev@gmail.com>

Swiddis · 2024-07-30T15:54:46Z

integrations/observability/nginx/nginx-1.0.0.json

@@ -44,6 +44,7 @@
 { "name": "logs", "version": "1.0.0" }
 ],
 "assets": [
+ { "name": "getting-started-tutorial", "version": "1.0.0", "extension": "ndjson", "type": "savedObjectBundle", "workflows": ["dashboards"] },


What is the intended flow to present this to the user? Making it a saved object bundle associated with the integration means that it won't be added to the cluster until after they set up the integration.

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

add Nginx getting-started catalog component for setting up fluent-bit…

7f12d29

… agent for collecting, transforming and shipping Nginx access logs into opensearch Signed-off-by: YANGDB <yang.db.dev@gmail.com>

YANG-DB requested review from bowenlan-amzn, khushbr, praveensameneni, ananzh, Swiddis and AMoo-Miki as code owners June 26, 2024 21:19

YANG-DB added enhancement New feature or request labels Jun 27, 2024

TackAdam mentioned this pull request Jun 28, 2024

[FEATURE]Getting Started opensearch-project/dashboards-observability#1929

Open

YANG-DB added 4 commits July 2, 2024 13:34

Merge branch 'main' into getting-started-nginx

7e39bc9

update Getting started as an optional inner field for integration

44273f1

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

update Getting started as an optional inner field for integration

16e3e63

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

update Getting started as an optional inner field for integration

e57d67e

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

Swiddis requested changes Jul 2, 2024

View reviewed changes

YANG-DB marked this pull request as draft July 3, 2024 02:20

YANG-DB added 6 commits July 3, 2024 10:38

update Getting started with commands and files - url

03a68ff

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

update Getting started with commands and files - url

50b7cb7

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

update Getting started for nginx integration

f47a889

add getting started dashboard asset + markdown Signed-off-by: YANGDB <yang.db.dev@gmail.com>

update Getting started for nginx integration

9d5a0b2

add getting started dashboard asset + markdown Signed-off-by: YANGDB <yang.db.dev@gmail.com>

update Getting started for nginx integration

3bd2dfe

add getting started dashboard asset + markdown Signed-off-by: YANGDB <yang.db.dev@gmail.com>

add schema steps section for the getting started

ce178b3

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

YANG-DB added integration integration related content and removed enhancement New feature or request labels Jul 12, 2024

YANG-DB marked this pull request as ready for review July 12, 2024 02:44

YANG-DB requested a review from Swiddis July 12, 2024 02:44

YANG-DB added 4 commits July 15, 2024 16:06

update getting started with index patterns

c7dacea

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

update two workflows:

23c5268

- quickstart - connect to an existing collector Signed-off-by: YANGDB <yang.db.dev@gmail.com>

add savedObjectId to the gallery dashboard images links

f2596c7

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

update nginx assets for getting started

36cced8

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

YANG-DB added 4 commits July 24, 2024 12:02

fix json validity issue

f14853f

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

move getting-started into its own folder to become an independed ndjs…

9d99ef4

…on resource - getting started - getting_started.json - metadata - Getting-Started.md - markdown - sample.json - sample data Signed-off-by: YANGDB <yang.db.dev@gmail.com>

move getting-started into its own folder to become an independed ndjs…

c6f7995

…on resource - getting started - getting_started.json - metadata - Getting-Started.md - markdown - data - sample data Signed-off-by: YANGDB <yang.db.dev@gmail.com>

move getting-started into its own folder to become an independed ndjs…

4e22bf4

…on resource - getting started - getting_started.json - metadata - Getting-Started.md - markdown - data - sample data Signed-off-by: YANGDB <yang.db.dev@gmail.com>

Swiddis reviewed Jul 30, 2024

View reviewed changes

Swiddis approved these changes Jul 30, 2024

View reviewed changes

Merge branch 'main' into getting-started-nginx

4038f8b

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

YANG-DB merged commit 47cd435 into opensearch-project:main Jul 30, 2024
1 of 2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add Nginx getting-started catalog component for setting up fluent-bit… #166

add Nginx getting-started catalog component for setting up fluent-bit… #166

YANG-DB commented Jun 26, 2024 •

edited

Loading

Swiddis Jul 2, 2024

YANG-DB Jul 3, 2024

Swiddis Jul 2, 2024

YANG-DB Jul 3, 2024

Swiddis Jul 2, 2024

YANG-DB Jul 3, 2024

Swiddis Jul 30, 2024

add Nginx getting-started catalog component for setting up fluent-bit… #166

add Nginx getting-started catalog component for setting up fluent-bit… #166

Conversation

YANG-DB commented Jun 26, 2024 • edited Loading

Description

Schema Fields :

Issues Resolved

Swiddis Jul 2, 2024

Choose a reason for hiding this comment

YANG-DB Jul 3, 2024

Choose a reason for hiding this comment

Swiddis Jul 2, 2024

Choose a reason for hiding this comment

YANG-DB Jul 3, 2024

Choose a reason for hiding this comment

Swiddis Jul 2, 2024

Choose a reason for hiding this comment

YANG-DB Jul 3, 2024

Choose a reason for hiding this comment

Swiddis Jul 30, 2024

Choose a reason for hiding this comment

YANG-DB commented Jun 26, 2024 •

edited

Loading