add Nginx getting-started catalog component for setting up fluent-bit… (

#166) * add Nginx getting-started catalog component for setting up fluent-bit agent for collecting, transforming and shipping Nginx access logs into opensearch Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update Getting started as an optional inner field for integration Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update Getting started as an optional inner field for integration Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update Getting started as an optional inner field for integration Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update Getting started with commands and files - url Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update Getting started with commands and files - url Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update Getting started for nginx integration add getting started dashboard asset + markdown Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update Getting started for nginx integration add getting started dashboard asset + markdown Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update Getting started for nginx integration add getting started dashboard asset + markdown Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add schema steps section for the getting started Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update getting started with index patterns Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update two workflows: - quickstart - connect to an existing collector Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add savedObjectId to the gallery dashboard images links Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update nginx assets for getting started Signed-off-by: YANGDB <yang.db.dev@gmail.com> * fix json validity issue Signed-off-by: YANGDB <yang.db.dev@gmail.com> * move getting-started into its own folder to become an independed ndjson resource - getting started - getting_started.json - metadata - Getting-Started.md - markdown - sample.json - sample data Signed-off-by: YANGDB <yang.db.dev@gmail.com> * move getting-started into its own folder to become an independed ndjson resource - getting started - getting_started.json - metadata - Getting-Started.md - markdown - data - sample data Signed-off-by: YANGDB <yang.db.dev@gmail.com> * move getting-started into its own folder to become an independed ndjson resource - getting started - getting_started.json - metadata - Getting-Started.md - markdown - data - sample data Signed-off-by: YANGDB <yang.db.dev@gmail.com> --------- Signed-off-by: YANGDB <yang.db.dev@gmail.com>
opensearch-project · Jul 30, 2024 · 47cd435 · 47cd435
1 parent e73b657
commit 47cd435
Show file tree

Hide file tree

Showing 15 changed files with 2,165 additions and 3 deletions.
diff --git a/integrations/observability/nginx/assets/getting-started-tutorial-1.0.0.ndjson b/integrations/observability/nginx/assets/getting-started-tutorial-1.0.0.ndjson
@@ -0,0 +1,3 @@
+{"attributes":{"description":"Getting started tutorial for Nginx & Fluent Bit agent collecting access logs and shipping into opensearch after transformation into SS4O logs specification","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Nginx-Flunent-Bit-Getting-Started","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Nginx-Flunent-Bit-Getting-Started\",\"type\":\"markdown\",\"aggs\":[],\"params\":{\"fontSize\":12,\"openLinksInNewTab\":false,\"markdown\":\"# Getting Started with Nginx Ingestion using Fluent Bit\\n\\nThis tutorial covers two different setups processes for getting started with Nginx ingestion: a live example using Docker and a self-managed setup with code snippets.\\n\\n## Live Example using Docker\\n\\n### Step 1: Create Docker Network\\nBefore running any Docker Compose files, create the Docker network.\\n```sh\\ndocker network create opensearch-net\\n```\\n\\n**Description**:\\nCreate a Docker network named opensearch-net for the OpenSearch and fluent-bit containers to communicate.\\nUse this specific command if your existing `opensearch` & `opensearch-dashboards` are already running within a docker-compose container.\\n\\nIf `opensearch` & `opensearch-dashboards` are running outside of a container scope - for example in your `localhost`, change the original docker network definition Into the following:\\n```yaml\\n network_mode: host\\n```\\n\\n### Step 2: Setup Docker `.env` File\\nDownload and set up the environment variables.\\n```sh\\nwget https://raw.githubusercontent.com/opensearch-project/opensearch-catalog/main/integrations/observability/nginx/getting-started/.env\\n```\\n\\n**Description**:\\nThe .env file contains environment variables required for Docker Compose to configure the OpenSearch and Fluent-Bit containers.\\n\\nUpdate the following parameters:\\n\\n```yaml\\n# OpenSearch Node1\\nOPENSEARCH_PORT=9200\\nOPENSEARCH_HOST=opensearch\\nOPENSEARCH_ADDR=${OPENSEARCH_HOST}:${OPENSEARCH_PORT}\\n\\n# OpenSearch Dashboard\\nOPENSEARCH_DASHBOARD_PORT=5601\\nOPENSEARCH_DASHBOARD_HOST=opensearch-dashboards\\nOPENSEARCH_DASHBOARD_ADDR=${OPENSEARCH_DASHBOARD_HOST}:${OPENSEARCH_DASHBOARD_PORT}\\n```\\n\\nIf running `opensearch` & `opensearch-dashboards` are running outside of a container scope - also update the host names `OPENSEARCH_HOST`, `OPENSEARCH_DASHBOARD_HOST` appearing in the .env file to be able to recognize your local running services.\\n\\n### Step 3: Setup Fluent Bit Folder\\nDownload the Fluent Bit configuration files.\\n```sh\\nwget https://raw.githubusercontent.com/opensearch-project/opensearch-catalog/main/integrations/observability/nginx/getting-started/fluent-bit/fluent-bit.conf \\\\\\n https://raw.githubusercontent.com/opensearch-project/opensearch-catalog/main/integrations/observability/nginx/getting-started/fluent-bit/otel-converter.lua \\\\\\n https://raw.githubusercontent.com/opensearch-project/opensearch-catalog/main/integrations/observability/nginx/getting-started/fluent-bit/parsers.conf\\n```\\n\\n**Description**:\\nGet the local fluent-bit relevant config files.\\n- Update the `Host` field to match the `opensearch` location - in case its not a part of a docker-compose service, or host name as defined by the docker-compose running your server\\n- Update the `Index` field to match the index naming specification as defined by the [simple schema for observability](https://github.com/opensearch-project/opensearch-catalog/blob/main/docs/schema/observability/Naming-convention.md).\\n\\n```yaml\\n\\n[OUTPUT]\\n Name opensearch\\n Match nginx.access\\n Host opensearch-node1\\n Port 9200\\n Index ss4o_logs-nginx-prod\\n Suppress_Type_Name On\\n tls On\\n tls.verify Off\\n HTTP_User admin\\n HTTP_Passwd my_%New%_passW0rd!@#\\n\\n[OUTPUT]\\n Name opensearch\\n Match apache.access\\n Host opensearch-node1\\n Port 9200\\n Index ss4o_logs-apache-prod\\n Suppress_Type_Name On\\n tls On\\n tls.verify Off\\n HTTP_User admin\\n HTTP_Passwd my_%New%_passW0rd!@#\\n\\n```\\n\\n\\n### Step 4: Run Docker Compose\\nDownload and run the Docker Compose file for the Nginx live example.\\n\\n```sh\\nwget -O nginx-node.yml https://raw.githubusercontent.com/opensearch-project/opensearch-catalog/main/integrations/observability/nginx/getting-started/nginx-node.yml\\n\\ndocker-compose -f nginx-node.yml up -d\\n```\\n**Description**:\\nRun the nginx-node docker compose after updating the `networks` definition to accommodate your existing `opensearch` service.\\n```yaml\\nnetworks:\\n opensearch-net:\\n external: true\\n```\\n\\n## Self-Managed Setup\\n\\nThe next part describe the details for manually updating the `fluent-bit` agent for running along-side the `nginx` service and transforming its logs\\ninto `simple schema for observability ` compliant json to be ingested into opensearch.\\n> All the files are present in the `getting-started` folder of this integration.\\n\\n### Step 1: Fluent Bit Parser\\nSet up the Fluent Bit parser configuration to parse Nginx access log fields.\\n\\n**parsers.conf**\\n```ini\\n[PARSER]\\n Name apache\\n Format regex\\n Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \\\\[(?<time>[^\\\\]]*)\\\\] \\\"(?<method>\\\\S+)(?: +(?<path>[^\\\"]*?)(?: +\\\\S*)?)?\\\" (?<code>[^ ]*) (?<size>[^ ]*)(?: \\\"(?<referer>[^\\\"]*)\\\" \\\"(?<agent>[^\\\"]*)\\\")?$\\n Time_Key time\\n Time_Format %d/%b/%Y:%H:%M:%S %z\\n\\n[PARSER]\\n Name nginx\\n Format regex\\n Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \\\\[(?<time>[^\\\\]]*)\\\\] \\\"(?<method>\\\\S+)(?: +(?<path>[^\\\"]*?)(?: +\\\\S*)?)?\\\" (?<code>[^ ]*) (?<size>[^ ]*)(?: \\\"(?<referer>[^\\\"]*)\\\" \\\"(?<agent>[^\\\"]*)\\\")\\n Time_Key time\\n Time_Format %d/%b/%Y:%H:%M:%S %z\\n```\\n\\n### Step 2: Fluent Bit Log Converter\\nSet up the Fluent Bit logs converter Lua script to convert Nginx access logs into Simple schema format.\\n\\n**otel-converter.lua**\\n```lua\\nlocal hexCharset = \\\"0123456789abcdef\\\"\\nlocal function randHex(length)\\n if length > 0 then\\n local index = math.random(1, #hexCharset)\\n return randHex(length - 1) .. hexCharset:sub(index, index)\\n else\\n return \\\"\\\"\\n end\\nend\\n\\nlocal function format_apache(c)\\n return string.format(\\n \\\"%s - %s [%s] \\\\\\\"%s %s HTTP/1.1\\\\\\\" %s %s\\\",\\n c.host,\\n c.user,\\n os.date(\\\"%d/%b/%Y:%H:%M:%S %z\\\"),\\n c.method,\\n c.path,\\n c.code,\\n c.size\\n )\\nend\\n\\nlocal function format_nginx(c)\\n return string.format(\\n \\\"%s %s %s [%s] \\\\\\\"%s %s HTTP/1.1\\\\\\\" %s %s \\\\\\\"%s\\\\\\\" \\\\\\\"%s\\\\\\\"\\\",\\n c.remote,\\n c.host,\\n c.user,\\n os.date(\\\"%d/%b/%Y:%H:%M:%S %z\\\"),\\n c.method,\\n c.path,\\n c.code,\\n c.size,\\n c.referer,\\n c.agent\\n )\\nend\\n\\nlocal formats = {\\n [\\\"apache.access\\\"] = format_apache,\\n [\\\"nginx.access\\\"] = format_nginx\\n}\\n\\nfunction convert_to_otel(tag, timestamp, record)\\n local data = {\\n traceId=randHex(32),\\n spanId=randHex(16),\\n [\\\"@timestamp\\\"]=os.date(\\\"!%Y-%m-%dT%H:%M:%S.000Z\\\"),\\n observedTimestamp=os.date(\\\"!%Y-%m-%dT%H:%M:%S.000Z\\\"),\\n body=formats[tag](record),\\n attributes={\\n data_stream={\\n dataset=tag,\\n namespace=\\\"production\\\",\\n type=\\\"logs\\\"\\n }\\n },\\n event={\\n category=\\\"web\\\",\\n name=\\\"access\\\",\\n domain=tag,\\n kind=\\\"event\\\",\\n result=\\\"success\\\",\\n type=\\\"access\\\"\\n },\\n http={\\n request={\\n method=record.method\\n },\\n response={\\n bytes=tonumber(record.size),\\n status_code=tonumber(record.code)\\n },\\n flavor=\\\"1.1\\\",\\n url=record.path\\n },\\n communication={\\n source={\\n address=\\\"127.0.0.1\\\",\\n ip=record.remote\\n }\\n }\\n }\\n return 1, timestamp, data\\nend\\n```\\n\\n### Step 3: Fluent Bit Setup\\nSet up the Fluent Bit configuration file to include log parsing and OpenSearch access.\\n\\n**fluent-bit.conf**\\n```ini\\n[SERVICE]\\n Parsers_File parsers.conf\\n\\n[INPUT]\\n Name forward\\n Port 24224\\n\\n[FILTER]\\n Name parser\\n Match nginx.access\\n Key_Name log\\n Parser nginx\\n\\n[FILTER]\\n Name parser\\n Match apache.access\\n Key_Name log\\n Parser apache\\n\\n[Filter]\\n Name lua\\n Match *\\n Script otel-converter.lua\\n call convert_to_otel\\n\\n[OUTPUT]\\n Name opensearch\\n Match nginx.access\\n Host ${opensearch-node1}\\n Port 9200\\n Index ${ss4o_logs-nginx-prod}\\n Suppress_Type_Name On\\n tls On\\n tls.verify Off\\n HTTP_User admin\\n HTTP_Passwd my_%New%_passW0rd!@#\\n\\n[OUTPUT]\\n Name opensearch\\n Match apache.access\\n Host ${opensearch-node1}\\n Port 9200\\n Index ${ss4o_logs-nginx-prod}\\n Suppress_Type_Name On\\n tls On\\n tls.verify Off\\n HTTP_User admin\\n HTTP_Passwd my_%New%_passW0rd!@#\\n\\n[OUTPUT]\\n Name stdout\\n Match nginx.access\\n```\\n\\n**Description**:\\nGet the local fluent-bit relevant config files.\\n- Update the `Host` field to match the `opensearch` location - in case its not a part of a docker-compose service, or host name as defined by the docker-compose running your server\\n- Update the `Index` field to match the index naming specification as defined by the [simple schema for observability](https://github.com/opensearch-project/opensearch-catalog/blob/main/docs/schema/observability/Naming-convention.md).\\n\\n\"}}"},"id":"3331b760-3e40-11ef-a07c-d3f2df91ca50","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2024-07-09T22:11:32.949Z","version":"WzMwLDJd"}
+{"attributes":{"description":"Getting started tutorial for Nginx & Fluent Bit agent collecting access logs and shipping into opensearch after transformation into SS4O logs specification","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[]}"},"optionsJSON":"{\"hidePanelTitles\":false,\"useMargins\":true}","panelsJSON":"[{\"version\":\"2.15.0\",\"gridData\":{\"x\":0,\"y\":0,\"w\":24,\"h\":15,\"i\":\"7fa335a4-e06c-434c-8e7c-93a0f3b13f96\"},\"panelIndex\":\"7fa335a4-e06c-434c-8e7c-93a0f3b13f96\",\"embeddableConfig\":{},\"panelRefName\":\"panel_0\"}]","timeRestore":false,"title":"Getting-Started-Nginx-FluentBit-Dashboard","version":1},"id":"nginx-getting-started-tutorial-1.0.0","migrationVersion":{"dashboard":"7.9.3"},"references":[{"id":"3331b760-3e40-11ef-a07c-d3f2df91ca50","name":"panel_0","type":"visualization"}],"type":"dashboard","updated_at":"2024-07-09T22:11:55.435Z","version":"WzMxLDJd"}
+{"exportedCount":2,"missingRefCount":0,"missingReferences":[]}
diff --git a/integrations/observability/nginx/assets/nginx-1.0.0.ndjson b/integrations/observability/nginx/assets/nginx-1.0.0.ndjson
diff --git a/integrations/observability/nginx/assets/nginx-index-patterns-1.0.0.ndjson b/integrations/observability/nginx/assets/nginx-index-patterns-1.0.0.ndjson
diff --git a/integrations/observability/nginx/getting-started/.env b/integrations/observability/nginx/getting-started/.env
@@ -0,0 +1,18 @@
+# Nginx Proxy
+NGINX_PORT=90
+NGINX_ADDR=nginx:${NGINX_PORT}
+
+# OpenSearch version
+OPENSEARCH_VERSION=2.15.0
+OPENSEARCH_ADMIN_PASSWORD=my_%New%_passW0rd!@#
+OPENSEARCH_INITIAL_ADMIN_PASSWORD=my_%New%_passW0rd!@#
+
+# OpenSearch Node1
+OPENSEARCH_PORT=9200
+OPENSEARCH_HOST=opensearch
+OPENSEARCH_ADDR=${OPENSEARCH_HOST}:${OPENSEARCH_PORT}
+
+# OpenSearch Dashboard
+OPENSEARCH_DASHBOARD_PORT=5601
+OPENSEARCH_DASHBOARD_HOST=opensearch-dashboards
+OPENSEARCH_DASHBOARD_ADDR=${OPENSEARCH_DASHBOARD_HOST}:${OPENSEARCH_DASHBOARD_PORT}