diff --git a/integrations/observability/csv_file/README.md b/integrations/observability/csv_file/README.md new file mode 100644 index 0000000..d20db83 --- /dev/null +++ b/integrations/observability/csv_file/README.md @@ -0,0 +1,38 @@ +# CSV Upload Integration + +> CSV File based Upload Integration + +## What is CSV Upload ? + +CSV upload is an example of parsing and loading a CSV file into opensearch index using an agent + +## What is CSV Integration ? + +An integration is a bundle of pre-canned assets which are bundled togather in a meaningful manner. + +**_CSV Upload_** integration includes docker live example including getting started instruction of using data-prepper or fluent-bit for +uploading the csv file into a dedicated index using a parser to transform the csv into json + +## Ingesting CVS Using Data-Prepper + ... + +## Ingesting CVS Using Flunet-Bit + +... + +--- +## Loading Integrations via DashboardManagement + +To update an integration template navigate to the DashboardManagement and select [savedObjects](https://localhost:5601/_dashboards/app/management/opensearch-dashboards/objects) and import the new artifact: + +1) Download the `nginx-1.0.0.ndjson` artifact from the [catalog release page](https://github.com/opensearch-project/opensearch-catalog/releases/edit/nginx-1.0.0) + +2) Go to the [DashboardManagement -> savedObjects ](https://localhost:5601/_dashboards/app/management/opensearch-dashboards/objects) + +![](https://github.com/opensearch-project/opensearch-catalog/assets/48943349/d96e9a78-e3de-4cce-ba66-23f7c084778d) + +![](https://github.com/opensearch-project/opensearch-catalog/assets/48943349/a63ae102-706a-4980-b758-fff7f6b24a94) + +3) Once there select import to load the recently downloaded integration artifact (`nginx-1.0.0.ndjson` suffix) + +4) Open the [nginx integration](https://localhost:5601/app/integrations#/available/nginx) and install diff --git a/integrations/observability/csv_file/assets/fluent-bit-csv-upload-1.0.0.ndjson b/integrations/observability/csv_file/assets/fluent-bit-csv-upload-1.0.0.ndjson new file mode 100644 index 0000000..b04c765 --- /dev/null +++ b/integrations/observability/csv_file/assets/fluent-bit-csv-upload-1.0.0.ndjson @@ -0,0 +1,4 @@ +{"attributes":{"description":"upload a csv file example using fluent-bit agent","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"fluent-bit-csv-upload","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"fluent-bit-csv-upload\",\"type\":\"markdown\",\"aggs\":[],\"params\":{\"fontSize\":12,\"openLinksInNewTab\":false,\"markdown\":\"# Uploading a CSV File into an OpenSearch Index Using Fluent Bit\\n\\nThis tutorial will guide you through the process of setting up Fluent Bit to monitor a directory for CSV files and upload their contents into an OpenSearch index.\\n\\n## Prerequisites\\n\\n- An OpenSearch instance running and accessible.\\n- Fluent Bit installed on your system.\\n- A directory containing your CSV files.\\n\\n## Step 1: Install Fluent Bit\\n\\n### On Linux:\\n\\n```bash\\ncurl -L https://fluentbit.io/releases/1.8/fluent-bit-1.8.11-linux-x86_64.tar.gz -o fluent-bit.tar.gz\\ntar -xvf fluent-bit.tar.gz\\ncd fluent-bit/bin\\n```\\n\\n### On macOS:\\n\\n```bash\\nbrew install fluent-bit\\n```\\n\\n### On Windows:\\n\\nDownload and extract Fluent Bit from [Fluent Bit releases](https://fluentbit.io/download/).\\n\\n## Step 2: Create Fluent Bit Configuration Files\\n\\n#### Create `fluent-bit.conf`\\n\\nThis is the main configuration file for Fluent Bit. It defines the input source, parser, and output destination.\\n\\n```ini\\n[SERVICE]\\n Flush 1\\n Log_Level info\\n Parsers_File parsers.conf\\n\\n[INPUT]\\n Name tail\\n Path /path/to/your/csv/files/*.csv\\n Parser csv\\n Tag csv\\n Refresh_Interval 5\\n Rotate_Wait 30\\n\\n[OUTPUT]\\n Name opensearch\\n Match *\\n Host your-opensearch-host\\n Port 9200\\n Index csv-index\\n HTTP_User your-username\\n HTTP_Passwd your-password\\n tls off\\n Suppress_Type_Name On\\n tls.verify off\\n```\\n\\n### Create `parsers.conf`\\n\\nThis file defines the CSV parser.\\n\\n```ini\\n[PARSER]\\n Name csv\\n Format regex\\n Regex ^(?[^,]+),(?[^,]+),(?[^,]+),(?[^,]+),(?[^,]+)$\\n Time_Key timestamp\\n Time_Format %Y-%m-%d %H:%M:%S\\n```\\n\\n### Direct the CSV folder location\\n\\nEnsure this file is in the directory you specified in the `Path` of the `fluent-bit.conf` file.\\n\\n\\n## Step 3: Run Fluent Bit\\n\\nNavigate to the directory containing the Fluent Bit executable and the configuration files. Then, start Fluent Bit with the configuration files.\\n\\n```bash\\n/path/to/fluent-bit/bin/fluent-bit -c /path/to/fluent-bit.conf\\n```\\n\\n## Step 4: Verify Data in OpenSearch\\n\\nAfter starting Fluent Bit, you can verify the data ingestion by accessing OpenSearch and searching for the `csv-index` index.\\n\\nFor example, you can use OpenSearch Dashboards or the OpenSearch API to query the index:\\n\\n### Using OpenSearch Dashboards:\\n\\n1. Open OpenSearch Dashboards in your browser.\\n2. Navigate to the \\\"Discover\\\" tab.\\n3. Select the `csv-index` index pattern.\\n4. Verify that the log data from your CSV files is being ingested and displayed.\\n\\n### Using the OpenSearch API:\\n\\n```bash\\ncurl -X GET \\\"http://your-opensearch-host:9200/csv-index/_search?pretty\\\"\\n```\\n\\n---\\n## Live Testing with Docker Compose\\nIf you prefer to test this setup using Docker Compose, you can use the following docker-compose.yml file to quickly set up an OpenSearch instance along with Fluent Bit:\\n\\nUnder the `getting-started` section you can examine a live docker-compose sample:\\n```yaml\\n/csv_file/getting-started/fluent-bit\\n|-- docker-complete.yml\\n|-- data/\\n |-- fluent-bit.conf\\n |-- parsers.conf\\n |-- logs.csv\\n\\n```\\nUse the [docker-compose](../getting-started/fluent-bit/docker-complete.yml) you can find a complete:\\n\\n`docker compose -f docker-complete.yml up -d` would instantiate the services and start sending the csv sample logs into an index. \\n\"}}"},"id":"0fad8910-43d9-11ef-a69e-0549ba61487e","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2024-07-17T16:55:31.713Z","version":"WzEsMV0="} +{"attributes":{"description":"upload a csv file example using fluent-bit agent","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[]}"},"optionsJSON":"{\"hidePanelTitles\":false,\"useMargins\":true}","panelsJSON":"[{\"version\":\"2.15.0\",\"gridData\":{\"x\":0,\"y\":0,\"w\":24,\"h\":15,\"i\":\"22a1a11f-7ecf-46c7-a73d-b6cb5eb07b45\"},\"panelIndex\":\"22a1a11f-7ecf-46c7-a73d-b6cb5eb07b45\",\"embeddableConfig\":{},\"panelRefName\":\"panel_0\"}]","timeRestore":false,"title":"csv-file-upload-fluent-bit-dashboard","version":1},"id":"1e4f1c40-43d9-11ef-a69e-0549ba61487e","migrationVersion":{"dashboard":"7.9.3"},"references":[{"id":"0fad8910-43d9-11ef-a69e-0549ba61487e","name":"panel_0","type":"visualization"}],"type":"dashboard","updated_at":"2024-07-17T16:55:31.713Z","version":"WzIsMV0="} +{"attributes":{"fields":"[{\"count\":0,\"name\":\"@timestamp\",\"type\":\"date\",\"esTypes\":[\"date\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"_id\",\"type\":\"string\",\"esTypes\":[\"_id\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"count\":0,\"name\":\"_index\",\"type\":\"string\",\"esTypes\":[\"_index\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"count\":0,\"name\":\"_score\",\"type\":\"number\",\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"_source\",\"type\":\"_source\",\"esTypes\":[\"_source\"],\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"_type\",\"type\":\"string\",\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"application\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"application.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"application\"}}},{\"count\":0,\"name\":\"host\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"host.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"host\"}}},{\"count\":0,\"name\":\"log_level\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"log_level.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"log_level\"}}},{\"count\":0,\"name\":\"message\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"message.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"message\"}}},{\"count\":0,\"name\":\"timestamp\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"timestamp.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"timestamp\"}}}]","timeFieldName":"@timestamp","title":"logs-index"},"id":"csv-getting-started-tutorial-1.0.0","migrationVersion":{"index-pattern":"7.6.0"},"references":[],"type":"index-pattern","updated_at":"2024-07-17T16:59:06.006Z","version":"WzMsMV0="} +{"exportedCount":3,"missingRefCount":0,"missingReferences":[]} \ No newline at end of file diff --git a/integrations/observability/csv_file/csv_file-1.0.0.json b/integrations/observability/csv_file/csv_file-1.0.0.json new file mode 100644 index 0000000..6740801 --- /dev/null +++ b/integrations/observability/csv_file/csv_file-1.0.0.json @@ -0,0 +1,46 @@ +{ + "name": "csv", + "version": "1.0.0", + "displayName": "csv-file", + "description": "Upload a CSV file", + "license": "Apache-2.0", + "type": "logs", + "labels": ["Logs", "Unstructured"], + "author": "OpenSearch", + "sourceUrl": "https://github.com/opensearch-project/opensearch-catalog/tree/main/integrations/observability/csv_file", + "workflows": [ + { + "name": "dashboards", + "label": "Dashboards & Visualizations", + "description": "Dashboards and indices that enable you to easily visualize important metrics.", + "enabled_by_default": false + } + ], + "statics": { + "logo": { + "annotation": "CSV Logo", + "path": "logo.svg" + }, + "gallery": [ + { + "annotation": "Fluent-Bit getting started tutorial", + "path": "fluent-bit-getting-started-dashboard.png" + }, + { + "annotation": "Data-Prepper Logo", + "path": "data-pepper.png" + }, + { + "annotation": "Fluent-Bit Logo", + "path": "fluentbit.png" + } + ] + }, + "components": [], + "assets": [ + { "name": "fluent-bit-csv-upload", "version": "1.0.0", "extension": "ndjson", "type": "savedObjectBundle", "workflows": ["dashboards"] } + ], + "sampleData": { + "path": "logs.csv" + } +} \ No newline at end of file diff --git a/integrations/observability/csv_file/data/logs.csv b/integrations/observability/csv_file/data/logs.csv new file mode 100644 index 0000000..d0136c7 --- /dev/null +++ b/integrations/observability/csv_file/data/logs.csv @@ -0,0 +1,10 @@ +2024-07-16 12:00:00,INFO,Application started successfully,App1,host1 +2024-07-16 12:01:00,DEBUG,User logged in,App1,host1 +2024-07-16 12:01:05,ERROR,Failed to load resource,App1,host1 +2024-07-16 12:02:00,WARN,Deprecated API used,App1,host1 +2024-07-16 12:03:00,INFO,Background job executed,App1,host1 +2024-07-16 12:04:00,DEBUG,Cache cleared,App1,host1 +2024-07-16 12:05:00,INFO,User logged out,App1,host1 +2024-07-16 12:06:00,ERROR,Database connection failed,App1,host1 +2024-07-16 12:07:00,INFO,Application shutdown initiated,App1,host1 +2024-07-16 12:08:00,INFO,Application shutdown completed,App1,host1 diff --git a/integrations/observability/csv_file/getting-started/Getting-Started.md b/integrations/observability/csv_file/getting-started/Getting-Started.md new file mode 100644 index 0000000..9ac46bc --- /dev/null +++ b/integrations/observability/csv_file/getting-started/Getting-Started.md @@ -0,0 +1,189 @@ +# Uploading a CSV File into an OpenSearch Index Using Fluent Bit + +This tutorial will guide you through the process of setting up Fluent Bit to monitor a directory for CSV files and upload their contents into an OpenSearch index. + +## Prerequisites + +- An OpenSearch instance running and accessible. +- Fluent Bit installed on your system. +- A directory containing your CSV files. + +## Step 1: Install Fluent Bit + +### On Linux: + +```bash +curl -L https://fluentbit.io/releases/1.8/fluent-bit-1.8.11-linux-x86_64.tar.gz -o fluent-bit.tar.gz +tar -xvf fluent-bit.tar.gz +cd fluent-bit/bin +``` + +### On macOS: + +```bash +brew install fluent-bit +``` + +### On Windows: + +Download and extract Fluent Bit from [Fluent Bit releases](https://fluentbit.io/download/). + +## Step 2: Create Fluent Bit Configuration Files + +#### Create `fluent-bit.conf` + +This is the main configuration file for Fluent Bit. It defines the input source, parser, and output destination. + +```ini +[SERVICE] + Flush 1 + Log_Level info + Parsers_File parsers.conf + +[INPUT] + Name tail + Path /path/to/your/csv/files/*.csv + Parser csv + Tag csv + Refresh_Interval 5 + Rotate_Wait 30 + +[OUTPUT] + Name opensearch + Match * + Host your-opensearch-host + Port 9200 + Index csv-index + HTTP_User your-username + HTTP_Passwd your-password + tls off + Suppress_Type_Name On + tls.verify off +``` + +### Create `parsers.conf` + +This file defines the CSV parser. + +```ini +[PARSER] + Name csv + Format regex + Regex ^(?[^,]+),(?[^,]+),(?[^,]+),(?[^,]+),(?[^,]+)$ + Time_Key timestamp + Time_Format %Y-%m-%d %H:%M:%S +``` + +### Direct the CSV folder location + +Ensure this file is in the directory you specified in the `Path` of the `fluent-bit.conf` file. + + +## Step 3: Run Fluent Bit + +Navigate to the directory containing the Fluent Bit executable and the configuration files. Then, start Fluent Bit with the configuration files. + +```bash +/path/to/fluent-bit/bin/fluent-bit -c /path/to/fluent-bit.conf +``` + +## Step 4: Verify Data in OpenSearch + +After starting Fluent Bit, you can verify the data ingestion by accessing OpenSearch and searching for the `csv-index` index. + +For example, you can use OpenSearch Dashboards or the OpenSearch API to query the index: + +### Using OpenSearch Dashboards: + +1. Open OpenSearch Dashboards in your browser. +2. Navigate to the "Discover" tab. +3. Select the `csv-index` index pattern. +4. Verify that the log data from your CSV files is being ingested and displayed. + +### Using the OpenSearch API: + +```bash +curl -X GET "http://your-opensearch-host:9200/csv-index/_search?pretty" +``` + +--- +## Live Testing with Docker Compose +If you prefer to test this setup using Docker Compose, you can use the following docker-compose.yml file to quickly set up an OpenSearch instance along with Fluent Bit: + +Under the `getting-started` section you can examine a live docker-compose sample: +```yaml +/csv_file/getting-started/fluent-bit +|-- docker-complete.yml +|-- data/ + |-- fluent-bit.conf + |-- parsers.conf + |-- logs.csv + +``` +Use the [docker-compose](fluent-bit/docker-complete.yml) you can find a complete: + +`docker compose -f docker-complete.yml up -d` would instantiate the services and start sending the csv sample logs into an index. + +--- +# Data-Prepper CSV Processor Tutorial + +The `csv` processor parses comma-separated values (CSVs) from the event into columns. + +## Configuration Options + +- **source** (String): The field in the event to be parsed. Default is `message`. +- **quote_character** (String): The text qualifier for a single column. Default is `"`. +- **delimiter** (String): The character separating each column. Default is `,`. +- **delete_header** (Boolean): Deletes the event header after parsing. Default is true. +- **column_names_source_key** (String): Specifies the CSV column names. +- **column_names** (List): User-specified column names. + +## Usage Examples + +### User-specified Column Names + +```yaml +csv-pipeline: + source: + file: + path: "/full/path/to/ingest.csv" + record_type: "event" + processor: + - csv: + column_names: ["col1", "col2"] + sink: + - stdout: +``` + +### Automatically Detect Column Names + +```yaml +csv-s3-pipeline: + source: + s3: + notification_type: "sqs" + codec: + newline: + skip_lines: 1 + header_destination: "header" + compression: none + sqs: + queue_url: "https://sqs..amazonaws.com//" + aws: + region: "" + processor: + - csv: + column_names_source_key: "header" + sink: + - stdout: +``` + +## Metrics + +- **recordsIn**: Ingress records count. +- **recordsOut**: Egress records count. +- **timeElapsed**: Execution time. +- **csvInvalidEvents**: Count of invalid events. + +For more details, visit the [CSV Processor Documentation](https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/processors/csv/). +``` \ No newline at end of file diff --git a/integrations/observability/csv_file/getting-started/data-prepper/pipeline.yml b/integrations/observability/csv_file/getting-started/data-prepper/pipeline.yml new file mode 100644 index 0000000..fdd4181 --- /dev/null +++ b/integrations/observability/csv_file/getting-started/data-prepper/pipeline.yml @@ -0,0 +1,20 @@ +# Copyright The OpenTelemetry Authors +# SPDX-License-Identifier: Apache-2.0 + +csv-pipeline: + source: + file: + path: "/full/path/to/ingest.csv" + record_type: "event" + processor: + - csv: + column_names: ["col1", "col2", "col3"] + sink: + - opensearch: + hosts: ["https://opensearch-node1:9200"] + username: "admin" + password: "my_%New%_passW0rd!@#" + insecure: true + index_type: custom + index: logs-index + bulk_size: 4 diff --git a/integrations/observability/csv_file/getting-started/fluent-bit/.env b/integrations/observability/csv_file/getting-started/fluent-bit/.env new file mode 100644 index 0000000..60ee804 --- /dev/null +++ b/integrations/observability/csv_file/getting-started/fluent-bit/.env @@ -0,0 +1,18 @@ +# Nginx Proxy +NGINX_PORT=90 +NGINX_ADDR=nginx:${NGINX_PORT} + +# OpenSearch version +OPENSEARCH_VERSION=2.15.0 +OPENSEARCH_ADMIN_PASSWORD=my_%New%_passW0rd!@# +OPENSEARCH_INITIAL_ADMIN_PASSWORD=my_%New%_passW0rd!@# + +# OpenSearch Node1 +OPENSEARCH_PORT=9200 +OPENSEARCH_HOST=opensearch +OPENSEARCH_ADDR=${OPENSEARCH_HOST}:${OPENSEARCH_PORT} + +# OpenSearch Dashboard +OPENSEARCH_DASHBOARD_PORT=5601 +OPENSEARCH_DASHBOARD_HOST=opensearch-dashboards +OPENSEARCH_DASHBOARD_ADDR=${OPENSEARCH_DASHBOARD_HOST}:${OPENSEARCH_DASHBOARD_PORT} diff --git a/integrations/observability/csv_file/getting-started/fluent-bit/data/fluent-bit.conf b/integrations/observability/csv_file/getting-started/fluent-bit/data/fluent-bit.conf new file mode 100644 index 0000000..5be828a --- /dev/null +++ b/integrations/observability/csv_file/getting-started/fluent-bit/data/fluent-bit.conf @@ -0,0 +1,28 @@ +[SERVICE] + Flush 1 + Log_Level info + Parsers_File parsers.conf + +[INPUT] + Name tail + Path /fluent-bit/data/*.csv + Parser csv + Tag csv + +[INPUT] + Name dummy + Dummy {"timestamp":"2024-07-16 12:09:00", "log_level":"INFO", "message":"Dummy log message", "application":"App2", "host":"host2"} + Tag dummy + +[OUTPUT] + Name opensearch + Host opensearch-node1 + Match * + Port 9200 + Type _doc + Index logs-index + tls On + tls.verify Off + Suppress_Type_Name On + HTTP_User admin + HTTP_Passwd my_%New%_passW0rd!@# \ No newline at end of file diff --git a/integrations/observability/csv_file/getting-started/fluent-bit/data/logs.csv b/integrations/observability/csv_file/getting-started/fluent-bit/data/logs.csv new file mode 100644 index 0000000..d0136c7 --- /dev/null +++ b/integrations/observability/csv_file/getting-started/fluent-bit/data/logs.csv @@ -0,0 +1,10 @@ +2024-07-16 12:00:00,INFO,Application started successfully,App1,host1 +2024-07-16 12:01:00,DEBUG,User logged in,App1,host1 +2024-07-16 12:01:05,ERROR,Failed to load resource,App1,host1 +2024-07-16 12:02:00,WARN,Deprecated API used,App1,host1 +2024-07-16 12:03:00,INFO,Background job executed,App1,host1 +2024-07-16 12:04:00,DEBUG,Cache cleared,App1,host1 +2024-07-16 12:05:00,INFO,User logged out,App1,host1 +2024-07-16 12:06:00,ERROR,Database connection failed,App1,host1 +2024-07-16 12:07:00,INFO,Application shutdown initiated,App1,host1 +2024-07-16 12:08:00,INFO,Application shutdown completed,App1,host1 diff --git a/integrations/observability/csv_file/getting-started/fluent-bit/data/parsers.conf b/integrations/observability/csv_file/getting-started/fluent-bit/data/parsers.conf new file mode 100644 index 0000000..056a359 --- /dev/null +++ b/integrations/observability/csv_file/getting-started/fluent-bit/data/parsers.conf @@ -0,0 +1,6 @@ +[PARSER] + Name csv + Format regex + Regex ^(?[^,]+),(?[^,]+),(?[^,]+),(?[^,]+),(?[^,]+)$ + Time_Key timestamp + Time_Format %Y-%m-%d %H:%M:%S diff --git a/integrations/observability/csv_file/getting-started/fluent-bit/docker-complete.yml b/integrations/observability/csv_file/getting-started/fluent-bit/docker-complete.yml new file mode 100644 index 0000000..64d0525 --- /dev/null +++ b/integrations/observability/csv_file/getting-started/fluent-bit/docker-complete.yml @@ -0,0 +1,58 @@ +version: '3.8' + +services: + opensearch-dashboards: + image: opensearchproject/opensearch-dashboards:${OPENSEARCH_VERSION} + container_name: opensearch-dashboards + ports: + - 5601:5601 + environment: + OPENSEARCH_HOSTS: '["https://opensearch-node1:9200"]' + depends_on: + - opensearch-node1 + volumes: + - ./opensearch_dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml + networks: + - opensearch-net + + opensearch-node1: + image: opensearchproject/opensearch:${OPENSEARCH_VERSION} + container_name: opensearch-node1 + environment: + - cluster.name=my-cluster + - node.name=opensearch-node1 + - discovery.seed_hosts=opensearch-node1 + - cluster.initial_master_nodes=opensearch-node1 + - bootstrap.memory_lock=true + - plugins.query.datasources.encryption.masterkey=8e3f206ea7c07cc1bfc5cf40 + - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" + - "OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_ADMIN_PASSWORD}" + ulimits: + memlock: + soft: -1 + hard: -1 + volumes: + - opensearch-data:/usr/share/opensearch/data + ports: + - 9200:9200 + - 9600:9600 + networks: + - opensearch-net + + fluent-bit: + image: fluent/fluent-bit:latest + container_name: fluent-bit + volumes: + - ./data:/fluent-bit/data + command: ["/fluent-bit/bin/fluent-bit", "-c", "/fluent-bit/data/fluent-bit.conf"] + depends_on: + - opensearch-node1 + networks: + - opensearch-net + +volumes: + opensearch-data: + +networks: + opensearch-net: + driver: bridge diff --git a/integrations/observability/csv_file/getting-started/fluent-bit/fluent-bit.yml b/integrations/observability/csv_file/getting-started/fluent-bit/fluent-bit.yml new file mode 100644 index 0000000..1897c8b --- /dev/null +++ b/integrations/observability/csv_file/getting-started/fluent-bit/fluent-bit.yml @@ -0,0 +1,18 @@ +version: '3.8' + +services: + fluent-bit: + image: fluent/fluent-bit:latest + container_name: fluent-bit + volumes: + - ./data:/fluent-bit/data + command: ["/fluent-bit/bin/fluent-bit", "-c", "/fluent-bit/data/fluent-bit.conf"] + networks: + - opensearch-net + +volumes: + opensearch-data: + +networks: + opensearch-net: + driver: bridge diff --git a/integrations/observability/csv_file/getting-started/fluent-bit/opensearch_dashboards.yml b/integrations/observability/csv_file/getting-started/fluent-bit/opensearch_dashboards.yml new file mode 100644 index 0000000..e386ddb --- /dev/null +++ b/integrations/observability/csv_file/getting-started/fluent-bit/opensearch_dashboards.yml @@ -0,0 +1,10 @@ +opensearch.hosts: ["https://opensearch-node1:9200"] +server.host: 0.0.0.0 +opensearch.ssl.verificationMode: none +opensearch.username: "admin" +opensearch.password: "my_%New%_passW0rd!@#" +opensearch.requestHeadersWhitelist: [ authorization,securitytenant ] +opensearch_security.multitenancy.enabled: false +opensearch_security.multitenancy.tenants.preferred: ["Private", "Global"] +opensearch_security.readonly_mode.roles: ["kibana_read_only"] +vis_type_vega.enableExternalUrls: true diff --git a/integrations/observability/csv_file/getting-started/getting-started-csv_file-1.0.0.json b/integrations/observability/csv_file/getting-started/getting-started-csv_file-1.0.0.json new file mode 100644 index 0000000..49d3303 --- /dev/null +++ b/integrations/observability/csv_file/getting-started/getting-started-csv_file-1.0.0.json @@ -0,0 +1,157 @@ +{ + "name": "csv", + "version": "1.0.0", + "displayName": "csv-file", + "description": "Upload a CSV file", + "license": "Apache-2.0", + "type": "logs", + "labels": ["Logs", "Unstructured"], + "author": "OpenSearch", + "sourceUrl": "https://github.com/opensearch-project/opensearch-catalog/tree/main/integrations/observability/getting-started/csv_file", + "statics": { + "logo": { + "annotation": "CSV Logo", + "path": "logo.svg" + }, + "gallery": [ + { + "annotation": "Fluent-Bit getting started tutorial", + "path": "fluent-bit-getting-started-dashboard.png", + "tags":["dashboard","tutorial"], + "savedObject-id": "csv-getting-started-tutorial-1.0.0" + }, + { + "annotation": "Data-Prepper Logo", + "path": "data-pepper.png" + }, + { + "annotation": "Fluent-Bit Logo", + "path": "fluentbit.png" + } + ] + }, + "assets": [ + { "name": "fluent-bit-csv-upload", "version": "1.0.0", "extension": "ndjson", "type": "savedObjectBundle", "workflows": ["dashboards"] } + ], + "getting-started": { + "info": "https://github.com/opensearch-project/opensearch-catalog/tree/main/integrations/observability/csv_file/info/Getting-Started.md", + "ingestion": ["fluent-bit","data-prepper"], + "structured": "true", + "technology": "csv", + "protocol": [], + "live-sample": "true", + "workflows": [ + { + "name": "Fluent-bit file upload", + "description": "This is a fluent-bit based CSV file upload getting started instructions tutorial", + "steps": [ + { + "name": "Fluent-Bit Parser", + "type": "console-cmd", + "phase": "ingestion", + "label": "Log Parsing", + "info": ["https://github.com/opensearch-project/opensearch-catalog/tree/main/integrations/observability/csv_file/getting-started/fluent-bit/data/parsers.conf"], + "description": "Setup Fluent-Bit parser config file parsing csv file", + "content": "[PARSER]\n Name csv\n Format regex\n Regex ^(?[^,]+),(?[^,]+),(?[^,]+),(?[^,]+),(?[^,]+)$\n Time_Key timestamp\n Time_Format %Y-%m-%d %H:%M:%S\n" + }, + { + "name": "Fluent-Bit Setup", + "type": "console-cmd", + "phase": "ingestion", + "label": "Agent Set-Up", + "info": ["https://github.com/opensearch-project/opensearch-catalog/tree/main/integrations/observability/csv_file/getting-started/fluent-bit/data/fluent-bit.conf"], + "description": "Setup Fluent-Bit conf file including logs parsing and OpenSearch access", + "input-params": [ + { + "name": "opensearch-node1", + "description": "Opensearch host", + "type": "host-name" + }, + { + "name": "ss4o_logs-nginx-prod", + "description": "logs sink index name", + "type": "index-name" + } + ], + "content": "[SERVICE]\n Flush 1\n Log_Level info\n Parsers_File parsers.conf\n\n[INPUT]\n Name tail\n Path /fluent-bit/data/*.csv\n Parser csv\n Tag csv\n\n[INPUT]\n Name dummy\n Dummy {\"timestamp\":\"2024-07-16 12:09:00\", \"log_level\":\"INFO\", \"message\":\"Dummy log message\", \"application\":\"App2\", \"host\":\"host2\"}\n Tag dummy\n\n[OUTPUT]\n Name opensearch\n Host opensearch-node1\n Match *\n Port 9200\n Type _doc\n Index logs-index\n tls On\n tls.verify Off\n Suppress_Type_Name On\n HTTP_User admin\n HTTP_Passwd my_%New%_passW0rd!@#" + } + ] + }, + { + "name": "QuickStart", + "description": "This is a docker-composed based getting started live example with CSV file upload", + "steps": [ + { + "name": "Create docker-network", + "label": "Environment setting", + "phase": "docker", + "type": "console-cmd", + "content": "docker network create opensearch-net", + "description": "Before running any docker-compose files, create the opensearch-net network" + }, + { + "name": "Setup docker .env file", + "label": "Environment Parameters", + "phase": "docker", + "type": "file-url", + "input-params": [ + { + "name": "OPENSEARCH_HOST", + "description": "Opensearch host", + "type": "host-name" + }, + { + "name": "OPENSEARCH_DASHBOARD_HOST", + "description": "Opensearch Dashboard host", + "type": "host-name" + } + ], + "info": ["https://github.com/opensearch-project/opensearch-catalog/tree/main/integrations/csv_file/getting-started/fluent-bit/.env"], + "content": "wget https://raw.githubusercontent.com/opensearch-catalog/tree/main/integrations/csv_file/getting-started/fluent-bit/.env", + "description": "Setup docker-compose env variables " + }, + { + "name": "Setup fluent-bit folder", + "label": "Environment Parameters", + "phase": "docker", + "type": "file-url", + "info": ["https://github.com/opensearch-project/opensearch-catalog/tree/main/integrations/csv_file/getting-started/fluent-bit/data/fluent-bit.conf", + "https://github.com/opensearch-project/opensearch-catalog/tree/main/integrations/csv_file/getting-started/fluent-bit/data/parsers.conf" + ], + "content": "wget https://raw.githubusercontent.com/opensearch-project/opensearch-catalog/main/integrations/csv_file/getting-started/fluent-bit/data/fluent-bit.conf \\\n https://raw.githubusercontent.com/opensearch-project/opensearch-catalog/main/integrations/observability/csv_file/getting-started/fluent-bit/data/parsers.conf\n", + "description": "Setup docker-compose fluent-bit's service configuration " + }, + { + "name": "Run docker-compose", + "label": "live container", + "type": "file-url", + "phase": "docker", + "info": ["https://github.com/opensearch-project/opensearch-catalog/tree/main/integrations/observability/csv_file/getting-started/fluent-bit.yml"], + "description": "Run docker-compose for the nginx live example and see the generated index", + "content": "wget -O fluent-bit.yml https://raw.githubusercontent.com/opensearch-project/opensearch-catalog/main/integrations/observability/csv_file/getting-started/fluent-bit.yml && docker-compose -f fluent-bit.yml up -d\n" + } + ] + } + ] + }, + "schema": [ + { + "type": "csv-logs", + "info": ["https://opensearch.org/docs/latest/im-plugin/index-templates"], + "content": "PUT _index_template/csv_logs_template", + "description": "Setup sample csv based schema logs index template in the dev console", + "index-template": "https://github.com/opensearch-project/opensearch-catalog/tree/main/integrations/observability/csv_file/schema/csv-logs-1.0.0.mapping.json", + "index-pattern-name": "logs-*" + } + ], + "index-patterns": { + "type": ["logs"], + "info": ["https://opensearch.org/docs/latest/dashboards/management/index-patterns/"], + "description": "Import index patterns `.ndjson` file to the saved objects", + "index-pattern": "https://github.com/opensearch-project/opensearch-catalog/tree/main/integrations/observability/csv_file/assets/fluent-bit-csv-upload-1.0.0.ndjson", + "index-patterns-name": ["logs-*"] + }, + "sampleData": { + "path": "logs.csv" + } +} \ No newline at end of file diff --git a/integrations/observability/csv_file/schema/csv-logs-1.0.0.mapping.json b/integrations/observability/csv_file/schema/csv-logs-1.0.0.mapping.json new file mode 100644 index 0000000..72af9ec --- /dev/null +++ b/integrations/observability/csv_file/schema/csv-logs-1.0.0.mapping.json @@ -0,0 +1,29 @@ +{ + "index_patterns": ["logs-*"], + "template": { + "settings": { + "number_of_shards": 1, + "number_of_replicas": 1 + }, + "mappings": { + "properties": { + "timestamp": { + "type": "date", + "format": "yyyy-MM-dd HH:mm:ss" + }, + "log_level": { + "type": "keyword" + }, + "message": { + "type": "text" + }, + "application": { + "type": "keyword" + }, + "host": { + "type": "keyword" + } + } + } + } +} diff --git a/integrations/observability/csv_file/static/data-pepper.png b/integrations/observability/csv_file/static/data-pepper.png new file mode 100644 index 0000000..9d69397 Binary files /dev/null and b/integrations/observability/csv_file/static/data-pepper.png differ diff --git a/integrations/observability/csv_file/static/fluent-bit-getting-started-dashboard.png b/integrations/observability/csv_file/static/fluent-bit-getting-started-dashboard.png new file mode 100644 index 0000000..b3a87ae Binary files /dev/null and b/integrations/observability/csv_file/static/fluent-bit-getting-started-dashboard.png differ diff --git a/integrations/observability/csv_file/static/fluentbit.png b/integrations/observability/csv_file/static/fluentbit.png new file mode 100644 index 0000000..aae238e Binary files /dev/null and b/integrations/observability/csv_file/static/fluentbit.png differ diff --git a/integrations/observability/csv_file/static/logo.png b/integrations/observability/csv_file/static/logo.png new file mode 100644 index 0000000..d9bf6b5 Binary files /dev/null and b/integrations/observability/csv_file/static/logo.png differ