Csv file getting started docker (#191)

* add csv upload file integration

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* update csv_file getting started docker content

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

---------

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

integrations/observability/csv_file/getting-started/Getting-Started.md

@@ -113,14 +113,14 @@ If you prefer to test this setup using Docker Compose, you can use the following
 Under the `getting-started` section you can examine a live docker-compose sample:
 ```yaml
 /csv_file/getting-started/fluent-bit
-|-- docker-complete.yml
+|-- docker-compose.yml
 |-- data/
  |-- fluent-bit.conf
  |-- parsers.conf
  |-- logs.csv
 
 ```
-Use the [docker-compose](fluent-bit/docker-complete.yml) you can find a complete:
+Use the [docker-compose](docker-compose.yml) you can find a complete:
 
 `docker compose -f docker-complete.yml up -d` would instantiate the services and start sending the csv sample logs into an index. 
 

integrations/observability/csv_file/getting-started/fluent-bit/docker-complete.yml → integrations/observability/csv_file/getting-started/docker-compose.yml

@@ -31,22 +31,20 @@ services:
  memlock:
  soft: -1
  hard: -1
- volumes:
- - opensearch-data:/usr/share/opensearch/data
  ports:
  - 9200:9200
  - 9600:9600
  networks:
  - opensearch-net
-
  fluent-bit:
- image: fluent/fluent-bit:latest
  container_name: fluent-bit
+ image: fluent/fluent-bit:latest
  volumes:
- - ./data:/fluent-bit/data
- command: ["/fluent-bit/bin/fluent-bit", "-c", "/fluent-bit/data/fluent-bit.conf"]
- depends_on:
- - opensearch-node1
+ - ./logs:/logs
+ - ./fluent-bit:/fluent-bit/etc
+ ports:
+ - "24224:24224"
+ - "24224:24224/udp"
  networks:
  - opensearch-net
 

integrations/observability/csv_file/getting-started/fluent-bit/data/fluent-bit.conf → integrations/observability/csv_file/getting-started/fluent-bit/fluent-bit.conf

@@ -1,13 +1,13 @@
 [SERVICE]
  Flush 1
- Log_Level info
+ Log_Level debug
  Parsers_File parsers.conf
 
 [INPUT]
- Name  tail
- Path  /fluent-bit/data/*.csv
- Parser  csv
- Tag  csv
+ Name tail
+ Path /logs/*.csv
+ Parser csv_log
+ Tag csv_logs
 
 [INPUT]
  Name dummy
@@ -25,4 +25,10 @@
  tls.verify Off
  Suppress_Type_Name On
  HTTP_User admin
- HTTP_Passwd my_%New%_passW0rd!@#
+ HTTP_Passwd my_%New%_passW0rd!@#
+ Trace_Output On
+ Trace_Error On
+
+[OUTPUT]
+ Name stdout
+ Match *

integrations/observability/csv_file/getting-started/fluent-bit/data/parsers.conf → integrations/observability/csv_file/getting-started/fluent-bit/parsers.conf

@@ -1,6 +1,6 @@
 [PARSER]
- Name csv
+ Name csv_log
  Format regex
- Regex ^(?<timestamp>[^,]+),(?<log_level>[^,]+),(?<message>[^,]+),(?<application>[^,]+),(?<host>[^,]+)$
+ Regex ^(?<timestamp>[^,]+),(?<log_level>[^,]+),(?<message>.*),(?<application>[^,]+),(?<host>[^,]+)$
  Time_Key timestamp
- Time_Format %Y-%m-%d %H:%M:%S
+ Time_Format %Y-%m-%d %H:%M:%S