Csv file getting started docker (#191)

* add csv upload file integration

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* update csv_file getting started docker content

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

---------

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

integrations/observability/csv_file/getting-started/Getting-Started.md

@@ -113,14 +113,14 @@ If you prefer to test this setup using Docker Compose, you can use the following
 Under the `getting-started` section you can examine a live docker-compose sample:
 ```yaml
 /csv_file/getting-started/fluent-bit
-|-- docker-complete.yml
+|-- docker-compose.yml
 |-- data/
     |-- fluent-bit.conf
     |-- parsers.conf
     |-- logs.csv
 
 ```
-Use the [docker-compose](fluent-bit/docker-complete.yml) you can find a complete:
+Use the [docker-compose](docker-compose.yml) you can find a complete:
 
 `docker compose -f docker-complete.yml up -d` would instantiate the services and start sending the csv sample logs into an index. 
 

integrations/observability/csv_file/getting-started/fluent-bit/docker-complete.yml → integrations/observability/csv_file/getting-started/docker-compose.yml

@@ -31,22 +31,20 @@ services:
       memlock:
         soft: -1
         hard: -1
-    volumes:
-      - opensearch-data:/usr/share/opensearch/data
     ports:
       - 9200:9200
       - 9600:9600
     networks:
       - opensearch-net
-
   fluent-bit:
-    image: fluent/fluent-bit:latest
     container_name: fluent-bit
+    image: fluent/fluent-bit:latest
     volumes:
-      - ./data:/fluent-bit/data
-    command: ["/fluent-bit/bin/fluent-bit", "-c", "/fluent-bit/data/fluent-bit.conf"]
-    depends_on:
-      - opensearch-node1
+      - ./logs:/logs
+      - ./fluent-bit:/fluent-bit/etc
+    ports:
+      - "24224:24224"
+      - "24224:24224/udp"
     networks:
       - opensearch-net
 

integrations/observability/csv_file/getting-started/fluent-bit/data/fluent-bit.conf → integrations/observability/csv_file/getting-started/fluent-bit/fluent-bit.conf

@@ -1,13 +1,13 @@
 [SERVICE]
     Flush        1
-    Log_Level    info
+    Log_Level    debug
     Parsers_File parsers.conf
 
 [INPUT]
-    Name         tail
-    Path         /fluent-bit/data/*.csv
-    Parser       csv
-    Tag          csv
+    Name        tail
+    Path        /logs/*.csv
+    Parser      csv_log
+    Tag         csv_logs
 
 [INPUT]
     Name         dummy
@@ -25,4 +25,10 @@
     tls.verify      Off
     Suppress_Type_Name On
     HTTP_User       admin
-    HTTP_Passwd     my_%New%_passW0rd!@#
+    HTTP_Passwd     my_%New%_passW0rd!@#
+    Trace_Output    On
+    Trace_Error     On
+
+[OUTPUT]
+    Name   stdout
+    Match  *

integrations/observability/csv_file/getting-started/fluent-bit/data/parsers.conf → integrations/observability/csv_file/getting-started/fluent-bit/parsers.conf

@@ -1,6 +1,6 @@
 [PARSER]
-    Name        csv
+    Name        csv_log
     Format      regex
-    Regex       ^(?<timestamp>[^,]+),(?<log_level>[^,]+),(?<message>[^,]+),(?<application>[^,]+),(?<host>[^,]+)$
+    Regex       ^(?<timestamp>[^,]+),(?<log_level>[^,]+),(?<message>.*),(?<application>[^,]+),(?<host>[^,]+)$
     Time_Key    timestamp
-    Time_Format %Y-%m-%d %H:%M:%S
+    Time_Format %Y-%m-%d %H:%M:%S