Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

applicative logs index patterns getting started docker #192

Merged
merged 4 commits into from
Jul 31, 2024
Merged

applicative logs index patterns getting started docker #192

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
76c8a89
add csv upload file integration
YANG-DB Jul 17, 2024
99fc72f
Merge branch 'opensearch-project:main' into main
YANG-DB Jul 31, 2024
961b696
update csv_file getting started docker content
YANG-DB Jul 31, 2024
ab00b7b
update fluent-bit index names getting started docker content
YANG-DB Jul 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
0 .../csv_file/getting-started/fluent-bit/.env → ...servability/csv_file/getting-started/.env
View file
Open in desktop
File renamed without changes.
4 changes: 2 additions & 2 deletions integrations/observability/csv_file/getting-started/Getting-Started.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,14 +113,14 @@ If you prefer to test this setup using Docker Compose, you can use the following
Under the `getting-started` section you can examine a live docker-compose sample:
```yaml
/csv_file/getting-started/fluent-bit
|-- docker-complete.yml
|-- docker-compose.yml
|-- data/
|-- fluent-bit.conf
|-- parsers.conf
|-- logs.csv

```
Use the [docker-compose](fluent-bit/docker-complete.yml) you can find a complete:
Use the [docker-compose](docker-compose.yml) you can find a complete:

`docker compose -f docker-complete.yml up -d` would instantiate the services and start sending the csv sample logs into an index.

Expand Down
14 changes: 6 additions & 8 deletions ...ng-started/fluent-bit/docker-complete.yml → ...v_file/getting-started/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,22 +31,20 @@ services:
memlock:
soft: -1
hard: -1
volumes:
- opensearch-data:/usr/share/opensearch/data
ports:
- 9200:9200
- 9600:9600
networks:
- opensearch-net

fluent-bit:
image: fluent/fluent-bit:latest
container_name: fluent-bit
image: fluent/fluent-bit:latest
volumes:
- ./data:/fluent-bit/data
command: ["/fluent-bit/bin/fluent-bit", "-c", "/fluent-bit/data/fluent-bit.conf"]
depends_on:
- opensearch-node1
- ./logs:/logs
- ./fluent-bit:/fluent-bit/etc
ports:
- "24224:24224"
- "24224:24224/udp"
networks:
- opensearch-net

Expand Down
18 changes: 12 additions & 6 deletions ...g-started/fluent-bit/data/fluent-bit.conf → ...etting-started/fluent-bit/fluent-bit.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
[SERVICE]
Flush 1
Log_Level info
Log_Level debug
Parsers_File parsers.conf

[INPUT]
Name tail
Path /fluent-bit/data/*.csv
Parser csv
Tag csv
Name tail
Path /logs/*.csv
Parser csv_log
Tag csv_logs

[INPUT]
Name dummy
Expand All @@ -25,4 +25,10 @@
tls.verify Off
Suppress_Type_Name On
HTTP_User admin
HTTP_Passwd my_%New%_passW0rd!@#
HTTP_Passwd my_%New%_passW0rd!@#
Trace_Output On
Trace_Error On

[OUTPUT]
Name stdout
Match *
18 changes: 0 additions & 18 deletions integrations/observability/csv_file/getting-started/fluent-bit/fluent-bit.yml
View file
Open in desktop

This file was deleted.

6 changes: 3 additions & 3 deletions ...ting-started/fluent-bit/data/parsers.conf → ...e/getting-started/fluent-bit/parsers.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[PARSER]
Name csv
Name csv_log
Format regex
Regex ^(?<timestamp>[^,]+),(?<log_level>[^,]+),(?<message>[^,]+),(?<application>[^,]+),(?<host>[^,]+)$
Regex ^(?<timestamp>[^,]+),(?<log_level>[^,]+),(?<message>.*),(?<application>[^,]+),(?<host>[^,]+)$
Time_Key timestamp
Time_Format %Y-%m-%d %H:%M:%S
Time_Format %Y-%m-%d %H:%M:%S
0 .../getting-started/fluent-bit/data/logs.csv → ...ty/csv_file/getting-started/logs/logs.csv
View file
Open in desktop
File renamed without changes.
0 ...rted/fluent-bit/opensearch_dashboards.yml → ...getting-started/opensearch_dashboards.yml
View file
Open in desktop
File renamed without changes.
2 changes: 1 addition & 1 deletion integrations/observability/golang_client/getting-started/fluent-bit/fluent-bit.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
Match *
Port 9200
Type _doc
Index applicative_logs
Index applicative_logs-golang
tls On
tls.verify Off
Suppress_Type_Name On
Expand Down
2 changes: 1 addition & 1 deletion integrations/observability/java_client/getting-started/fluent-bit/fluent-bit.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
Match *
Port 9200
Type _doc
Index applicative_logs
Index applicative_logs-java
tls On
tls.verify Off
Suppress_Type_Name On
Expand Down
2 changes: 1 addition & 1 deletion integrations/observability/python_client/getting-started/fluent-bit/fluent-bit.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
Match *
Port 9200
Type _doc
Index applicative_logs
Index applicative_logs-python
tls On
tls.verify Off
Suppress_Type_Name On
Expand Down