From 59e5842acc2cb694f7d1f3fa64b84d028c93039f Mon Sep 17 00:00:00 2001
From: Matthew Bystedt <matthew.bystedt@gov.bc.ca>
Date: Mon, 9 Dec 2024 13:21:00 -0800
Subject: [PATCH] docs: update to bc gov
---
README.md | 6 ++---
docs/indextemplate.md | 10 ++++-----
docs/monitor.md | 8 +++----
docs/testing.md | 51 ++++++++++++++++++++++++++++++++++++++-----
4 files changed, 58 insertions(+), 17 deletions(-)
diff --git a/README.md b/README.md
index 103a56c..8493a38 100644
--- a/README.md
+++ b/README.md
@@ -2,7 +2,7 @@
[NR APM (Application Performance and Monitoring) Stack](https://apm.io.nrs.gov.bc.ca/_plugin/_dashboards) allows teams to tactically respond to potential issues and strategically investigate their KPIs. It is delivered using OpenSearch hosted on AWS. OpenSearch is a open source search and analytics suite derived from Elasticsearch & Kibana.
-<b>This README is for developers deploying NR APM Stack. See our Github site for [integration documentation](https://bcdevops.github.io/nr-apm-stack/).</b>
+<b>This README is for developers deploying NR APM Stack. See our Github site for [integration documentation](https://bcgov.github.io/nr-apm-stack/).</b>
# More Documentation
@@ -16,11 +16,11 @@ https://apps.nrs.gov.bc.ca/int/confluence/x/GaRvBQ
For developers and product owners, our integration documentation is located here:
-https://bcdevops.github.io/nr-apm-stack/
+https://bcgov.github.io/nr-apm-stack/
# Getting Started
-This project contains all the source code and supporting files for the APM Stack. It consists of a AWS SAM template, GitHub Actions and a Workflow CLI.
+This project contains all the source code and supporting files for the APM Stack. It consists of a AWS SAM template, GitHub Actions and a Workflow CLI.
AWS SAM is used to deploy the infrastructure on AWS. The infrastructure includes an AWS Lambda application that retrieves documents from an Kinesis endpoint, processes them and passes them on to OpenSearch. GitHub Actions are used to automate the deployment and maintaince of the product.
diff --git a/docs/indextemplate.md b/docs/indextemplate.md
index 0d4f7e7..3b90b7b 100644
--- a/docs/indextemplate.md
+++ b/docs/indextemplate.md
@@ -24,12 +24,12 @@ The 'nrm - (type) - (qualifier)' portion (or without the qualifier) will be refe
| nrm-audit-vault | Vault audit data | | |
| nrm-metrics | Server & Process metrics | | |
| nrm-tomcat-catalina | Tomcat catalina logs | | |
-| nrm-tomcat-localhost | Tomcat localhost logs | | |
+| nrm-tomcat-localhost | Tomcat localhost logs | | |
Aliases: The index name is always an alias as well.
## Index Lifecycle
-Generally, fresher data is examined more often so more resources are utilized to make that faster. All indices start on hot data nodes.
+Generally, fresher data is examined more often so more resources are utilized to make that faster. All indices start on hot data nodes.
| Index Name | Performance | Scale Down at | Merge at | Warm at | Delete at | Rollup |
|----------------------|-------------|---------------|----------|---------|-----------|--------|
@@ -62,7 +62,7 @@ Explanation of why certain qualifiers are used or not
| audit | (blank) | Never. Audit logs are specific to an application so a generic type would not make sense. |
## Standard Lifecycle Modifiers
-These modifiers are for edge cases where the long term cost of a subset of the data greatly outweighs the utility of keeping it around. Lifecycle modifiers are not recommended because fewer indices decreases the overall data size and speeds query response. A fair bit of analysis should go into any decision to use one.
+These modifiers are for edge cases where the long term cost of a subset of the data greatly outweighs the utility of keeping it around. Lifecycle modifiers are not recommended because fewer indices decreases the overall data size and speeds query response. A fair bit of analysis should go into any decision to use one.
| Modifier | Description |
|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
@@ -71,9 +71,9 @@ These modifiers are for edge cases where the long term cost of a subset of the d
## Index Format & Lifecycle Implementation
The above information is an adaption of the infrastructure code.
-Index templates: [https://github.com/BCDevOps/nr-apm-stack/tree/master/workflow-cli/configuration-opensearch/index_template](https://github.com/BCDevOps/nr-apm-stack/tree/master/workflow-cli/configuration-opensearch/index_template)
+Index templates: [https://github.com/bcgov/nr-apm-stack/tree/master/workflow-cli/configuration-opensearch/index_template](https://github.com/bcgov/nr-apm-stack/tree/master/workflow-cli/configuration-opensearch/index_template)
-Index policies: [https://github.com/BCDevOps/nr-apm-stack/tree/master/workflow-cli/configuration-opensearch/state_management_policy](https://github.com/BCDevOps/nr-apm-stack/tree/master/workflow-cli/configuration-opensearch/state_management_policy)
+Index policies: [https://github.com/bcgov/nr-apm-stack/tree/master/workflow-cli/configuration-opensearch/state_management_policy](https://github.com/bcgov/nr-apm-stack/tree/master/workflow-cli/configuration-opensearch/state_management_policy)
## Reference
[Troubleshooting Elasticsearch ILM: Common issues and fixes](https://www.elastic.co/blog/troubleshooting-elasticsearch-ilm-common-issues-and-fixes)
\ No newline at end of file
diff --git a/docs/monitor.md b/docs/monitor.md
index f9b096b..fb735bf 100644
--- a/docs/monitor.md
+++ b/docs/monitor.md
@@ -1,7 +1,7 @@
# Create a Monitor for Application/Service
## Add a new AWS SNS Topic if required
-- Config SNS Topic File: [topics.json](https://github.com/BCDevOps/nr-apm-stack/blob/main/terraform/topics.json)
+- Config SNS Topic File: [topics.json](https://github.com/bcgov/nr-apm-stack/blob/main/terraform/topics.json)
| Field Name | Description | Example |
|-------------|----------------------------------------------|-----------------------------|
@@ -16,13 +16,13 @@ After deployment the new SNS topic should be displayed in OpenSearch->Notificati
- Command to generate file monitors.json from [nr-funbucks](https://github.com/bcgov-nr/nr-funbucks): ./bin/dev monitors
-- Copy monitors.json to [nr-apm-stack](https://github.com/BCDevOps/nr-apm-stack) under terraform
+- Copy monitors.json to [nr-apm-stack](https://github.com/bcgov/nr-apm-stack) under terraform
-- PR for [nr-apm-stack](https://github.com/BCDevOps/nr-apm-stack) for OneTeam to review and deploy to AWS
+- PR for [nr-apm-stack](https://github.com/bcgov/nr-apm-stack) for OneTeam to review and deploy to AWS
## Create a Monitor for Application/Service with Terraform
-- Configuration File for application alerts: [app-alert.yaml](https://github.com/BCDevOps/nr-apm-stack/blob/main/terraform/app-alert.yaml)
+- Configuration File for application alerts: [app-alert.yaml](https://github.com/bcgov/nr-apm-stack/blob/main/terraform/app-alert.yaml)
- Add/Modify entries for monitors in the file with query and fields listed below
diff --git a/docs/testing.md b/docs/testing.md
index f870a51..3183a54 100644
--- a/docs/testing.md
+++ b/docs/testing.md
@@ -1,20 +1,20 @@
# Event Stream Processing Lambda
-Github: [BCDevOps/nr-apm-stack/event-stream-processing](https://github.com/BCDevOps/nr-apm-stack/tree/main/event-stream-processing)
+Github: [bcgov/nr-apm-stack/event-stream-processing](https://github.com/bcgov/nr-apm-stack/tree/main/event-stream-processing)
## Local Testing
The following will start an http server listening on port 3000.
```
-podman run --rm -p 3000:3000 ghcr.io/bcdevops/nr-apm-stack-lambda:main
+podman run --rm -p 3000:3000 ghcr.io/bcgov/nr-apm-stack-lambda:main
```
The root (/) will respond with the processed JSON. If for some reason you can't see the response (using Fluent Bit), you can have it print by setting the query parameter 'print' to be 'true' (?print=true).
### Sending Test Data - curl
-The simpliest way is to just use a curl command. Switch to the [event-stream-processing directory](https://github.com/BCDevOps/nr-apm-stack/tree/main/event-stream-processing) and run:
+The simpliest way is to just use a curl command. Switch to the [event-stream-processing directory](https://github.com/bcgov/nr-apm-stack/tree/main/event-stream-processing) and run:
```
curl -s -X POST -H "Content-Type: application/json" -d @samples/access-logs.json localhost:3000
@@ -24,8 +24,49 @@ or
curl -s -X POST -H "Content-Type: application/json" -d @samples/access-logs.json "http://localhost:3000?print=true"
```
+Note that `-d` stands for `data`, and in the above example, a file is being sent, in this case `samples/access-logs.json` file. The `@` symbol should be used when sending files. However if you want, you can send a json string as follows:
+
+```
+curl -s -X POST \
+ -H "Content-Type: application/json" \
+ -d '{"date":1698269530.952,"@timestamp":"2023-10-25T21:32:10.952Z","log.logger":"com.zaxxer.hikari.pool.HikariPool","host":{"name":["encsabcamlt1288"],"os":{}},"@metadata.keyAsPath":"true","agent.type":"fluentbit","agent.version":"2.1","event.sequence":29000,"message":"NrBeApiPool - Added connection oracle.jdbc.driver.T4CConnection@14abd185","log.file.path":"/logs/oracle-api.log","ecs.version":"8.8","organization.name":"TeamSPAR","event.category":"web","agent.name":"nr-spar-202","organization.id":"org@domain.bc.ca","service.type":"oracle_api","event.dataset":"application.log.utc","service.name":"spar_oracle_api","event.kind":"event","event.ingested":"diagnostic","service.environment":"development","labels.project":"spar-oracle-api","log.level":"INFO"}' \
+ localhost:3000
+```
+
## Testing with Funbucks
-Funbucks is tool for generating Fluent Bit templated configurations for servers and Kubernetes (OpenShift) deployments. The Fluent Bit configuration can be setup to read in a sample file and send to a locally running Event Stream Processing Lambda for testing.
+Funbucks is a tool for generating Fluent Bit templated configurations for servers and Kubernetes (OpenShift) deployments. The Fluent Bit configuration can be setup to read in a sample file and send to a locally running Event Stream Processing Lambda for testing.
+
+- First your should start up the Event Stream Processing Lambda as above (the http server listening on port 3000).
+- Then inside [nr-funbucks repo](https://github.com/bcgov-nr/nr-funbuck) you generate a configuration for your server using the `-l` flag.
+ - The `-l` flag tells funbucks to generate configuration files for a **local** Event Stream Processing Lambda
+ - You should run: `./bin/dev gen -s -l app_spar_oracle` where `app_spar_oracle` is your application configuration id.
+- Finally, you run Fluenbit either locally or in a container to send the output to the Event Stream Processing Lambda.
+
+In case you need, here's how you can build and run Fluentbit locally with Docker:
+
+Create a Dockerfile with this content:
+```sh
+FROM fluent/fluent-bit:2.1-debug
+ADD . /fluent-bit/etc/
+```
+
+Then create your docker image with:
+```sh
+docker build -t fluentbit-local .
+```
+
+And then run your image with:
+```sh
+docker run -ti --rm \
+ -e FLUENT_VERSION=2.1 \
+ -e AGENT_NAME=testing-agent \
+ -e FLUENT_CONF_HOME=/fluent-bit/etc/ \
+ --network host \
+ -v /logs:/logs \
+ fluentbit-local
+```
+
+Note that `/logs` it's referring to the application log path configuration, defined at the input.conf file.
-You simply start up the Event Stream Processing Lambda as above. In the Funbucks repo, you generate a configuration for your server using the '-l' flag. Finally, you run Fluenbit either locally or in a container to send the output to the Event Stream Processing Lambda. See the Funbucks repository for more details: https://github.com/bcgov-nr/nr-funbucks#readme
+See the Funbucks repository for more details: https://github.com/bcgov-nr/nr-funbucks#readme