docs: update to bc gov (#392)

bcgov · Dec 9, 2024 · aa3c983 · aa3c983
1 parent ec4105f
commit aa3c983
Show file tree

Hide file tree

Showing 4 changed files with 58 additions and 17 deletions.
diff --git a/README.md b/README.md
@@ -2,7 +2,7 @@
 
 [NR APM (Application Performance and Monitoring) Stack](https://apm.io.nrs.gov.bc.ca/_plugin/_dashboards) allows teams to tactically respond to potential issues and strategically investigate their KPIs. It is delivered using OpenSearch hosted on AWS. OpenSearch is a open source search and analytics suite derived from Elasticsearch & Kibana.
 
-<b>This README is for developers deploying NR APM Stack. See our Github site for [integration documentation](https://bcdevops.github.io/nr-apm-stack/).</b>
+<b>This README is for developers deploying NR APM Stack. See our Github site for [integration documentation](https://bcgov.github.io/nr-apm-stack/).</b>
 
 # More Documentation
 
@@ -16,11 +16,11 @@ https://apps.nrs.gov.bc.ca/int/confluence/x/GaRvBQ
 
 For developers and product owners, our integration documentation is located here:
 
-https://bcdevops.github.io/nr-apm-stack/
+https://bcgov.github.io/nr-apm-stack/
 
 # Getting Started
 
-This project contains all the source code and supporting files for the APM Stack. It consists of a AWS SAM template, GitHub Actions and a Workflow CLI. 
+This project contains all the source code and supporting files for the APM Stack. It consists of a AWS SAM template, GitHub Actions and a Workflow CLI.
 
 AWS SAM is used to deploy the infrastructure on AWS. The infrastructure includes an AWS Lambda application that retrieves documents from an Kinesis endpoint, processes them and passes them on to OpenSearch. GitHub Actions are used to automate the deployment and maintaince of the product.
 

diff --git a/docs/indextemplate.md b/docs/indextemplate.md
@@ -24,12 +24,12 @@ The 'nrm - (type) - (qualifier)' portion (or without the qualifier) will be refe
 | nrm-audit-vault      	| Vault audit data         	|            	|                                                                                      	|
 | nrm-metrics          	| Server & Process metrics 	|            	|                                                                                      	|
 | nrm-tomcat-catalina  	| Tomcat catalina logs     	|            	|                                                                                      	|
-| nrm-tomcat-localhost 	| Tomcat localhost logs    	|            	|                                                                                      	|	
+| nrm-tomcat-localhost 	| Tomcat localhost logs    	|            	|                                                                                      	|
 
 Aliases: The index name is always an alias as well.
 
 ## Index Lifecycle
-Generally, fresher data is examined more often so more resources are utilized to make that faster. All indices start on hot data nodes. 
+Generally, fresher data is examined more often so more resources are utilized to make that faster. All indices start on hot data nodes.
 
 | Index Name           | Performance | Scale Down at | Merge at | Warm at | Delete at | Rollup |
 |----------------------|-------------|---------------|----------|---------|-----------|--------|
@@ -62,7 +62,7 @@ Explanation of why certain qualifiers are used or not
 | audit | (blank)   | Never. Audit logs are specific to an application so a generic type would not make sense.                           |
 
 ## Standard Lifecycle Modifiers
-These modifiers are for edge cases where the long term cost of a subset of the data greatly outweighs the utility of keeping it around. Lifecycle modifiers are not recommended because fewer indices decreases the overall data size and speeds query response. A fair bit of analysis should go into any decision to use one. 
+These modifiers are for edge cases where the long term cost of a subset of the data greatly outweighs the utility of keeping it around. Lifecycle modifiers are not recommended because fewer indices decreases the overall data size and speeds query response. A fair bit of analysis should go into any decision to use one.
 
 | Modifier | Description                                                                                                                                                                                                                                                                  |
 |----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
@@ -71,9 +71,9 @@ These modifiers are for edge cases where the long term cost of a subset of the d
 ## Index Format & Lifecycle Implementation
 The above information is an adaption of the infrastructure code.
 
-Index templates: [https://github.com/BCDevOps/nr-apm-stack/tree/master/workflow-cli/configuration-opensearch/index_template](https://github.com/BCDevOps/nr-apm-stack/tree/master/workflow-cli/configuration-opensearch/index_template)
+Index templates: [https://github.com/bcgov/nr-apm-stack/tree/master/workflow-cli/configuration-opensearch/index_template](https://github.com/bcgov/nr-apm-stack/tree/master/workflow-cli/configuration-opensearch/index_template)
 
-Index policies: [https://github.com/BCDevOps/nr-apm-stack/tree/master/workflow-cli/configuration-opensearch/state_management_policy](https://github.com/BCDevOps/nr-apm-stack/tree/master/workflow-cli/configuration-opensearch/state_management_policy)
+Index policies: [https://github.com/bcgov/nr-apm-stack/tree/master/workflow-cli/configuration-opensearch/state_management_policy](https://github.com/bcgov/nr-apm-stack/tree/master/workflow-cli/configuration-opensearch/state_management_policy)
 
 ## Reference
 [Troubleshooting Elasticsearch ILM: Common issues and fixes](https://www.elastic.co/blog/troubleshooting-elasticsearch-ilm-common-issues-and-fixes)
diff --git a/docs/monitor.md b/docs/monitor.md
@@ -1,7 +1,7 @@
 # Create a Monitor for Application/Service
 
 ## Add a new AWS SNS Topic if required
-- Config SNS Topic File: [topics.json](https://github.com/BCDevOps/nr-apm-stack/blob/main/terraform/topics.json)
+- Config SNS Topic File: [topics.json](https://github.com/bcgov/nr-apm-stack/blob/main/terraform/topics.json)
 
 | Field Name  | Description                                  | Example                     |
 |-------------|----------------------------------------------|-----------------------------|
@@ -16,13 +16,13 @@ After deployment the new SNS topic should be displayed in OpenSearch->Notificati
 
 - Command to generate file monitors.json from [nr-funbucks](https://github.com/bcgov-nr/nr-funbucks): ./bin/dev monitors
 
-- Copy monitors.json to [nr-apm-stack](https://github.com/BCDevOps/nr-apm-stack) under terraform
+- Copy monitors.json to [nr-apm-stack](https://github.com/bcgov/nr-apm-stack) under terraform
 
-- PR for [nr-apm-stack](https://github.com/BCDevOps/nr-apm-stack) for OneTeam to review and deploy to AWS
+- PR for [nr-apm-stack](https://github.com/bcgov/nr-apm-stack) for OneTeam to review and deploy to AWS
 
 ## Create a Monitor for Application/Service with Terraform
 
-- Configuration File for application alerts: [app-alert.yaml](https://github.com/BCDevOps/nr-apm-stack/blob/main/terraform/app-alert.yaml)
+- Configuration File for application alerts: [app-alert.yaml](https://github.com/bcgov/nr-apm-stack/blob/main/terraform/app-alert.yaml)
 
 - Add/Modify entries for monitors in the file with query and fields listed below
 

diff --git a/docs/testing.md b/docs/testing.md
@@ -1,20 +1,20 @@
 # Event Stream Processing Lambda
 
-Github: [BCDevOps/nr-apm-stack/event-stream-processing](https://github.com/BCDevOps/nr-apm-stack/tree/main/event-stream-processing)
+Github: [bcgov/nr-apm-stack/event-stream-processing](https://github.com/bcgov/nr-apm-stack/tree/main/event-stream-processing)
 
 ## Local Testing
 
 The following will start an http server listening on port 3000.
 
 ```
-podman run --rm -p 3000:3000 ghcr.io/bcdevops/nr-apm-stack-lambda:main
+podman run --rm -p 3000:3000 ghcr.io/bcgov/nr-apm-stack-lambda:main
 ```
 
 The root (/) will respond with the processed JSON. If for some reason you can't see the response (using Fluent Bit), you can have it print by setting the query parameter 'print' to be 'true' (?print=true).
 
 ### Sending Test Data - curl
 
-The simpliest way is to just use a curl command. Switch to the [event-stream-processing directory](https://github.com/BCDevOps/nr-apm-stack/tree/main/event-stream-processing) and run:
+The simpliest way is to just use a curl command. Switch to the [event-stream-processing directory](https://github.com/bcgov/nr-apm-stack/tree/main/event-stream-processing) and run:
 
 ```
 curl -s -X POST -H "Content-Type: application/json" -d @samples/access-logs.json localhost:3000
@@ -24,8 +24,49 @@ or
 curl -s -X POST -H "Content-Type: application/json" -d @samples/access-logs.json "http://localhost:3000?print=true"
 ```
 
+Note that `-d` stands for `data`, and in the above example, a file is being sent, in this case `samples/access-logs.json` file. The `@` symbol should be used when sending files. However if you want, you can send a json string as follows:
+
+```
+curl -s -X POST \
+  -H "Content-Type: application/json" \
+  -d '{"date":1698269530.952,"@timestamp":"2023-10-25T21:32:10.952Z","log.logger":"com.zaxxer.hikari.pool.HikariPool","host":{"name":["encsabcamlt1288"],"os":{}},"@metadata.keyAsPath":"true","agent.type":"fluentbit","agent.version":"2.1","event.sequence":29000,"message":"NrBeApiPool - Added connection oracle.jdbc.driver.T4CConnection@14abd185","log.file.path":"/logs/oracle-api.log","ecs.version":"8.8","organization.name":"TeamSPAR","event.category":"web","agent.name":"nr-spar-202","organization.id":"org@domain.bc.ca","service.type":"oracle_api","event.dataset":"application.log.utc","service.name":"spar_oracle_api","event.kind":"event","event.ingested":"diagnostic","service.environment":"development","labels.project":"spar-oracle-api","log.level":"INFO"}' \
+  localhost:3000
+```
+
 ## Testing with Funbucks
 
-Funbucks is tool for generating Fluent Bit templated configurations for servers and Kubernetes (OpenShift) deployments. The Fluent Bit configuration can be setup to read in a sample file and send to a locally running Event Stream Processing Lambda for testing.
+Funbucks is a tool for generating Fluent Bit templated configurations for servers and Kubernetes (OpenShift) deployments. The Fluent Bit configuration can be setup to read in a sample file and send to a locally running Event Stream Processing Lambda for testing.
+
+- First your should start up the Event Stream Processing Lambda as above (the http server listening on port 3000).
+- Then inside [nr-funbucks repo](https://github.com/bcgov-nr/nr-funbuck) you generate a configuration for your server using the `-l` flag.
+  - The `-l` flag tells funbucks to generate configuration files for a **local** Event Stream Processing Lambda
+  - You should run: `./bin/dev gen -s -l app_spar_oracle` where `app_spar_oracle` is your application configuration id.
+- Finally, you run Fluenbit either locally or in a container to send the output to the Event Stream Processing Lambda.
+
+In case you need, here's how you can build and run Fluentbit locally with Docker:
+
+Create a Dockerfile with this content:
+```sh
+FROM fluent/fluent-bit:2.1-debug
+ADD . /fluent-bit/etc/
+```
+
+Then create your docker image with:
+```sh
+docker build -t fluentbit-local .
+```
+
+And then run your image with:
+```sh
+docker run -ti --rm \
+ -e FLUENT_VERSION=2.1 \
+ -e AGENT_NAME=testing-agent \
+ -e FLUENT_CONF_HOME=/fluent-bit/etc/ \
+ --network host \
+ -v /logs:/logs \
+ fluentbit-local
+```
+
+Note that `/logs` it's referring to the application log path configuration, defined at the input.conf file.
 
-You simply start up the Event Stream Processing Lambda as above. In the Funbucks repo, you generate a configuration for your server using the '-l' flag. Finally, you run Fluenbit either locally or in a container to send the output to the Event Stream Processing Lambda. See the Funbucks repository for more details: https://github.com/bcgov-nr/nr-funbucks#readme
+See the Funbucks repository for more details: https://github.com/bcgov-nr/nr-funbucks#readme