Skip to content

Commit

Permalink
Merge pull request #1375 from hunch7/kaa-1581-mongo-master
Browse files Browse the repository at this point in the history 
KAA-1581: MongoDB log appender - reviewed. [master]
  • Loading branch information
@vadimol
vadimol authored Dec 16, 2016
2 parents f39d41f + db38143 commit 2439b32
Show file tree
Hide file tree
Showing 2 changed files with 120 additions and 142 deletions.
2 changes: 1 addition & 1 deletion ...ramming-guide/Key-platform-features/Data-collection/Flume-log-appender/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -380,7 +380,7 @@ This Avro schema is obtained trough the [server REST API call]({{root_url}}Progr

The example below uses the **Data collection demo** from [Kaa Sandbox]({{root_url}}Glossary/#kaa-sandbox).

To play around with the log appender:
To play around with the log appender:

1. Open the Data collection demo from Kaa Sandbox, follow the application installation instructions, and run the application.

Expand Down
260 changes: 119 additions & 141 deletions ...mming-guide/Key-platform-features/Data-collection/MongoDB-log-appender/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
layout: page
title: MongoDB log appender
permalink: /:path/
nav: /:path/Programming-guide/Key-platform-features/Data-collection/MongoDB-log-appender
sort_idx: 70
---

Expand All @@ -11,101 +10,36 @@ sort_idx: 70
* TOC
{:toc}

The MongoDB log appender is responsible for transferring logs from the Operations service to the MongoDB database.
The logs are stored in the table named `logs\_$applicationToken`, where `$applicationToken` matches the [token of the current application]({{root_url}}Glossary/#application-token).
The MongoDB log appender is used to transfer logs from the [Operations service]({{root_url}}Glossary/#operations-service) to the [MongoDB](https://www.mongodb.com/) database.
The logs are stored in the `logs\_$applicationToken` table, where `$applicationToken` matches the [token of the current application]({{root_url}}Glossary/#application-token).

# Creating MongoDB log appender in Admin UI
## Create MongoDB log appender

The easiest way to create a MongoDB log appender for your application is by using Admin UI.
To create a MongoDB log appender for your application using the [Administration UI]({{root_url}}Glossary/#administration-ui):

To create a log appender of the MongoDB storage type, do the following:
1. Log in to the **Administration UI** page as a [tenant developer]({{root_url}}Glossary/#tenant-developer).

1. In the **Log appenders** window, click **Add log appender**.
2. Enter the log appender name and description, select the minimum and maximum supported log schema version, and select necessary log metadata fields.
3. Set the log appender type to _MongoDB_.
![Add log appender in Admin UI](attach/add-log-appender-in-admin-ui.png)
4. Fill in the MongoDB log appender [configuration](#configuration) form.
5. Click **Add** button. Log appender is ready and operational at this point.
![MongoDB log appender configuration part 1](attach/MongoDB-log-appender-configuration1.png)
![MongoDB log appender configuration part 2](attach/MongoDB-log-appender-configuration2.png)
2. Click **Applications** and open the **Log appenders** page of your application.
Click **Add log appender**.

# Creating MongoDB log appender with Admin REST API
3. On the **Log appender details** page, enter the necessary information and set the **Type** field to **MongoDB**.
![Add log appender in Admin UI](attach/add-log-appender-in-admin-ui.png)

It is also possible to create a MongoDB log appender for your application by using [Admin REST API]({{root_url}}Programming-guide/Server-REST-APIs/#!/Logging/editLogAppender).
The following example illustrates how to create the MongoDB log appender via Admin Rest API.
4. Fill in the **Configuration** section for your log appender and click **Add**.
See [Configure log appender](#configure-log-appender).

## Configuration
![MongoDB log appender configuration part 1](attach/MongoDB-log-appender-configuration1.png)
![MongoDB log appender configuration part 2](attach/MongoDB-log-appender-configuration2.png)

The MongoDB log appender configuration must match to
[this]({{github_url}}server/appenders/mongo-appender/src/main/avro/mongodb-appender-config.avsc) Avro schema.
Alternatively, you can use the [server REST API]({{root_url}}Programming-guide/Server-REST-APIs/#!/Logging/editLogAppender) to create or edit your MongoDB log appender.

Fields of avro schema:

* MongoDB nodes - list of MongoDB hosts.
* Authentication credentials - credentials used to authenticate on MongoDB cluster.
* Other fields which configure of connection to MongoDB.

|name |description |
|-----------------------|-----------------------------------------------------------------------|
|dbName |Name of the database |
|connectionsPerHost |Max number of connections per host |
|maxWaitTime |Max wait time for connection in milliseconds |
|connectionTimeout |Connection timeout in milliseconds |
|socketTimeout |Socket timeout in milliseconds |
|socketKeepalive |Turn on socket keep alive (boolean value) |
|includeClientProfile |Whether to include client-side endpoint profile data (boolean value) |
|includeServerProfile |Whether to include server-side endpoint profile data (boolean value) |

An example configuration that matches to previously introduced Avro schema is as below:

```json
{
"mongoServers":[
{
"host":"127.0.0.1",
"port":27017
}
],
"mongoCredentials":[
{
"user":"user",
"password":"password"
}
],
"dbName":"kaa",
"connectionsPerHost":{
"int":30
},
"maxWaitTime":{
"int":120000
},
"connectionTimeout":{
"int":5000
},
"socketTimeout":{
"int":0
},
"socketKeepalive":{
"boolean":false
},
"includeClientProfile":{
"boolean":false
},
"includeServerProfile":{
"boolean":false
}
}
```

## Administration

The following Admin REST API call example illustrates how to create a new MongoDB log appender.
The following example illustrates how to create an instance of MongoDB log appender using the server REST API.

```bash
curl -v -S -u devuser:devuser123 -X POST -H 'Content-Type: application/json' -d @mongoDBLogAppender.json "http://localhost:8080/kaaAdmin/rest/api/logAppender" | python -mjson.tool
```

where file `mongoDBLogAppender.json` contains the following data:
where file `mongoDBLogAppender.json` contains the following data.

```
{
Expand All @@ -114,7 +48,7 @@ where file `mongoDBLogAppender.json` contains the following data:
"applicationId":"5",
"applicationToken":"82635305199158071549",
"name":"Sample MongoDB log appender",
"description":"Sample MngoDB log appender",
"description":"Sample MongoDB log appender",
"headerStructure":[
"KEYHASH",
"VERSION",
Expand All @@ -129,7 +63,7 @@ where file `mongoDBLogAppender.json` contains the following data:
}
```

Example result:
Below is an example result.

```json
{
Expand Down Expand Up @@ -157,12 +91,76 @@ Example result:
}
```

# Playing with MongoDB log appender
## Configure log appender

The MongoDB log appender configuration must match [this Avro schema]({{github_url}}server/appenders/mongo-appender/src/main/avro/mongodb-appender-config.avsc).

You can configure the following log appender settings:

We'll use [Data collection demo](https://github.com/kaaproject/sample-apps/tree/master/datacollection/source) from Kaa Sandbox. Our example will send data
to Kaa and then persist it to MongoDB. Also, we'll do selection queries on persisted data.
* **MongoDB nodes** -- list of MongoDB hosts.
* **Authentication credentials** -- credentials used to authenticate on MongoDB cluster.

The rest of the connection settings are described in the table below.

|Name |Description |
|-----------------------|-----------------------------------------------------------------------|
|`dbName` |Database name. |
|`connectionsPerHost` |Maximum number of connections per host. |
|`maxWaitTime` |Maximum wait time for connection in milliseconds. |
|`connectionTimeout` |Connection timeout in milliseconds. |
|`socketTimeout` |Socket timeout in milliseconds. |
|`socketKeepalive` |Turn on socket keep alive (boolean value). |
|`includeClientProfile` |Specifies whether to include client-side endpoint profile data (boolean value). |
|`includeServerProfile` |Whether to include server-side endpoint profile data (boolean value). |

We have next log schema:
Below is an example configuration that matches the mentioned Avro schema.

```json
{
"mongoServers":[
{
"host":"127.0.0.1",
"port":27017
}
],
"mongoCredentials":[
{
"user":"user",
"password":"password"
}
],
"dbName":"kaa",
"connectionsPerHost":{
"int":30
},
"maxWaitTime":{
"int":120000
},
"connectionTimeout":{
"int":5000
},
"socketTimeout":{
"int":0
},
"socketKeepalive":{
"boolean":false
},
"includeClientProfile":{
"boolean":false
},
"includeServerProfile":{
"boolean":false
}
}
```

## Playing with MongoDB log appender

The example below uses the **Data collection demo** from [Kaa Sandbox]({{root_url}}Glossary/#kaa-sandbox).
The log appender will send data to Kaa and then persist it to MongoDB.
Some selection queries will be demonstrated using the persisted data.

Below is the log schema for the application.

```json
{
Expand All @@ -183,7 +181,7 @@ We have next log schema:
}
```

The following JSON example matches the previous schema.
The following JSON example matches the above schema.

```json
{
Expand All @@ -193,55 +191,35 @@ The following JSON example matches the previous schema.

```

1. Go to Data collection demos in Sandbox.
![Data collection demo in Sandbox](attach/data-collection-demo-in-sandbox.png)
2. In the Admin UI follow to **Data collection demo** application.
![Data collection Demo Admin UI](attach/mongodb-log-appender2.png)
![Add log appender](attach/mongodb-log-appender3.png)
3. There can be one MongoDB log appender. You can add new with your parameters.
4. Enter name of the new appender.
5. Select **MongoDB** appender type.
![Log appender configuration](attach/mongodb-log-appender4.png)
6. Add new node in the **Configuration** section (localhost:27017).
![Add new node](attach/mongodb-log-appender5.png)
7. Also you can add some **Authentication credentials**.
![Authentication credentials](attach/mongodb-log-appender6.png)
8. And other important parameters of configuration. You can change them or use default.
![Other configuration parameters](attach/mongodb-log-appender7.png)
9. Click **Add** button on the top of the screen to create and deploy appender.
![Add button](attach/mongodb-log-appender8.png)
10. Verify that newly created appender has appeared in list.
![Verify newly created log appender](attach/mongodb-log-appender9.png)
11. Now use instructions from Sandbox to run Data collection demo application.
12. After this you should see something like below:

```bash
Data collection demo started
Received new sample period: 1
Sampled temperature 28 1474622330
Sampled temperature 31 1474622331
Sampled temperature 32 1474622332
Sampled temperature 30 1474622333
Sampled temperature 28 1474622334
...
```

13. Let's verify that our logs have been persisted in MongoDB. Go to Sandbox VM and run next command to connect MongoDB:
```bash
mongo kaa
db.logs_$your_application_token$.find()
```
14. You should observe similar output:
```bash
{ "_id" : ObjectId("57d916e8d55fb2073ae3cfbd"), "header" : { "endpointKeyHash" : { "string" : "H0Oclp3Wn/QS25dZCQSbV5ZkjRo=" }, "applicationToken" : { "string" : "65691512829156876532" }, "headerVersion" : { "int" : 1 }, "timestamp" : { "long" : NumberLong("1473844968489") }, "logSchemaVersion" : null }, "event" : { "temperature" : 28, "timeStamp" : 1474622330 } }
{ "_id" : ObjectId("57d916e8d55fb2073ae3cfbe"), "header" : { "endpointKeyHash" : { "string" : "H0Oclp3Wn/QS25dZCQSbV5ZkjRo=" }, "applicationToken" : { "string" : "65691512829156876532" }, "headerVersion" : { "int" : 1 }, "timestamp" : { "long" : NumberLong("1473844968489") }, "logSchemaVersion" : null }, "event" : { "temperature" : 31, "timeStamp" : 1474622331 } }
{ "_id" : ObjectId("57d916e8d55fb2073ae3cfbf"), "header" : { "endpointKeyHash" : { "string" : "H0Oclp3Wn/QS25dZCQSbV5ZkjRo=" }, "applicationToken" : { "string" : "65691512829156876532" }, "headerVersion" : { "int" : 1 }, "timestamp" : { "long" : NumberLong("1473844968489") }, "logSchemaVersion" : null }, "event" : { "temperature" : 32, "timeStamp" : 1474622332 } }
{ "_id" : ObjectId("57d916e8d55fb2073ae3cfc0"), "header" : { "endpointKeyHash" : { "string" : "H0Oclp3Wn/QS25dZCQSbV5ZkjRo=" }, "applicationToken" : { "string" : "65691512829156876532" }, "headerVersion" : { "int" : 1 }, "timestamp" : { "long" : NumberLong("1473844968489") }, "logSchemaVersion" : null }, "event" : { "temperature" : 30, "timeStamp" : 1474622333 } }
{ "_id" : ObjectId("57d916e8d55fb2073ae3cfc1"), "header" : { "endpointKeyHash" : { "string" : "H0Oclp3Wn/QS25dZCQSbV5ZkjRo=" }, "applicationToken" : { "string" : "65691512829156876532" }, "headerVersion" : { "int" : 1 }, "timestamp" : { "long" : NumberLong("1473844968489") }, "logSchemaVersion" : null }, "event" : { "temperature" : 28, "timeStamp" : 1474622334 } }
...
```
If your output doesn't match above one, please follow our [troubleshooting guide]({{root_url}}Administration-guide/Troubleshooting).
To play around with the MongoDB log appender:

1. Open the Data collection demo from Kaa Sandbox, follow the application installation instructions, and run the application.

2. Create a MongoDB log appender as described [above](#create-mongodb-log-appender) or use the one pre-installed in the sample application.

3. Your running Data collection demo application will display the output similar to the example below.

Data collection demo started
Received new sample period: 1
Sampled temperature 28 1474622330
Sampled temperature 31 1474622331
Sampled temperature 32 1474622332
Sampled temperature 30 1474622333
Sampled temperature 28 1474622334
...

4. To verify that your logs have been persisted to MongoDB, open the Sandbox VM and run the following command.

mongo kaa
db.logs_$your_application_token$.find()

5. The following output will be displayed.

{ "_id" : ObjectId("57d916e8d55fb2073ae3cfbd"), "header" : { "endpointKeyHash" : { "string" : "H0Oclp3Wn/QS25dZCQSbV5ZkjRo=" }, "applicationToken" : { "string" : "65691512829156876532" }, "headerVersion" : { "int" : 1 }, "timestamp" : { "long" : NumberLong("1473844968489") }, "logSchemaVersion" : null }, "event" : { "temperature" : 28, "timeStamp" : 1474622330 } }
{ "_id" : ObjectId("57d916e8d55fb2073ae3cfbe"), "header" : { "endpointKeyHash" : { "string" : "H0Oclp3Wn/QS25dZCQSbV5ZkjRo=" }, "applicationToken" : { "string" : "65691512829156876532" }, "headerVersion" : { "int" : 1 }, "timestamp" : { "long" : NumberLong("1473844968489") }, "logSchemaVersion" : null }, "event" : { "temperature" : 31, "timeStamp" : 1474622331 } }
{ "_id" : ObjectId("57d916e8d55fb2073ae3cfbf"), "header" : { "endpointKeyHash" : { "string" : "H0Oclp3Wn/QS25dZCQSbV5ZkjRo=" }, "applicationToken" : { "string" : "65691512829156876532" }, "headerVersion" : { "int" : 1 }, "timestamp" : { "long" : NumberLong("1473844968489") }, "logSchemaVersion" : null }, "event" : { "temperature" : 32, "timeStamp" : 1474622332 } }
{ "_id" : ObjectId("57d916e8d55fb2073ae3cfc0"), "header" : { "endpointKeyHash" : { "string" : "H0Oclp3Wn/QS25dZCQSbV5ZkjRo=" }, "applicationToken" : { "string" : "65691512829156876532" }, "headerVersion" : { "int" : 1 }, "timestamp" : { "long" : NumberLong("1473844968489") }, "logSchemaVersion" : null }, "event" : { "temperature" : 30, "timeStamp" : 1474622333 } }
{ "_id" : ObjectId("57d916e8d55fb2073ae3cfc1"), "header" : { "endpointKeyHash" : { "string" : "H0Oclp3Wn/QS25dZCQSbV5ZkjRo=" }, "applicationToken" : { "string" : "65691512829156876532" }, "headerVersion" : { "int" : 1 }, "timestamp" : { "long" : NumberLong("1473844968489") }, "logSchemaVersion" : null }, "event" : { "temperature" : 28, "timeStamp" : 1474622334 } }
...
If you don't get the desired output or experience other problems, see [Troubleshooting]({{root_url}}Administration-guide/Troubleshooting).

0 comments on commit 2439b32

Please sign in to comment.