feat: detect schema for a log event #966
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
server can detect schema for a sample log event before creating stream
helps user to create the static schema
new API endpoint `POST /logstream/schema/detect` with body as a json object
returns schema
eg. for request json -
```
{
"datetime": "2024-10-21T05:40:58.280Z",
"b": 2.0,
"c": 1,
"host": "backend",
"a": false,
"id": "duzrixscdpavbdgc",
"message": "Tom is interested in mathematics.",
"method": "GET",
"p_metadata": "state=fatal",
"p_tags": "environment=development",
"p_timestamp": "2024-10-15T14:00:00+05:30",
"referrer": "http://www.facebook.com/",
"status": 404,
"user-identifier": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36 OPR/91.0.4516.20"
}
```
API returns
```
{
"fields": [
{
"name": "datetime",
"data_type": {
"Timestamp": [
"Millisecond",
null
]
},
"nullable": true,
"dict_id": 0,
"dict_is_ordered": false,
"metadata": {}
},
{
"name": "b",
"data_type": "Float64",
"nullable": true,
"dict_id": 0,
"dict_is_ordered": false,
"metadata": {}
},
{
"name": "c",
"data_type": "Int64",
"nullable": true,
"dict_id": 0,
"dict_is_ordered": false,
"metadata": {}
},
{
"name": "host",
"data_type": "Utf8",
"nullable": true,
"dict_id": 0,
"dict_is_ordered": false,
"metadata": {}
},
{
"name": "a",
"data_type": "Boolean",
"nullable": true,
"dict_id": 0,
"dict_is_ordered": false,
"metadata": {}
},
{
"name": "id",
"data_type": "Utf8",
"nullable": true,
"dict_id": 0,
"dict_is_ordered": false,
"metadata": {}
},
{
"name": "message",
"data_type": "Utf8",
"nullable": true,
"dict_id": 0,
"dict_is_ordered": false,
"metadata": {}
},
{
"name": "method",
"data_type": "Utf8",
"nullable": true,
"dict_id": 0,
"dict_is_ordered": false,
"metadata": {}
},
{
"name": "p_metadata",
"data_type": "Utf8",
"nullable": true,
"dict_id": 0,
"dict_is_ordered": false,
"metadata": {}
},
{
"name": "p_tags",
"data_type": "Utf8",
"nullable": true,
"dict_id": 0,
"dict_is_ordered": false,
"metadata": {}
},
{
"name": "p_timestamp",
"data_type": {
"Timestamp": [
"Millisecond",
null
]
},
"nullable": true,
"dict_id": 0,
"dict_is_ordered": false,
"metadata": {}
},
{
"name": "referrer",
"data_type": "Utf8",
"nullable": true,
"dict_id": 0,
"dict_is_ordered": false,
"metadata": {}
},
{
"name": "status",
"data_type": "Int64",
"nullable": true,
"dict_id": 0,
"dict_is_ordered": false,
"metadata": {}
},
{
"name": "user-identifier",
"data_type": "Utf8",
"nullable": true,
"dict_id": 0,
"dict_is_ordered": false,
"metadata": {}
}
],
"metadata": {}
}
console then asks user to confirm this schema or change as per his requirement
once confirmed, he can create stream with the static schema
```
nitisht
reviewed
Oct 23, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
server can detect schema for a sample log event before creating stream helps user to create the static schema
new API endpoint
POST /logstream/schema/detectwith body as a json object returns schemaeg. for request json -
API returns
console then asks user to confirm this schema or change as per his requirement
once confirmed, he can create stream with the static schema