diff --git a/visualizations/observability/http/README.md b/visualizations/observability/http/README.md new file mode 100644 index 0000000..65c2a4c --- /dev/null +++ b/visualizations/observability/http/README.md @@ -0,0 +1,110 @@ + +# Http Network Graph Widget + +## Introduction +The Http Network Graph Widget for OpenSearch Dashboards uses Vega for complex visualization of service interactions. It's designed to map out http relationships . + +## Info +This widget graphically presents http network sankey graph, ingested via [data-prepper logs pipelines](https://opensearch.org/docs/2.4/data-prepper/pipelines/configuration/sources/otel-logs-source/) or using standard OTEL collector ingestion alternative. +See additional [instruction](../../vega-visualizations.md) on how to use and build [vega based visualization](https://opensearch.org/docs/latest/dashboards/visualize/viz-index/#vega) in the dashboards. + +![Http Network Visualization](http-network-graph.png) + +## Vega Integration +Vega's integration allows for customized, interactive graph creation, enabling detailed visual analysis of http network graph. + +## Structure +The widget includes: +- **Queries**: The actual queries used to fetch the visualized data (whether using PPL or DQL) +- **PPL Query Based Widget**: Vega spec that uses OpenSearch's Piped Processing Language for data querying. +- **DQL Query Based Widget**: Vega spec that employs OpenSearch DSL for data querying. +- **Visualization**: The actual `ndjson` visualization predefined with dql based vega specs +- **Self-Contained Widget**: Predefined for use in [Vega Editor](https://vega.github.io/editor/), allowing customization. + +## Data Model +Adheres to OTEL specs: +- [Http-communication sematic convention](https://github.com/opensearch-project/opensearch-catalog/blob/main/schema/observability/logs/communication-1.0.0.mapping) +- [Http with aws-vpc flow logs](https://github.com/opensearch-project/opensearch-catalog/blob/main/schema/observability/logs/aws/aws_vpc_flow-1.0.0.mapping) + +## Widget Queries + +Represent http source / target based aggregation. + +- **DQL Based**: +Avg duration per service per time bucket using the `date_histogram` aggregation and binding the interval to the external dashboard timeframe context + +- **AWS VPC based Query** +```json5 + { + "%context%": "true", + "%timefield%": "@timestamp", + "index": "ss4o_logs_vpc-*-*", + "body": { + "size": 0, + "aggs": { + "table": { + "composite": { + "size": 10000, + "sources": [ + { + "stk1": { + "terms": { + "field": "aws.vpc.srcaddr" + } + } + }, + { + "stk2": { + "terms": { + "field": "aws.vpc.dstaddr" + } + } + } + ] + } + } + } + } +} +``` + +- **HTTP sematic convention based Query** +```json5 + { + "%context%": "true", + "%timefield%": "@timestamp", + "index": "ss4o_logs-*-*", + "body": { + "size": 0, + "aggs": { + "table": { + "composite": { + "size": 10000, + "sources": [ + { + "stk1": { + "terms": { + "field": "communication.source.address" + } + } + }, + { + "stk2": { + "terms": { + "field": "communication.destination.address" + } + } + } + ] + } + } + } + } +} +``` + +## Prerequisites +Required indices: `ss4o_logs-*-*` Or `ss4o_logs_vpc-*-*`. + +## Try Me +[Open the widget in the Vega Editor]([Open the Chart in the Vega Editor](https://vega.github.io/editor/#/url/vega/N4IgJAzgxgFgpgWwIYgFwhgF0wBwqgegIDc4BzJAOjIEtMYBXAI0poHsDp5kTykSArJQBWENgDsQAGhAATJJhSoA2qHFIEcNCABOSAO4ARBShkAzNjuSY0oHDrY44OzAE9tSMmR19M7cRCUikwANnCUTAxQANZwmBAgAL4ymHoBFlZoqiBuTtoZCAwhpiBwAB722vKYDAiUsa6UEJjRAIzSIEgJ6M1tSVKguVroBUUl5ZXo1bX1cI29AEwdXdqL-YOueSOWhcUdEzpVCjOybFAA+lBsDOI2Mis9NABeWokAujLESCEMcN2DbDY0TQrRSNE0snO1xsqDM3wgcBk5wgMCQOlk-xybEUIRBMggUSgfwgZiKeJAEGiNBwTlkaAADOYkDQwnTUPTkhg6JjMNjvmgFgsZMgysirj40OIiiEZDBuVk3pzPN5fP4eUhQsNQEgzJhnOcGrYKS12uhWgA2BaUVoAFitrQWQlaAHZnR01mb6ZQvV6FvT6f0QJEYnFutlDahQL1TSBWj7vZQ-QH8S0lmbLda7dbHdbXYHThcrjcYa1Qca+p6E77-e7U9oLfasw6nXn8c9hqXkqAI1GTfWM7b7TmXW6U9E07H49WA5yC5drrcQWXo-3G0OW6PyxO41XEzW2y8l+9EieuyB1JptOI2LI-u7rjoido9EYTB1Ukh0jsshstiAzCyeqHDIBzaAAhN41w4AAynAYRQLyOgAAQAD4oUhkEMDBcFwAhlhNCaSEALxEUh0x1NGqHoZh2HwYhBHjsRpHkQxSxnkM+Q7GM+wVIcUzHBRJoANQsWs9zdCAhrsZswz-mwIR0uYNBwRiWTlu0Y5LB8nRhsaSAxB03g4AAknSioDDkMmcVY3EgbxRw1IJ+nRExSEAOTRm5SEAPxkQJDGtCJ-mLEhqB+Y5rFBRFK7iasliYAAYspCnrJZf7NM5hkOFhTDuCoekGdpYguEaAEqXFLhJeVMiWLefFyH8RLiLINDiGQgZlSljyHtJf6jHsdmTCAAAULGuPSUUzK4rQAJQEEssXoK4UBJOZagaLJNEJPiD5Pug163ttORpCS34qL+snKj4FB6llUG5WpGUGTIRmmSA2mdap+UQO2701Xgj21H9OlqbyOLvb1slPcCL3ZTgD3fYohU7SVkb-slbIgK9ikgLVzhVI1cDNa17Wcp92hg-ykPWbs4z2T0UDfHAw1ua4blSOFU30jNywSdATOyAAmjOFkcdsNkDaU9MUozYQs2zHNjbNvOrLLcBC+01Pi7TPFDaJSMuSR7meSr6A6DQZBYAAMhqcGpWLckS3Tev+ZTIQEKcyCtfLbkzcorRvKbIBOI+ROKGQrxreeG0E80rUKP4AByN53jtDCh1eKdHR+X6ZOdaWyQBIRAbr9X685rkeambmrWeF6yerEdHWI6d7eeWfvidBQ-gX+SAfjg1l8FBuVybWsgCEgLRFhHRmA4CCx346h+BIyeHR0EaSXMs8Y7phraQ8yg5GiEc2FHDshK10Q4AoMAdJYymLugMAP08Ejg-iqJ-i1ngSPyaeh0FkaUCDMmY+0Vv5ZaPNOQt1DgADWAdLfmcs3JlHZsbE0vskJCSQkwT8sh9A0FkPQFmaDoEpBPnEIBaMQEyzAazdBLFFA6FPpQKBgZmGnwQTQpBatSHoKruOX2SRx79WdvVNIEcfZ+3pG8AAtMg5mDCIHRXbDzRaxoHCxAAOpEPoPbKy2tbJSxdqol4Hs2Be3ENI-2gcNEhyauHSO7x8RqzDOtS86AyjvkMUGfBHRJGyUIcQu+MhPbMkkIjPomlgY31kC1NqAB5BgJd2TegEDIOJCSyDGXEOIfGaSACcAg64xyWj4v8l98logCZ+CO2h4AWywB0cJrUjTVCUO3deSlypLU1qUzxIAriT2Ar3dAtV464hkIE7QjM9RkEsO4MJliIntLfGbAwxhFA716epERFl66rANonDa2dfETPUFM3QdThhH2grtLQMhDB-CXgnCQwNWmRKPjFLcEMD5lDvPnB+YdtBMGxLyBerimbaG8SBcQVxbxGmKEwOCmIsLVC1DkcoMIoxqyOc5E5mgjrk3QF8H4rxTxnmBU-CecBdTujxeUlxIBkA6GiO4sZwdb4dEOegRucAACyaIYb-nnms7ZfLZBN0DFAS+OA0CpF+HChFmL0UKExc0LRtzBgvO0DRWCdFLBIQAGTGownDA1uF6LQxIoI1oNcoVhFmfJSwOyuq-LPIo51Iy3WYxXMyzVQI4C6JCaVDGRytUhv0ZyG++i0Ykq5dGlMWrEk3ygHQPKUYLaXPAvqnCeFkKmqQsNGiAAJNgpAdABVcvrNoVFzVQXLZW1iNbh7jhmj5JCXpCmhS7ZQAAzIGJ4rVbzeLRj9MgOb0AQQtfmxCJqzUlrhk25w1aja1taPWstFbV0hXXW2hYHbfKbrCiLLE8k-DyvHdm-k-FoqERwV5QASYTuWwZzQSjFH0AB1MBeRwQUBQo1grtg5uzb0ZgsGPqQtBlmb6AOYCAxFBxYdPCInctaAApJB9yM0a4nllDuvivYU1pozUaMlvwQSUtFr4nwCEeVlKxnDIVbLZ5irRh0vVcMEiciJiq4BtwCmgCLricdjKhkutGQm7GgZglxtxdCrxHQ8HNSo5yNVt00ZjoU06pTPT3XQ0DJm9GuzFFCzPa4NMwnw2gNZILTWSkQghFTfpMjaMKPDC9OaTkvIL3UiNBOqdch-LYzfV5aDf6kLwcQzMN2oGpDgew+F4tkXossWQ7cVDoHMNJdw-sjAhGw1OZc+mtw5HviUdQK0ajF0KbYrY5Y8VnStqBlakBfSfhSBoDhCEBEyqU5Gg05i7TFIb2iboSgtBKiZjQw7TgmLdRzaW0wDbFFIRO0qYIXomA-C5tIQABy9rkft8hIBloBbG9oBbTQ1bmbfaJW79m5psXuJfSdF3J23uCxFJb1tbbrd8m5MIuovJhTcr939gY8EIiqZijz2gEBENkE6sm79tFwCaTikA8P0BgpSj5+r17PvjY1BAa7ZnhZyIewLJ7SEAB8SFWj9s7SxULYO8Onho3+GiLT1nZF5d9pACVLDQRfvoAAgk5joHn3HMpzqdPO2QHZFxLoPLjUFLUFtrtpPjA2aGCaIyAMoK0idBbkzAeaRnyvkrQP2jkMhzfW8q-t+3GAMfLad8MO3nOWXCo5Q7HnMgBc0QAEp-DiA1heHH1mC+FzoUXbAJdS94-CvXoAw5CYkzofJOgQ9IBagwTEOPvOOdE6AHHIAADECUBAJWdAlcXgZA2xE99oSvUBWgd47031IQao133cxV4YCxOSNI9-GmzoAefP3d80-DIBHcT96VPuG2hHc+6G0aRwrmyuD5t1V+fL9K1b9I7v8vQ+GSUGdD71l7Ke4Oz1GUO4IA2vOA6zQLrsJ4SIlKKnxF+vUlQBjcw1l8mMoI19ttAxdgYQvQSkZBzsl8uoV9wCZ9McoCigYDKAMlcYzAzAEQYQR9Xts1W90BHF8ZORoc4JWo4cL90BEd4kUdzA0dZ8scK88c6QCcn8SCQAE99AkJJdcRKVFRhC2xicOVg9l1CtPhaDgBOR3l844BSBbgJIAABGiFjaIVABAa4BEKQkATfdAYAaMVAcuGIW1TyBdd9agHQEyWQDmRYEw4eZycw6uSw1nGw0yRIVKRQsOCSbQwvOAaEDoAwkAWQ2uA5RjPNQ1UZHHHrPrXGL5dPJQ+IbQNQ5jYVVAWVGgGIMCYInADFbQIwg2Rw+9Zyewk0EombA2FwtoLyItdw2w8o8cSopyMwoiQRBYOos1Bozw7w5I-3XxLI56X3NlCQqCMPfAgxb+UIIY4EZlEIuI5xbSK4cQACdqNGGZNGOZcgRZNSSvAQG0AAIX7UKUKQ6Er3NHpEKQWAAGFkwq9DB+1Lj9tzRzjCkXR6RDj9tzibjxd9sABRcXf484wwc0Q4mvTcSvQ4wpP4-bb4mQSvQwP4w4m0AQc48XcXc0AQAQV4hE-450c0c0G0G4iGe4R8IrcbfYo4k4s4pUHwJQazKXdAKk4404wMWHALXvFvZkg41k2kzJblcdLk2SFkmkwMOjLHETNvXksUzkHQIoDVYU6U6ktkmBL+RUrVZUvkpvVwBAPHCkrU2UlIQnRkyk2QMwOAAQOlDhPQFkA0nklU-knIOgXTUAYZV1Zk80uAftOAb4mBNwV0rGBgIhOAORZFO2CfJkqvfbWQQpfbJAQdTkMgYM28ORPwTAQMqUz0i0n0v02GKCNMl0zFLMqvL03MwMGiBRZgdMzMlkM0nM30-LJAMoGgHkbI6IG4yTNvBYQpftfbG0TcT5Tsn1ZknsvsgcrKIhYcj0qvMc-st0Cg5yTCVTdAeXG+HwRcTkRfZsRkN3NA1AF3XcpAFJNgH6Q8NGZXOgDoFYxQVqCSLJEmERIAA)) diff --git a/visualizations/observability/http/data/aws-vpc-bulk_data.json b/visualizations/observability/http/data/aws-vpc-bulk_data.json new file mode 100644 index 0000000..8448fb2 --- /dev/null +++ b/visualizations/observability/http/data/aws-vpc-bulk_data.json @@ -0,0 +1,1467 @@ +[ + { + "@timestamp": "2023-07-17T08:14:05.000Z", + "body": "2 111111111111 eni-0e250409d410e1290 162.142.125.177 10.0.0.200 38471 12313 6 1 44 1674898496 1674898507 ACCEPT OK", + "event": { + "result": "ACCEPT", + "name": "flow_log", + "domain": "vpc.flow_log" + }, + "attributes": { + "data_stream": { + "dataset": "vpc.flow_log", + "namespace": "production", + "type": "logs_vpc" + } + }, + "cloud": { + "provider": "aws", + "account": { + "id": "111111111111" + }, + "region": "ap-southeast-2", + "resource_id": "vpc-0d4d4e82b7d743527", + "platform": "aws_vpc" + }, + "aws": { + "s3": { + "bucket": "centralizedlogging-loghubloggingbucket0fa53b76-t57zyhgb8c2", + "key": "AWSLogs/111111111111/vpcflowlogs/us-east-2/2023/01/28/111111111111_vpcflowlogs_us-east-2_fl-023c6afa025ee5a04_20230128T0930Z_3a9dfd9d.log.gz" + }, + "vpc": { + "version" : "2", + "account-id" : "111111111111", + "interface-id" : "eni-0e250409d410e1290", + "region": "ap-southeast-2", + "vpc-id": "vpc-0d4d4e82b7d743527", + "subnet-id": "subnet-aaaaaaaa012345678", + "az-id": "apse2-az3", + "instance-id": "i-0c50d5961bcb2d47b", + "srcaddr" : "162.142.125.177", + "dstaddr" : "10.0.0.200", + "srcport" : 38471, + "dstport" : 12313, + "protocol" : "6", + "packets" : 1, + "bytes" : 44, + "pkt-src-aws-service": "S3", + "pkt-dst-aws-service": "-", + "flow-direction": "ingress", + "start" : "1674898496", + "end" : "1674898507", + "action" : "ACCEPT", + "log-status" : "OK" + } + }, + "communication": { + "source": { + "address": "162.142.125.177", + "port": 38471, + "packets": 1, + "bytes" : 44 + }, + "destination": { + "address": "10.0.0.200", + "port": 12313 + } + } + }, + { + "@timestamp": "2023-07-17T08:14:05.000Z", + "body": "2 111111111111 eni-0e250409d410e1290 162.142.125.177 10.0.0.200 38471 12313 6 1 44 1674898496 1674898507 ACCEPT OK", + "event": { + "result": "ACCEPT", + "name": "flow_log", + "domain": "vpc.flow_log" + }, + "attributes": { + "data_stream": { + "dataset": "vpc.flow_log", + "namespace": "production", + "type": "logs_vpc" + } + }, + "cloud": { + "provider": "aws", + "account": { + "id": "111111111111" + }, + "region": "ap-southeast-2", + "resource_id": "vpc-0d4d4e82b7d743527", + "platform": "aws_vpc" + }, + "aws": { + "s3": { + "bucket": "centralizedlogging-loghubloggingbucket0fa53b76-t57zyhgb8c2", + "key": "AWSLogs/111111111111/vpcflowlogs/us-east-2/2023/01/28/111111111111_vpcflowlogs_us-east-2_fl-023c6afa025ee5a04_20230128T0930Z_3a9dfd9d.log.gz" + }, + "vpc": { + "version" : "2", + "account-id" : "111111111111", + "interface-id" : "eni-0e250409d410e1290", + "region": "ap-southeast-2", + "vpc-id": "vpc-0d4d4e82b7d743527", + "subnet-id": "subnet-aaaaaaaa012345678", + "az-id": "apse2-az3", + "instance-id": "i-0c50d5961bcb2d47b", + "srcaddr" : "10.0.0.200", + "dstaddr" : "162.142.125.177", + "srcport" : 12313, + "dstport" : 38471, + "protocol" : "6", + "packets" : 1, + "bytes" : 440, + "pkt-src-aws-service": "-", + "pkt-dst-aws-service": "S3", + "flow-direction": "egress", + "start" : "1674898496", + "end" : "1674898507", + "action" : "REJECT", + "log-status" : "OK" + } + }, + "communication": { + "source": { + "address": "10.0.0.200", + "port": 12313, + "packets": 1, + "bytes" : 440 + }, + "destination": { + "address": "162.142.125.177", + "port": 38471 + } + } + }, + { + "@timestamp": "2023-07-17T08:14:05.000Z", + "body": "2 111111111111 eni-0e250409d410e1290 162.142.125.177 10.0.0.200 38471 12313 6 1 44 1674898496 1674898507 ACCEPT OK", + "event": { + "result": "ACCEPT", + "name": "flow_log", + "domain": "vpc.flow_log" + }, + "attributes": { + "data_stream": { + "dataset": "vpc.flow_log", + "namespace": "production", + "type": "logs_vpc" + } + }, + "cloud": { + "provider": "aws", + "account": { + "id": "111111111111" + }, + "region": "ap-southeast-2", + "resource_id": "vpc-0d4d4e82b7d743527", + "platform": "aws_vpc" + }, + "aws": { + "s3": { + "bucket": "centralizedlogging-loghubloggingbucket0fa53b76-t57zyhgb8c2", + "key": "AWSLogs/111111111111/vpcflowlogs/us-east-2/2023/01/28/111111111111_vpcflowlogs_us-east-2_fl-023c6afa025ee5a04_20230128T0930Z_3a9dfd9d.log.gz" + }, + "vpc": { + "version" : "2", + "account-id" : "111111111111", + "interface-id" : "eni-0e250409d410e1290", + "region": "ap-southeast-2", + "vpc-id": "vpc-0d4d4e82b7d743527", + "subnet-id": "subnet-aaaaaaaa012345678", + "az-id": "apse2-az3", + "instance-id": "i-0c50d5961bcb2d47b", + "srcaddr" : "162.142.125.177", + "dstaddr" : "10.0.0.200", + "srcport" : 38471, + "dstport" : 12313, + "protocol" : "6", + "packets" : 1, + "bytes" : 44, + "pkt-src-aws-service": "S3", + "pkt-dst-aws-service": "-", + "flow-direction": "ingress", + "start" : "1674898496", + "end" : "1674898507", + "action" : "ACCEPT", + "log-status" : "OK" + } + }, + "communication": { + "source": { + "address": "162.142.125.177", + "port": 38471, + "packets": 1, + "bytes" : 44 + }, + "destination": { + "address": "10.0.0.200", + "port": 12313 + } + } + }, + { + "@timestamp": "2023-07-17T08:14:05.000Z", + "body": "2 111111111111 eni-0e250409d410e1290 162.142.125.177 10.0.0.200 38471 12313 6 1 44 1674898496 1674898507 ACCEPT OK", + "event": { + "result": "ACCEPT", + "name": "flow_log", + "domain": "vpc.flow_log" + }, + "attributes": { + "data_stream": { + "dataset": "vpc.flow_log", + "namespace": "production", + "type": "logs_vpc" + } + }, + "cloud": { + "provider": "aws", + "account": { + "id": "111111111111" + }, + "region": "ap-southeast-2", + "resource_id": "vpc-0d4d4e82b7d743527", + "platform": "aws_vpc" + }, + "aws": { + "s3": { + "bucket": "centralizedlogging-loghubloggingbucket0fa53b76-t57zyhgb8c2", + "key": "AWSLogs/111111111111/vpcflowlogs/us-east-2/2023/01/28/111111111111_vpcflowlogs_us-east-2_fl-023c6afa025ee5a04_20230128T0930Z_3a9dfd9d.log.gz" + }, + "vpc": { + "version" : "2", + "account-id" : "111111111111", + "interface-id" : "eni-0e250409d410e1290", + "region": "ap-southeast-2", + "vpc-id": "vpc-0d4d4e82b7d743527", + "subnet-id": "subnet-aaaaaaaa012345678", + "az-id": "apse2-az3", + "instance-id": "i-0c50d5961bcb2d47b", + "srcaddr" : "10.0.0.200", + "dstaddr" : "162.142.125.177", + "srcport" : 12313, + "dstport" : 38471, + "protocol" : "6", + "packets" : 1, + "bytes" : 440, + "pkt-src-aws-service": "-", + "pkt-dst-aws-service": "S3", + "flow-direction": "egress", + "start" : "1674898496", + "end" : "1674898507", + "action" : "REJECT", + "log-status" : "OK" + } + }, + "communication": { + "source": { + "address": "10.0.0.200", + "port": 12313, + "packets": 1, + "bytes" : 440 + }, + "destination": { + "address": "162.142.125.177", + "port": 38471 + } + } + }, { + "@timestamp": "2023-07-17T08:14:05.000Z", + "body": "2 111111111111 eni-0e250409d410e1290 162.142.125.177 10.0.0.200 38471 12313 6 1 44 1674898496 1674898507 ACCEPT OK", + "event": { + "result": "ACCEPT", + "name": "flow_log", + "domain": "vpc.flow_log" + }, + "attributes": { + "data_stream": { + "dataset": "vpc.flow_log", + "namespace": "production", + "type": "logs_vpc" + } + }, + "cloud": { + "provider": "aws", + "account": { + "id": "111111111111" + }, + "region": "ap-southeast-2", + "resource_id": "vpc-0d4d4e82b7d743527", + "platform": "aws_vpc" + }, + "aws": { + "s3": { + "bucket": "centralizedlogging-loghubloggingbucket0fa53b76-t57zyhgb8c2", + "key": "AWSLogs/111111111111/vpcflowlogs/us-east-2/2023/01/28/111111111111_vpcflowlogs_us-east-2_fl-023c6afa025ee5a04_20230128T0930Z_3a9dfd9d.log.gz" + }, + "vpc": { + "version" : "2", + "account-id" : "111111111111", + "interface-id" : "eni-0e250409d410e1290", + "region": "ap-southeast-2", + "vpc-id": "vpc-0d4d4e82b7d743527", + "subnet-id": "subnet-aaaaaaaa012345678", + "az-id": "apse2-az3", + "instance-id": "i-0c50d5961bcb2d47b", + "srcaddr" : "162.142.125.177", + "dstaddr" : "10.0.0.200", + "srcport" : 38471, + "dstport" : 12313, + "protocol" : "6", + "packets" : 1, + "bytes" : 44, + "pkt-src-aws-service": "S3", + "pkt-dst-aws-service": "-", + "flow-direction": "ingress", + "start" : "1674898496", + "end" : "1674898507", + "action" : "ACCEPT", + "log-status" : "OK" + } + }, + "communication": { + "source": { + "address": "162.142.125.177", + "port": 38471, + "packets": 1, + "bytes" : 44 + }, + "destination": { + "address": "10.0.0.200", + "port": 12313 + } + } +}, + { + "@timestamp": "2023-07-17T08:14:05.000Z", + "body": "2 111111111111 eni-0e250409d410e1290 162.142.125.177 10.0.0.200 38471 12313 6 1 44 1674898496 1674898507 ACCEPT OK", + "event": { + "result": "ACCEPT", + "name": "flow_log", + "domain": "vpc.flow_log" + }, + "attributes": { + "data_stream": { + "dataset": "vpc.flow_log", + "namespace": "production", + "type": "logs_vpc" + } + }, + "cloud": { + "provider": "aws", + "account": { + "id": "111111111111" + }, + "region": "ap-southeast-2", + "resource_id": "vpc-0d4d4e82b7d743527", + "platform": "aws_vpc" + }, + "aws": { + "s3": { + "bucket": "centralizedlogging-loghubloggingbucket0fa53b76-t57zyhgb8c2", + "key": "AWSLogs/111111111111/vpcflowlogs/us-east-2/2023/01/28/111111111111_vpcflowlogs_us-east-2_fl-023c6afa025ee5a04_20230128T0930Z_3a9dfd9d.log.gz" + }, + "vpc": { + "version" : "2", + "account-id" : "111111111111", + "interface-id" : "eni-0e250409d410e1290", + "region": "ap-southeast-2", + "vpc-id": "vpc-0d4d4e82b7d743527", + "subnet-id": "subnet-aaaaaaaa012345678", + "az-id": "apse2-az3", + "instance-id": "i-0c50d5961bcb2d47b", + "srcaddr" : "10.0.0.200", + "dstaddr" : "162.142.125.177", + "srcport" : 12313, + "dstport" : 38471, + "protocol" : "6", + "packets" : 1, + "bytes" : 440, + "pkt-src-aws-service": "-", + "pkt-dst-aws-service": "S3", + "flow-direction": "egress", + "start" : "1674898496", + "end" : "1674898507", + "action" : "REJECT", + "log-status" : "OK" + } + }, + "communication": { + "source": { + "address": "10.0.0.200", + "port": 12313, + "packets": 1, + "bytes" : 440 + }, + "destination": { + "address": "162.142.125.177", + "port": 38471 + } + } + }, { + "@timestamp": "2023-07-17T08:14:05.000Z", + "body": "2 111111111111 eni-0e250409d410e1290 162.142.125.177 10.0.0.200 38471 12313 6 1 44 1674898496 1674898507 ACCEPT OK", + "event": { + "result": "ACCEPT", + "name": "flow_log", + "domain": "vpc.flow_log" + }, + "attributes": { + "data_stream": { + "dataset": "vpc.flow_log", + "namespace": "production", + "type": "logs_vpc" + } + }, + "cloud": { + "provider": "aws", + "account": { + "id": "111111111111" + }, + "region": "ap-southeast-2", + "resource_id": "vpc-0d4d4e82b7d743527", + "platform": "aws_vpc" + }, + "aws": { + "s3": { + "bucket": "centralizedlogging-loghubloggingbucket0fa53b76-t57zyhgb8c2", + "key": "AWSLogs/111111111111/vpcflowlogs/us-east-2/2023/01/28/111111111111_vpcflowlogs_us-east-2_fl-023c6afa025ee5a04_20230128T0930Z_3a9dfd9d.log.gz" + }, + "vpc": { + "version" : "2", + "account-id" : "111111111111", + "interface-id" : "eni-0e250409d410e1290", + "region": "ap-southeast-2", + "vpc-id": "vpc-0d4d4e82b7d743527", + "subnet-id": "subnet-aaaaaaaa012345678", + "az-id": "apse2-az3", + "instance-id": "i-0c50d5961bcb2d47b", + "srcaddr" : "162.142.125.177", + "dstaddr" : "10.0.0.200", + "srcport" : 38471, + "dstport" : 12313, + "protocol" : "6", + "packets" : 1, + "bytes" : 44, + "pkt-src-aws-service": "S3", + "pkt-dst-aws-service": "-", + "flow-direction": "ingress", + "start" : "1674898496", + "end" : "1674898507", + "action" : "ACCEPT", + "log-status" : "OK" + } + }, + "communication": { + "source": { + "address": "162.142.125.177", + "port": 38471, + "packets": 1, + "bytes" : 44 + }, + "destination": { + "address": "10.0.0.200", + "port": 12313 + } + } +}, + { + "@timestamp": "2023-07-17T08:14:05.000Z", + "body": "2 111111111111 eni-0e250409d410e1290 162.142.125.177 10.0.0.200 38471 12313 6 1 44 1674898496 1674898507 ACCEPT OK", + "event": { + "result": "ACCEPT", + "name": "flow_log", + "domain": "vpc.flow_log" + }, + "attributes": { + "data_stream": { + "dataset": "vpc.flow_log", + "namespace": "production", + "type": "logs_vpc" + } + }, + "cloud": { + "provider": "aws", + "account": { + "id": "111111111111" + }, + "region": "ap-southeast-2", + "resource_id": "vpc-0d4d4e82b7d743527", + "platform": "aws_vpc" + }, + "aws": { + "s3": { + "bucket": "centralizedlogging-loghubloggingbucket0fa53b76-t57zyhgb8c2", + "key": "AWSLogs/111111111111/vpcflowlogs/us-east-2/2023/01/28/111111111111_vpcflowlogs_us-east-2_fl-023c6afa025ee5a04_20230128T0930Z_3a9dfd9d.log.gz" + }, + "vpc": { + "version" : "2", + "account-id" : "111111111111", + "interface-id" : "eni-0e250409d410e1290", + "region": "ap-southeast-2", + "vpc-id": "vpc-0d4d4e82b7d743527", + "subnet-id": "subnet-aaaaaaaa012345678", + "az-id": "apse2-az3", + "instance-id": "i-0c50d5961bcb2d47b", + "srcaddr" : "10.0.0.200", + "dstaddr" : "162.142.125.177", + "srcport" : 12313, + "dstport" : 38471, + "protocol" : "6", + "packets" : 1, + "bytes" : 440, + "pkt-src-aws-service": "-", + "pkt-dst-aws-service": "S3", + "flow-direction": "egress", + "start" : "1674898496", + "end" : "1674898507", + "action" : "REJECT", + "log-status" : "OK" + } + }, + "communication": { + "source": { + "address": "10.0.0.200", + "port": 12313, + "packets": 1, + "bytes" : 440 + }, + "destination": { + "address": "162.142.125.177", + "port": 38471 + } + } + }, { + "@timestamp": "2023-07-17T08:14:05.000Z", + "body": "2 111111111111 eni-0e250409d410e1290 162.142.125.177 10.0.0.200 38471 12313 6 1 44 1674898496 1674898507 ACCEPT OK", + "event": { + "result": "ACCEPT", + "name": "flow_log", + "domain": "vpc.flow_log" + }, + "attributes": { + "data_stream": { + "dataset": "vpc.flow_log", + "namespace": "production", + "type": "logs_vpc" + } + }, + "cloud": { + "provider": "aws", + "account": { + "id": "111111111111" + }, + "region": "ap-southeast-2", + "resource_id": "vpc-0d4d4e82b7d743527", + "platform": "aws_vpc" + }, + "aws": { + "s3": { + "bucket": "centralizedlogging-loghubloggingbucket0fa53b76-t57zyhgb8c2", + "key": "AWSLogs/111111111111/vpcflowlogs/us-east-2/2023/01/28/111111111111_vpcflowlogs_us-east-2_fl-023c6afa025ee5a04_20230128T0930Z_3a9dfd9d.log.gz" + }, + "vpc": { + "version" : "2", + "account-id" : "111111111111", + "interface-id" : "eni-0e250409d410e1290", + "region": "ap-southeast-2", + "vpc-id": "vpc-0d4d4e82b7d743527", + "subnet-id": "subnet-aaaaaaaa012345678", + "az-id": "apse2-az3", + "instance-id": "i-0c50d5961bcb2d47b", + "srcaddr" : "162.142.125.177", + "dstaddr" : "10.0.0.200", + "srcport" : 38471, + "dstport" : 12313, + "protocol" : "6", + "packets" : 1, + "bytes" : 44, + "pkt-src-aws-service": "S3", + "pkt-dst-aws-service": "-", + "flow-direction": "ingress", + "start" : "1674898496", + "end" : "1674898507", + "action" : "ACCEPT", + "log-status" : "OK" + } + }, + "communication": { + "source": { + "address": "162.142.125.177", + "port": 38471, + "packets": 1, + "bytes" : 44 + }, + "destination": { + "address": "10.0.0.200", + "port": 12313 + } + } +}, + { + "@timestamp": "2023-07-17T08:14:05.000Z", + "body": "2 111111111111 eni-0e250409d410e1290 162.142.125.177 10.0.0.200 38471 12313 6 1 44 1674898496 1674898507 ACCEPT OK", + "event": { + "result": "ACCEPT", + "name": "flow_log", + "domain": "vpc.flow_log" + }, + "attributes": { + "data_stream": { + "dataset": "vpc.flow_log", + "namespace": "production", + "type": "logs_vpc" + } + }, + "cloud": { + "provider": "aws", + "account": { + "id": "111111111111" + }, + "region": "ap-southeast-2", + "resource_id": "vpc-0d4d4e82b7d743527", + "platform": "aws_vpc" + }, + "aws": { + "s3": { + "bucket": "centralizedlogging-loghubloggingbucket0fa53b76-t57zyhgb8c2", + "key": "AWSLogs/111111111111/vpcflowlogs/us-east-2/2023/01/28/111111111111_vpcflowlogs_us-east-2_fl-023c6afa025ee5a04_20230128T0930Z_3a9dfd9d.log.gz" + }, + "vpc": { + "version" : "2", + "account-id" : "111111111111", + "interface-id" : "eni-0e250409d410e1290", + "region": "ap-southeast-2", + "vpc-id": "vpc-0d4d4e82b7d743527", + "subnet-id": "subnet-aaaaaaaa012345678", + "az-id": "apse2-az3", + "instance-id": "i-0c50d5961bcb2d47b", + "srcaddr" : "10.0.0.200", + "dstaddr" : "162.142.125.177", + "srcport" : 12313, + "dstport" : 38471, + "protocol" : "6", + "packets" : 1, + "bytes" : 440, + "pkt-src-aws-service": "-", + "pkt-dst-aws-service": "S3", + "flow-direction": "egress", + "start" : "1674898496", + "end" : "1674898507", + "action" : "REJECT", + "log-status" : "OK" + } + }, + "communication": { + "source": { + "address": "10.0.0.200", + "port": 12313, + "packets": 1, + "bytes" : 440 + }, + "destination": { + "address": "162.142.125.177", + "port": 38471 + } + } + }, { + "@timestamp": "2023-07-17T08:14:05.000Z", + "body": "2 111111111111 eni-0e250409d410e1290 162.142.125.177 10.0.0.200 38471 12313 6 1 44 1674898496 1674898507 ACCEPT OK", + "event": { + "result": "ACCEPT", + "name": "flow_log", + "domain": "vpc.flow_log" + }, + "attributes": { + "data_stream": { + "dataset": "vpc.flow_log", + "namespace": "production", + "type": "logs_vpc" + } + }, + "cloud": { + "provider": "aws", + "account": { + "id": "111111111111" + }, + "region": "ap-southeast-2", + "resource_id": "vpc-0d4d4e82b7d743527", + "platform": "aws_vpc" + }, + "aws": { + "s3": { + "bucket": "centralizedlogging-loghubloggingbucket0fa53b76-t57zyhgb8c2", + "key": "AWSLogs/111111111111/vpcflowlogs/us-east-2/2023/01/28/111111111111_vpcflowlogs_us-east-2_fl-023c6afa025ee5a04_20230128T0930Z_3a9dfd9d.log.gz" + }, + "vpc": { + "version" : "2", + "account-id" : "111111111111", + "interface-id" : "eni-0e250409d410e1290", + "region": "ap-southeast-2", + "vpc-id": "vpc-0d4d4e82b7d743527", + "subnet-id": "subnet-aaaaaaaa012345678", + "az-id": "apse2-az3", + "instance-id": "i-0c50d5961bcb2d47b", + "srcaddr" : "162.142.125.177", + "dstaddr" : "10.0.0.200", + "srcport" : 38471, + "dstport" : 12313, + "protocol" : "6", + "packets" : 1, + "bytes" : 44, + "pkt-src-aws-service": "S3", + "pkt-dst-aws-service": "-", + "flow-direction": "ingress", + "start" : "1674898496", + "end" : "1674898507", + "action" : "ACCEPT", + "log-status" : "OK" + } + }, + "communication": { + "source": { + "address": "162.142.125.177", + "port": 38471, + "packets": 1, + "bytes" : 44 + }, + "destination": { + "address": "10.0.0.200", + "port": 12313 + } + } +}, + { + "@timestamp": "2023-07-17T08:14:05.000Z", + "body": "2 111111111111 eni-0e250409d410e1290 162.142.125.177 10.0.0.200 38471 12313 6 1 44 1674898496 1674898507 ACCEPT OK", + "event": { + "result": "ACCEPT", + "name": "flow_log", + "domain": "vpc.flow_log" + }, + "attributes": { + "data_stream": { + "dataset": "vpc.flow_log", + "namespace": "production", + "type": "logs_vpc" + } + }, + "cloud": { + "provider": "aws", + "account": { + "id": "111111111111" + }, + "region": "ap-southeast-2", + "resource_id": "vpc-0d4d4e82b7d743527", + "platform": "aws_vpc" + }, + "aws": { + "s3": { + "bucket": "centralizedlogging-loghubloggingbucket0fa53b76-t57zyhgb8c2", + "key": "AWSLogs/111111111111/vpcflowlogs/us-east-2/2023/01/28/111111111111_vpcflowlogs_us-east-2_fl-023c6afa025ee5a04_20230128T0930Z_3a9dfd9d.log.gz" + }, + "vpc": { + "version" : "2", + "account-id" : "111111111111", + "interface-id" : "eni-0e250409d410e1290", + "region": "ap-southeast-2", + "vpc-id": "vpc-0d4d4e82b7d743527", + "subnet-id": "subnet-aaaaaaaa012345678", + "az-id": "apse2-az3", + "instance-id": "i-0c50d5961bcb2d47b", + "srcaddr" : "10.0.0.200", + "dstaddr" : "162.142.125.177", + "srcport" : 12313, + "dstport" : 38471, + "protocol" : "6", + "packets" : 1, + "bytes" : 440, + "pkt-src-aws-service": "-", + "pkt-dst-aws-service": "S3", + "flow-direction": "egress", + "start" : "1674898496", + "end" : "1674898507", + "action" : "REJECT", + "log-status" : "OK" + } + }, + "communication": { + "source": { + "address": "10.0.0.200", + "port": 12313, + "packets": 1, + "bytes" : 440 + }, + "destination": { + "address": "162.142.125.177", + "port": 38471 + } + } + }, { + "@timestamp": "2023-07-17T08:14:05.000Z", + "body": "2 111111111111 eni-0e250409d410e1290 162.142.125.177 10.0.0.200 38471 12313 6 1 44 1674898496 1674898507 ACCEPT OK", + "event": { + "result": "ACCEPT", + "name": "flow_log", + "domain": "vpc.flow_log" + }, + "attributes": { + "data_stream": { + "dataset": "vpc.flow_log", + "namespace": "production", + "type": "logs_vpc" + } + }, + "cloud": { + "provider": "aws", + "account": { + "id": "111111111111" + }, + "region": "ap-southeast-2", + "resource_id": "vpc-0d4d4e82b7d743527", + "platform": "aws_vpc" + }, + "aws": { + "s3": { + "bucket": "centralizedlogging-loghubloggingbucket0fa53b76-t57zyhgb8c2", + "key": "AWSLogs/111111111111/vpcflowlogs/us-east-2/2023/01/28/111111111111_vpcflowlogs_us-east-2_fl-023c6afa025ee5a04_20230128T0930Z_3a9dfd9d.log.gz" + }, + "vpc": { + "version" : "2", + "account-id" : "111111111111", + "interface-id" : "eni-0e250409d410e1290", + "region": "ap-southeast-2", + "vpc-id": "vpc-0d4d4e82b7d743527", + "subnet-id": "subnet-aaaaaaaa012345678", + "az-id": "apse2-az3", + "instance-id": "i-0c50d5961bcb2d47b", + "srcaddr" : "162.142.125.177", + "dstaddr" : "10.0.0.200", + "srcport" : 38471, + "dstport" : 12313, + "protocol" : "6", + "packets" : 1, + "bytes" : 44, + "pkt-src-aws-service": "S3", + "pkt-dst-aws-service": "-", + "flow-direction": "ingress", + "start" : "1674898496", + "end" : "1674898507", + "action" : "ACCEPT", + "log-status" : "OK" + } + }, + "communication": { + "source": { + "address": "162.142.125.177", + "port": 38471, + "packets": 1, + "bytes" : 44 + }, + "destination": { + "address": "10.0.0.200", + "port": 12313 + } + } +}, + { + "@timestamp": "2023-07-17T08:14:05.000Z", + "body": "2 111111111111 eni-0e250409d410e1290 162.142.125.177 10.0.0.200 38471 12313 6 1 44 1674898496 1674898507 ACCEPT OK", + "event": { + "result": "ACCEPT", + "name": "flow_log", + "domain": "vpc.flow_log" + }, + "attributes": { + "data_stream": { + "dataset": "vpc.flow_log", + "namespace": "production", + "type": "logs_vpc" + } + }, + "cloud": { + "provider": "aws", + "account": { + "id": "111111111111" + }, + "region": "ap-southeast-2", + "resource_id": "vpc-0d4d4e82b7d743527", + "platform": "aws_vpc" + }, + "aws": { + "s3": { + "bucket": "centralizedlogging-loghubloggingbucket0fa53b76-t57zyhgb8c2", + "key": "AWSLogs/111111111111/vpcflowlogs/us-east-2/2023/01/28/111111111111_vpcflowlogs_us-east-2_fl-023c6afa025ee5a04_20230128T0930Z_3a9dfd9d.log.gz" + }, + "vpc": { + "version" : "2", + "account-id" : "111111111111", + "interface-id" : "eni-0e250409d410e1290", + "region": "ap-southeast-2", + "vpc-id": "vpc-0d4d4e82b7d743527", + "subnet-id": "subnet-aaaaaaaa012345678", + "az-id": "apse2-az3", + "instance-id": "i-0c50d5961bcb2d47b", + "srcaddr" : "10.0.0.200", + "dstaddr" : "162.142.125.177", + "srcport" : 12313, + "dstport" : 38471, + "protocol" : "6", + "packets" : 1, + "bytes" : 440, + "pkt-src-aws-service": "-", + "pkt-dst-aws-service": "S3", + "flow-direction": "egress", + "start" : "1674898496", + "end" : "1674898507", + "action" : "REJECT", + "log-status" : "OK" + } + }, + "communication": { + "source": { + "address": "10.0.0.200", + "port": 12313, + "packets": 1, + "bytes" : 440 + }, + "destination": { + "address": "162.142.125.177", + "port": 38471 + } + } + }, { + "@timestamp": "2023-07-17T08:14:05.000Z", + "body": "2 111111111111 eni-0e250409d410e1290 162.142.125.177 10.0.0.200 38471 12313 6 1 44 1674898496 1674898507 ACCEPT OK", + "event": { + "result": "ACCEPT", + "name": "flow_log", + "domain": "vpc.flow_log" + }, + "attributes": { + "data_stream": { + "dataset": "vpc.flow_log", + "namespace": "production", + "type": "logs_vpc" + } + }, + "cloud": { + "provider": "aws", + "account": { + "id": "111111111111" + }, + "region": "ap-southeast-2", + "resource_id": "vpc-0d4d4e82b7d743527", + "platform": "aws_vpc" + }, + "aws": { + "s3": { + "bucket": "centralizedlogging-loghubloggingbucket0fa53b76-t57zyhgb8c2", + "key": "AWSLogs/111111111111/vpcflowlogs/us-east-2/2023/01/28/111111111111_vpcflowlogs_us-east-2_fl-023c6afa025ee5a04_20230128T0930Z_3a9dfd9d.log.gz" + }, + "vpc": { + "version" : "2", + "account-id" : "111111111111", + "interface-id" : "eni-0e250409d410e1290", + "region": "ap-southeast-2", + "vpc-id": "vpc-0d4d4e82b7d743527", + "subnet-id": "subnet-aaaaaaaa012345678", + "az-id": "apse2-az3", + "instance-id": "i-0c50d5961bcb2d47b", + "srcaddr" : "162.142.125.177", + "dstaddr" : "10.0.0.200", + "srcport" : 38471, + "dstport" : 12313, + "protocol" : "6", + "packets" : 1, + "bytes" : 44, + "pkt-src-aws-service": "S3", + "pkt-dst-aws-service": "-", + "flow-direction": "ingress", + "start" : "1674898496", + "end" : "1674898507", + "action" : "ACCEPT", + "log-status" : "OK" + } + }, + "communication": { + "source": { + "address": "162.142.125.177", + "port": 38471, + "packets": 1, + "bytes" : 44 + }, + "destination": { + "address": "10.0.0.200", + "port": 12313 + } + } +}, + { + "@timestamp": "2023-07-17T08:14:05.000Z", + "body": "2 111111111111 eni-0e250409d410e1290 162.142.125.177 10.0.0.200 38471 12313 6 1 44 1674898496 1674898507 ACCEPT OK", + "event": { + "result": "ACCEPT", + "name": "flow_log", + "domain": "vpc.flow_log" + }, + "attributes": { + "data_stream": { + "dataset": "vpc.flow_log", + "namespace": "production", + "type": "logs_vpc" + } + }, + "cloud": { + "provider": "aws", + "account": { + "id": "111111111111" + }, + "region": "ap-southeast-2", + "resource_id": "vpc-0d4d4e82b7d743527", + "platform": "aws_vpc" + }, + "aws": { + "s3": { + "bucket": "centralizedlogging-loghubloggingbucket0fa53b76-t57zyhgb8c2", + "key": "AWSLogs/111111111111/vpcflowlogs/us-east-2/2023/01/28/111111111111_vpcflowlogs_us-east-2_fl-023c6afa025ee5a04_20230128T0930Z_3a9dfd9d.log.gz" + }, + "vpc": { + "version" : "2", + "account-id" : "111111111111", + "interface-id" : "eni-0e250409d410e1290", + "region": "ap-southeast-2", + "vpc-id": "vpc-0d4d4e82b7d743527", + "subnet-id": "subnet-aaaaaaaa012345678", + "az-id": "apse2-az3", + "instance-id": "i-0c50d5961bcb2d47b", + "srcaddr" : "10.0.0.200", + "dstaddr" : "162.142.125.177", + "srcport" : 12313, + "dstport" : 38471, + "protocol" : "6", + "packets" : 1, + "bytes" : 440, + "pkt-src-aws-service": "-", + "pkt-dst-aws-service": "S3", + "flow-direction": "egress", + "start" : "1674898496", + "end" : "1674898507", + "action" : "REJECT", + "log-status" : "OK" + } + }, + "communication": { + "source": { + "address": "10.0.0.200", + "port": 12313, + "packets": 1, + "bytes" : 440 + }, + "destination": { + "address": "162.142.125.177", + "port": 38471 + } + } + }, { + "@timestamp": "2023-07-17T08:14:05.000Z", + "body": "2 111111111111 eni-0e250409d410e1290 162.142.125.177 10.0.0.200 38471 12313 6 1 44 1674898496 1674898507 ACCEPT OK", + "event": { + "result": "ACCEPT", + "name": "flow_log", + "domain": "vpc.flow_log" + }, + "attributes": { + "data_stream": { + "dataset": "vpc.flow_log", + "namespace": "production", + "type": "logs_vpc" + } + }, + "cloud": { + "provider": "aws", + "account": { + "id": "111111111111" + }, + "region": "ap-southeast-2", + "resource_id": "vpc-0d4d4e82b7d743527", + "platform": "aws_vpc" + }, + "aws": { + "s3": { + "bucket": "centralizedlogging-loghubloggingbucket0fa53b76-t57zyhgb8c2", + "key": "AWSLogs/111111111111/vpcflowlogs/us-east-2/2023/01/28/111111111111_vpcflowlogs_us-east-2_fl-023c6afa025ee5a04_20230128T0930Z_3a9dfd9d.log.gz" + }, + "vpc": { + "version" : "2", + "account-id" : "111111111111", + "interface-id" : "eni-0e250409d410e1290", + "region": "ap-southeast-2", + "vpc-id": "vpc-0d4d4e82b7d743527", + "subnet-id": "subnet-aaaaaaaa012345678", + "az-id": "apse2-az3", + "instance-id": "i-0c50d5961bcb2d47b", + "srcaddr" : "162.142.125.177", + "dstaddr" : "10.0.0.200", + "srcport" : 38471, + "dstport" : 12313, + "protocol" : "6", + "packets" : 1, + "bytes" : 44, + "pkt-src-aws-service": "S3", + "pkt-dst-aws-service": "-", + "flow-direction": "ingress", + "start" : "1674898496", + "end" : "1674898507", + "action" : "ACCEPT", + "log-status" : "OK" + } + }, + "communication": { + "source": { + "address": "162.142.125.177", + "port": 38471, + "packets": 1, + "bytes" : 44 + }, + "destination": { + "address": "10.0.0.200", + "port": 12313 + } + } +}, + { + "@timestamp": "2023-07-17T08:14:05.000Z", + "body": "2 111111111111 eni-0e250409d410e1290 162.142.125.177 10.0.0.200 38471 12313 6 1 44 1674898496 1674898507 ACCEPT OK", + "event": { + "result": "ACCEPT", + "name": "flow_log", + "domain": "vpc.flow_log" + }, + "attributes": { + "data_stream": { + "dataset": "vpc.flow_log", + "namespace": "production", + "type": "logs_vpc" + } + }, + "cloud": { + "provider": "aws", + "account": { + "id": "111111111111" + }, + "region": "ap-southeast-2", + "resource_id": "vpc-0d4d4e82b7d743527", + "platform": "aws_vpc" + }, + "aws": { + "s3": { + "bucket": "centralizedlogging-loghubloggingbucket0fa53b76-t57zyhgb8c2", + "key": "AWSLogs/111111111111/vpcflowlogs/us-east-2/2023/01/28/111111111111_vpcflowlogs_us-east-2_fl-023c6afa025ee5a04_20230128T0930Z_3a9dfd9d.log.gz" + }, + "vpc": { + "version" : "2", + "account-id" : "111111111111", + "interface-id" : "eni-0e250409d410e1290", + "region": "ap-southeast-2", + "vpc-id": "vpc-0d4d4e82b7d743527", + "subnet-id": "subnet-aaaaaaaa012345678", + "az-id": "apse2-az3", + "instance-id": "i-0c50d5961bcb2d47b", + "srcaddr" : "10.0.0.200", + "dstaddr" : "162.142.125.177", + "srcport" : 12313, + "dstport" : 38471, + "protocol" : "6", + "packets" : 1, + "bytes" : 440, + "pkt-src-aws-service": "-", + "pkt-dst-aws-service": "S3", + "flow-direction": "egress", + "start" : "1674898496", + "end" : "1674898507", + "action" : "REJECT", + "log-status" : "OK" + } + }, + "communication": { + "source": { + "address": "10.0.0.200", + "port": 12313, + "packets": 1, + "bytes" : 440 + }, + "destination": { + "address": "162.142.125.177", + "port": 38471 + } + } + }, { + "@timestamp": "2023-07-17T08:14:05.000Z", + "body": "2 111111111111 eni-0e250409d410e1290 162.142.125.177 10.0.0.200 38471 12313 6 1 44 1674898496 1674898507 ACCEPT OK", + "event": { + "result": "ACCEPT", + "name": "flow_log", + "domain": "vpc.flow_log" + }, + "attributes": { + "data_stream": { + "dataset": "vpc.flow_log", + "namespace": "production", + "type": "logs_vpc" + } + }, + "cloud": { + "provider": "aws", + "account": { + "id": "111111111111" + }, + "region": "ap-southeast-2", + "resource_id": "vpc-0d4d4e82b7d743527", + "platform": "aws_vpc" + }, + "aws": { + "s3": { + "bucket": "centralizedlogging-loghubloggingbucket0fa53b76-t57zyhgb8c2", + "key": "AWSLogs/111111111111/vpcflowlogs/us-east-2/2023/01/28/111111111111_vpcflowlogs_us-east-2_fl-023c6afa025ee5a04_20230128T0930Z_3a9dfd9d.log.gz" + }, + "vpc": { + "version" : "2", + "account-id" : "111111111111", + "interface-id" : "eni-0e250409d410e1290", + "region": "ap-southeast-2", + "vpc-id": "vpc-0d4d4e82b7d743527", + "subnet-id": "subnet-aaaaaaaa012345678", + "az-id": "apse2-az3", + "instance-id": "i-0c50d5961bcb2d47b", + "srcaddr" : "162.142.125.177", + "dstaddr" : "10.0.0.200", + "srcport" : 38471, + "dstport" : 12313, + "protocol" : "6", + "packets" : 1, + "bytes" : 44, + "pkt-src-aws-service": "S3", + "pkt-dst-aws-service": "-", + "flow-direction": "ingress", + "start" : "1674898496", + "end" : "1674898507", + "action" : "ACCEPT", + "log-status" : "OK" + } + }, + "communication": { + "source": { + "address": "162.142.125.177", + "port": 38471, + "packets": 1, + "bytes" : 44 + }, + "destination": { + "address": "10.0.0.200", + "port": 12313 + } + } +}, + { + "@timestamp": "2023-07-17T08:14:05.000Z", + "body": "2 111111111111 eni-0e250409d410e1290 162.142.125.177 10.0.0.200 38471 12313 6 1 44 1674898496 1674898507 ACCEPT OK", + "event": { + "result": "ACCEPT", + "name": "flow_log", + "domain": "vpc.flow_log" + }, + "attributes": { + "data_stream": { + "dataset": "vpc.flow_log", + "namespace": "production", + "type": "logs_vpc" + } + }, + "cloud": { + "provider": "aws", + "account": { + "id": "111111111111" + }, + "region": "ap-southeast-2", + "resource_id": "vpc-0d4d4e82b7d743527", + "platform": "aws_vpc" + }, + "aws": { + "s3": { + "bucket": "centralizedlogging-loghubloggingbucket0fa53b76-t57zyhgb8c2", + "key": "AWSLogs/111111111111/vpcflowlogs/us-east-2/2023/01/28/111111111111_vpcflowlogs_us-east-2_fl-023c6afa025ee5a04_20230128T0930Z_3a9dfd9d.log.gz" + }, + "vpc": { + "version" : "2", + "account-id" : "111111111111", + "interface-id" : "eni-0e250409d410e1290", + "region": "ap-southeast-2", + "vpc-id": "vpc-0d4d4e82b7d743527", + "subnet-id": "subnet-aaaaaaaa012345678", + "az-id": "apse2-az3", + "instance-id": "i-0c50d5961bcb2d47b", + "srcaddr" : "10.0.0.200", + "dstaddr" : "162.142.125.177", + "srcport" : 12313, + "dstport" : 38471, + "protocol" : "6", + "packets" : 1, + "bytes" : 440, + "pkt-src-aws-service": "-", + "pkt-dst-aws-service": "S3", + "flow-direction": "egress", + "start" : "1674898496", + "end" : "1674898507", + "action" : "REJECT", + "log-status" : "OK" + } + }, + "communication": { + "source": { + "address": "10.0.0.200", + "port": 12313, + "packets": 1, + "bytes" : 440 + }, + "destination": { + "address": "162.142.125.177", + "port": 38471 + } + } + }, { + "@timestamp": "2023-07-17T08:14:05.000Z", + "body": "2 111111111111 eni-0e250409d410e1290 162.142.125.177 10.0.0.200 38471 12313 6 1 44 1674898496 1674898507 ACCEPT OK", + "event": { + "result": "ACCEPT", + "name": "flow_log", + "domain": "vpc.flow_log" + }, + "attributes": { + "data_stream": { + "dataset": "vpc.flow_log", + "namespace": "production", + "type": "logs_vpc" + } + }, + "cloud": { + "provider": "aws", + "account": { + "id": "111111111111" + }, + "region": "ap-southeast-2", + "resource_id": "vpc-0d4d4e82b7d743527", + "platform": "aws_vpc" + }, + "aws": { + "s3": { + "bucket": "centralizedlogging-loghubloggingbucket0fa53b76-t57zyhgb8c2", + "key": "AWSLogs/111111111111/vpcflowlogs/us-east-2/2023/01/28/111111111111_vpcflowlogs_us-east-2_fl-023c6afa025ee5a04_20230128T0930Z_3a9dfd9d.log.gz" + }, + "vpc": { + "version" : "2", + "account-id" : "111111111111", + "interface-id" : "eni-0e250409d410e1290", + "region": "ap-southeast-2", + "vpc-id": "vpc-0d4d4e82b7d743527", + "subnet-id": "subnet-aaaaaaaa012345678", + "az-id": "apse2-az3", + "instance-id": "i-0c50d5961bcb2d47b", + "srcaddr" : "162.142.125.177", + "dstaddr" : "10.0.0.200", + "srcport" : 38471, + "dstport" : 12313, + "protocol" : "6", + "packets" : 1, + "bytes" : 44, + "pkt-src-aws-service": "S3", + "pkt-dst-aws-service": "-", + "flow-direction": "ingress", + "start" : "1674898496", + "end" : "1674898507", + "action" : "ACCEPT", + "log-status" : "OK" + } + }, + "communication": { + "source": { + "address": "162.142.125.177", + "port": 38471, + "packets": 1, + "bytes" : 44 + }, + "destination": { + "address": "10.0.0.200", + "port": 12313 + } + } +}, + { + "@timestamp": "2023-07-17T08:14:05.000Z", + "body": "2 111111111111 eni-0e250409d410e1290 162.142.125.177 10.0.0.200 38471 12313 6 1 44 1674898496 1674898507 ACCEPT OK", + "event": { + "result": "ACCEPT", + "name": "flow_log", + "domain": "vpc.flow_log" + }, + "attributes": { + "data_stream": { + "dataset": "vpc.flow_log", + "namespace": "production", + "type": "logs_vpc" + } + }, + "cloud": { + "provider": "aws", + "account": { + "id": "111111111111" + }, + "region": "ap-southeast-2", + "resource_id": "vpc-0d4d4e82b7d743527", + "platform": "aws_vpc" + }, + "aws": { + "s3": { + "bucket": "centralizedlogging-loghubloggingbucket0fa53b76-t57zyhgb8c2", + "key": "AWSLogs/111111111111/vpcflowlogs/us-east-2/2023/01/28/111111111111_vpcflowlogs_us-east-2_fl-023c6afa025ee5a04_20230128T0930Z_3a9dfd9d.log.gz" + }, + "vpc": { + "version" : "2", + "account-id" : "111111111111", + "interface-id" : "eni-0e250409d410e1290", + "region": "ap-southeast-2", + "vpc-id": "vpc-0d4d4e82b7d743527", + "subnet-id": "subnet-aaaaaaaa012345678", + "az-id": "apse2-az3", + "instance-id": "i-0c50d5961bcb2d47b", + "srcaddr" : "10.0.0.200", + "dstaddr" : "162.142.125.177", + "srcport" : 12313, + "dstport" : 38471, + "protocol" : "6", + "packets" : 1, + "bytes" : 440, + "pkt-src-aws-service": "-", + "pkt-dst-aws-service": "S3", + "flow-direction": "egress", + "start" : "1674898496", + "end" : "1674898507", + "action" : "REJECT", + "log-status" : "OK" + } + }, + "communication": { + "source": { + "address": "10.0.0.200", + "port": 12313, + "packets": 1, + "bytes" : 440 + }, + "destination": { + "address": "162.142.125.177", + "port": 38471 + } + } + } +] \ No newline at end of file diff --git a/visualizations/observability/http/dql-http-network-graph-widget.json b/visualizations/observability/http/dql-http-network-graph-widget.json new file mode 100644 index 0000000..9e62536 --- /dev/null +++ b/visualizations/observability/http/dql-http-network-graph-widget.json @@ -0,0 +1,620 @@ +{ + "$schema": "https://vega.github.io/schema/vega/v5.json", + "data": [ + { + "name": "rawData", + "format": { + "property": "aggregations.table.buckets" + }, + "transform": [ + { + "type": "formula", + "expr": "datum.key.stk1", + "as": "stk1" + }, + { + "type": "formula", + "expr": "datum.key.stk2", + "as": "stk2" + }, + { + "type": "formula", + "expr": "datum.doc_count", + "as": "size" + } + ], + "url": { + "index": "ss4o_logs_vpc-*-*", + "body": { + "size": 0, + "aggs": { + "table": { + "composite": { + "size": 10000, + "sources": [ + { + "stk1": { + "terms": { + "field": "aws.vpc.srcaddr" + } + } + }, + { + "stk2": { + "terms": { + "field": "aws.vpc.dstaddr" + } + } + } + ] + } + } + } + } + } + }, + { + "name": "nodes", + "source": "rawData", + "transform": [ + { + "type": "filter", + "expr": "!groupSelector || groupSelector.stk1 == datum.stk1 || groupSelector.stk2 == datum.stk2" + }, + { + "type": "formula", + "expr": "datum.stk1+datum.stk2", + "as": "key" + }, + { + "type": "fold", + "fields": [ + "stk1", + "stk2" + ], + "as": [ + "stack", + "grpId" + ] + }, + { + "type": "formula", + "expr": "datum.stack == 'stk1' ? datum.stk1+datum.stk2 : datum.stk2+datum.stk1", + "as": "sortField" + }, + { + "type": "stack", + "groupby": [ + "stack" + ], + "sort": { + "field": "sortField", + "order": "descending" + }, + "field": "size" + }, + { + "type": "formula", + "expr": "(datum.y0+datum.y1)/2", + "as": "yc" + } + ] + }, + { + "name": "groups", + "source": "nodes", + "transform": [ + { + "type": "aggregate", + "groupby": [ + "stack", + "grpId" + ], + "fields": [ + "size" + ], + "ops": [ + "sum" + ], + "as": [ + "total" + ] + }, + { + "type": "stack", + "groupby": [ + "stack" + ], + "sort": { + "field": "grpId", + "order": "descending" + }, + "field": "total" + }, + { + "type": "formula", + "expr": "scale('y', datum.y0)", + "as": "scaledY0" + }, + { + "type": "formula", + "expr": "scale('y', datum.y1)", + "as": "scaledY1" + }, + { + "type": "formula", + "expr": "datum.stack == 'stk1'", + "as": "rightLabel" + }, + { + "type": "formula", + "expr": "datum.total/domain('y')[1]", + "as": "percentage" + } + ] + }, + { + "name": "destinationNodes", + "source": "nodes", + "transform": [ + { + "type": "filter", + "expr": "datum.stack == 'stk2'" + } + ] + }, + { + "name": "edges", + "source": "nodes", + "transform": [ + { + "type": "filter", + "expr": "datum.stack == 'stk1'" + }, + { + "type": "lookup", + "from": "destinationNodes", + "key": "key", + "fields": [ + "key" + ], + "as": [ + "target" + ] + }, + { + "type": "linkpath", + "orient": "horizontal", + "shape": "diagonal", + "sourceY": { + "expr": "scale('y', datum.yc)" + }, + "sourceX": { + "expr": "scale('x', 'stk1') + bandwidth('x')" + }, + "targetY": { + "expr": "scale('y', datum.target.yc)" + }, + "targetX": { + "expr": "scale('x', 'stk2')" + } + }, + { + "type": "formula", + "expr": "range('y')[0]-scale('y', datum.size)", + "as": "strokeWidth" + }, + { + "type": "formula", + "expr": "datum.size/domain('y')[1]", + "as": "percentage" + } + ] + } + ], + "scales": [ + { + "name": "x", + "type": "band", + "range": "width", + "domain": [ + "stk1", + "stk2" + ], + "paddingOuter": 0.05, + "paddingInner": 0.95 + }, + { + "name": "y", + "type": "linear", + "range": "height", + "domain": { + "data": "nodes", + "field": "y1" + } + }, + { + "name": "color", + "type": "ordinal", + "range": "category", + "domain": { + "data": "rawData", + "field": "stk1" + } + }, + { + "name": "stackNames", + "type": "ordinal", + "range": [ + "Source", + "Destination" + ], + "domain": [ + "stk1", + "stk2" + ] + } + ], + "axes": [ + { + "orient": "bottom", + "scale": "x", + "encode": { + "labels": { + "update": { + "text": { + "scale": "stackNames", + "field": "value" + } + } + } + } + }, + { + "orient": "left", + "scale": "y" + } + ], + "marks": [ + { + "type": "path", + "name": "edgeMark", + "from": { + "data": "edges" + }, + "clip": true, + "encode": { + "update": { + "stroke": [ + { + "test": "groupSelector && groupSelector.stack=='stk1'", + "scale": "color", + "field": "stk2" + }, + { + "scale": "color", + "field": "stk1" + } + ], + "strokeWidth": { + "field": "strokeWidth" + }, + "path": { + "field": "path" + }, + "strokeOpacity": { + "signal": "!groupSelector && (groupHover.stk1 == datum.stk1 || groupHover.stk2 == datum.stk2) ? 0.9 : 0.3" + }, + "zindex": { + "signal": "!groupSelector && (groupHover.stk1 == datum.stk1 || groupHover.stk2 == datum.stk2) ? 1 : 0" + }, + "tooltip": { + "signal": "datum.stk1 + ' → ' + datum.stk2 + '\t' + format(datum.size, ',.0f') + ' (' + format(datum.percentage, '.1%') + ')'" + } + }, + "hover": { + "strokeOpacity": { + "value": 1 + } + } + } + }, + { + "type": "rect", + "name": "groupMark", + "from": { + "data": "groups" + }, + "encode": { + "enter": { + "fill": { + "scale": "color", + "field": "grpId" + }, + "width": { + "scale": "x", + "band": 1 + } + }, + "update": { + "x": { + "scale": "x", + "field": "stack" + }, + "y": { + "field": "scaledY0" + }, + "y2": { + "field": "scaledY1" + }, + "fillOpacity": { + "value": 0.6 + }, + "tooltip": { + "signal": "datum.grpId + ' ' + format(datum.total, ',.0f') + ' (' + format(datum.percentage, '.1%') + ')'" + } + }, + "hover": { + "fillOpacity": { + "value": 1 + } + } + } + }, + { + "type": "text", + "from": { + "data": "groups" + }, + "interactive": false, + "encode": { + "update": { + "x": { + "signal": "scale('x', datum.stack) + (datum.rightLabel ? bandwidth('x') + 8 : -8)" + }, + "yc": { + "signal": "(datum.scaledY0 + datum.scaledY1)/2" + }, + "align": { + "signal": "datum.rightLabel ? 'left' : 'right'" + }, + "baseline": { + "value": "middle" + }, + "fontWeight": { + "value": "bold" + }, + "text": { + "signal": "abs(datum.scaledY0-datum.scaledY1) > 13 ? datum.grpId : ''" + } + } + } + }, + { + "type": "group", + "data": [ + { + "name": "dataForShowAll", + "values": [ + {} + ], + "transform": [ + { + "type": "filter", + "expr": "groupSelector" + } + ] + } + ], + "encode": { + "enter": { + "xc": { + "signal": "width/2" + }, + "y": { + "value": 30 + }, + "width": { + "value": 80 + }, + "height": { + "value": 30 + } + } + }, + "marks": [ + { + "type": "group", + "name": "groupReset", + "from": { + "data": "dataForShowAll" + }, + "encode": { + "enter": { + "cornerRadius": { + "value": 6 + }, + "fill": { + "value": "#F5F7FA" + }, + "stroke": { + "value": "#c1c1c1" + }, + "strokeWidth": { + "value": 2 + }, + "height": { + "field": { + "group": "height" + } + }, + "width": { + "field": { + "group": "width" + } + } + }, + "update": { + "opacity": { + "value": 1 + } + }, + "hover": { + "opacity": { + "value": 0.7 + } + } + }, + "marks": [ + { + "type": "text", + "interactive": false, + "encode": { + "enter": { + "xc": { + "field": { + "group": "width" + }, + "mult": 0.5 + }, + "yc": { + "field": { + "group": "height" + }, + "mult": 0.5, + "offset": 2 + }, + "align": { + "value": "center" + }, + "baseline": { + "value": "middle" + }, + "fontWeight": { + "value": "bold" + }, + "text": { + "value": "Show All" + } + } + } + } + ] + } + ] + } + ], + "signals": [ + { + "name": "groupHover", + "value": {}, + "on": [ + { + "events": "@groupMark:mouseover", + "update": "{stk1:datum.stack=='stk1' && datum.grpId, stk2:datum.stack=='stk2' && datum.grpId}" + }, + { + "events": "mouseout", + "update": "{}" + } + ] + }, + { + "name": "groupSelector", + "value": false, + "on": [ + { + "events": "@groupMark:click!", + "update": "{stack:datum.stack, stk1:datum.stack=='stk1' && datum.grpId, stk2:datum.stack=='stk2' && datum.grpId}" + }, + { + "events": [ + { + "type": "click", + "markname": "groupReset" + }, + { + "type": "dblclick" + } + ], + "update": "false" + } + ] + } + ], + "config": { + "range": { + "category": [ + "#54B399", + "#6092C0", + "#D36086", + "#9170B8", + "#CA8EAE", + "#D6BF57", + "#B9A888", + "#DA8B45", + "#AA6556", + "#E7664C" + ] + }, + "arc": { + "fill": "#54B399" + }, + "area": { + "fill": "#54B399" + }, + "line": { + "stroke": "#54B399" + }, + "path": { + "stroke": "#54B399" + }, + "rect": { + "fill": "#54B399" + }, + "rule": { + "stroke": "#54B399" + }, + "shape": { + "stroke": "#54B399" + }, + "symbol": { + "fill": "#54B399" + }, + "text": { + "fill": "#dfe5ef" + }, + "trail": { + "fill": "#54B399" + }, + "title": { + "color": "#dfe3e8" + }, + "style": { + "guide-label": { + "fill": "#8d98a3" + }, + "guide-title": { + "fill": "#dfe3e8" + }, + "group-title": { + "fill": "#dfe3e8" + }, + "group-subtitle": { + "fill": "#dfe3e8" + } + }, + "axis": { + "tickColor": "#293847", + "domainColor": "#293847", + "gridColor": "#293847" + }, + "background": "transparent" + }, + "width": "container", + "height": "container", + "autosize": { + "type": "fit", + "contains": "padding" + } +} \ No newline at end of file diff --git a/visualizations/observability/http/dql-query.json b/visualizations/observability/http/dql-query.json new file mode 100644 index 0000000..ad296be --- /dev/null +++ b/visualizations/observability/http/dql-query.json @@ -0,0 +1,61 @@ +[ + { + "http-network-graph-widget":[ { + "index": "ss4o_logs_vpc-*-*", + "body": { + "size": 0, + "aggs": { + "table": { + "composite": { + "size": 10000, + "sources": [ + { + "stk1": { + "terms": { + "field": "aws.vpc.srcaddr" + } + } + }, + { + "stk2": { + "terms": { + "field": "aws.vpc.dstaddr" + } + } + } + ] + } + } + } + } + },{ + "index": "ss4o_logs_vpc-*-*", + "body": { + "size": 0, + "aggs": { + "table": { + "composite": { + "size": 10000, + "sources": [ + { + "stk1": { + "terms": { + "field": "communication.source.address" + } + } + }, + { + "stk2": { + "terms": { + "field": "communication.destination.address" + } + } + } + ] + } + } + } + } + }] + } +] \ No newline at end of file diff --git a/visualizations/observability/http/http-network-graph.ndjson b/visualizations/observability/http/http-network-graph.ndjson new file mode 100644 index 0000000..bd45575 --- /dev/null +++ b/visualizations/observability/http/http-network-graph.ndjson @@ -0,0 +1,2 @@ +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"[AWS VPC Flow Logs 1.0] Flow","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"[AWS VPC Flow Logs 1.0] Flow\",\"type\":\"vega\",\"aggs\":[],\"params\":{\"spec\":\"{ \\n $schema: https://vega.github.io/schema/vega/v5.json\\n data: [\\n\\t{\\n \\t// query OpenSearch based on the currently selected time range and filter string\\n \\tname: rawData\\n \\turl: {\\n \\tindex: ss4o_logs_vpc-*-*\\n \\tbody: {\\n \\tsize: 0\\n \\taggs: {\\n \\ttable: {\\n \\tcomposite: {\\n \\tsize: 10000\\n \\tsources: [\\n \\t{\\n \\tstk1: {\\n \\tterms: {field: \\\"aws.vpc.srcaddr\\\"}\\n \\t}\\n \\t}\\n \\t{\\n \\tstk2: {\\n \\tterms: {field: \\\"aws.vpc.dstaddr\\\"}\\n \\t}\\n \\t}\\n \\t]\\n \\t}\\n \\t}\\n \\t}\\n \\t}\\n \\t}\\n \\t// From the result, take just the data we are interested in\\n \\tformat: {property: \\\"aggregations.table.buckets\\\"}\\n \\t// Convert key.stk1 -> stk1 for simpler access below\\n \\ttransform: [\\n \\t{type: \\\"formula\\\", expr: \\\"datum.key.stk1\\\", as: \\\"stk1\\\"}\\n \\t{type: \\\"formula\\\", expr: \\\"datum.key.stk2\\\", as: \\\"stk2\\\"}\\n \\t{type: \\\"formula\\\", expr: \\\"datum.doc_count\\\", as: \\\"size\\\"}\\n \\t]\\n\\t}\\n\\t{\\n \\tname: nodes\\n \\tsource: rawData\\n \\ttransform: [\\n \\t// when a country is selected, filter out unrelated data\\n \\t{\\n \\ttype: filter\\n \\texpr: !groupSelector || groupSelector.stk1 == datum.stk1 || groupSelector.stk2 == datum.stk2\\n \\t}\\n \\t// Set new key for later lookups - identifies each node\\n \\t{type: \\\"formula\\\", expr: \\\"datum.stk1+datum.stk2\\\", as: \\\"key\\\"}\\n \\t// instead of each table row, create two new rows,\\n \\t// one for the source (stack=stk1) and one for destination node (stack=stk2).\\n \\t// The country code stored in stk1 and stk2 fields is placed into grpId field.\\n \\t{\\n \\ttype: fold\\n \\tfields: [\\\"stk1\\\", \\\"stk2\\\"]\\n \\tas: [\\\"stack\\\", \\\"grpId\\\"]\\n \\t}\\n \\t// Create a sortkey, different for stk1 and stk2 stacks.\\n \\t{\\n \\ttype: formula\\n \\texpr: datum.stack == 'stk1' ? datum.stk1+datum.stk2 : datum.stk2+datum.stk1\\n \\tas: sortField\\n \\t}\\n \\t// Calculate y0 and y1 positions for stacking nodes one on top of the other,\\n \\t// independently for each stack, and ensuring they are in the proper order,\\n \\t// alphabetical from the top (reversed on the y axis)\\n \\t{\\n \\ttype: stack\\n \\tgroupby: [\\\"stack\\\"]\\n \\tsort: {field: \\\"sortField\\\", order: \\\"descending\\\"}\\n \\tfield: size\\n \\t}\\n \\t// calculate vertical center point for each node, used to draw edges\\n \\t{type: \\\"formula\\\", expr: \\\"(datum.y0+datum.y1)/2\\\", as: \\\"yc\\\"}\\n \\t]\\n\\t}\\n\\t{\\n \\tname: groups\\n \\tsource: nodes\\n \\ttransform: [\\n \\t// combine all nodes into country groups, summing up the doc counts\\n \\t{\\n \\ttype: aggregate\\n \\tgroupby: [\\\"stack\\\", \\\"grpId\\\"]\\n \\tfields: [\\\"size\\\"]\\n \\tops: [\\\"sum\\\"]\\n \\tas: [\\\"total\\\"]\\n \\t}\\n \\t// re-calculate the stacking y0,y1 values\\n \\t{\\n \\ttype: stack\\n \\tgroupby: [\\\"stack\\\"]\\n \\tsort: {field: \\\"grpId\\\", order: \\\"descending\\\"}\\n \\tfield: total\\n \\t}\\n \\t// project y0 and y1 values to screen coordinates\\n \\t// doing it once here instead of doing it several times in marks\\n \\t{type: \\\"formula\\\", expr: \\\"scale('y', datum.y0)\\\", as: \\\"scaledY0\\\"}\\n \\t{type: \\\"formula\\\", expr: \\\"scale('y', datum.y1)\\\", as: \\\"scaledY1\\\"}\\n \\t// boolean flag if the label should be on the right of the stack\\n \\t{type: \\\"formula\\\", expr: \\\"datum.stack == 'stk1'\\\", as: \\\"rightLabel\\\"}\\n \\t// Calculate traffic percentage for this country using \\\"y\\\" scale\\n \\t// domain upper bound, which represents the total traffic\\n \\t{\\n \\ttype: formula\\n \\texpr: datum.total/domain('y')[1]\\n \\tas: percentage\\n \\t}\\n \\t]\\n\\t}\\n\\t{\\n \\t// This is a temp lookup table with all the 'stk2' stack nodes\\n \\tname: destinationNodes\\n \\tsource: nodes\\n \\ttransform: [\\n \\t{type: \\\"filter\\\", expr: \\\"datum.stack == 'stk2'\\\"}\\n \\t]\\n\\t}\\n\\t{\\n \\tname: edges\\n \\tsource: nodes\\n \\ttransform: [\\n \\t// we only want nodes from the left stack\\n \\t{type: \\\"filter\\\", expr: \\\"datum.stack == 'stk1'\\\"}\\n \\t// find corresponding node from the right stack, keep it as \\\"target\\\"\\n \\t{\\n \\ttype: lookup\\n \\tfrom: destinationNodes\\n \\tkey: key\\n \\tfields: [\\\"key\\\"]\\n \\tas: [\\\"target\\\"]\\n \\t}\\n \\t// calculate SVG link path between stk1 and stk2 stacks for the node pair\\n \\t{\\n \\ttype: linkpath\\n \\torient: horizontal\\n \\tshape: diagonal\\n \\tsourceY: {expr: \\\"scale('y', datum.yc)\\\"}\\n \\tsourceX: {expr: \\\"scale('x', 'stk1') + bandwidth('x')\\\"}\\n \\ttargetY: {expr: \\\"scale('y', datum.target.yc)\\\"}\\n \\ttargetX: {expr: \\\"scale('x', 'stk2')\\\"}\\n \\t}\\n \\t// A little trick to calculate the thickness of the line.\\n \\t// The value needs to be the same as the hight of the node, but scaling\\n \\t// size to screen's height gives inversed value because screen's Y\\n \\t// coordinate goes from the top to the bottom, whereas the graph's Y=0\\n \\t// is at the bottom. So subtracting scaled doc count from screen height\\n \\t// (which is the \\\"lower\\\" bound of the \\\"y\\\" scale) gives us the right value\\n \\t{\\n \\ttype: formula\\n \\texpr: range('y')[0]-scale('y', datum.size)\\n \\tas: strokeWidth\\n \\t}\\n \\t// Tooltip needs individual link's percentage of all traffic\\n \\t{\\n \\ttype: formula\\n \\texpr: datum.size/domain('y')[1]\\n \\tas: percentage\\n \\t}\\n \\t]\\n\\t}\\n ]\\n scales: [\\n\\t{\\n \\t// calculates horizontal stack positioning\\n \\tname: x\\n \\ttype: band\\n \\trange: width\\n \\tdomain: [\\\"stk1\\\", \\\"stk2\\\"]\\n \\tpaddingOuter: 0.05\\n \\tpaddingInner: 0.95\\n\\t}\\n\\t{\\n \\t// this scale goes up as high as the highest y1 value of all nodes\\n \\tname: y\\n \\ttype: linear\\n \\trange: height\\n \\tdomain: {data: \\\"nodes\\\", field: \\\"y1\\\"}\\n\\t}\\n\\t{\\n \\t// use rawData to ensure the colors stay the same when clicking.\\n \\tname: color\\n \\ttype: ordinal\\n \\trange: category\\n \\tdomain: {data: \\\"rawData\\\", field: \\\"stk1\\\"}\\n\\t}\\n\\t{\\n \\t// this scale is used to map internal ids (stk1, stk2) to stack names\\n \\tname: stackNames\\n \\ttype: ordinal\\n \\trange: [\\\"Source\\\", \\\"Destination\\\"]\\n \\tdomain: [\\\"stk1\\\", \\\"stk2\\\"]\\n\\t}\\n ]\\n axes: [\\n\\t{\\n \\t// x axis should use custom label formatting to print proper stack names\\n \\torient: bottom\\n \\tscale: x\\n \\tencode: {\\n \\tlabels: {\\n \\tupdate: {\\n \\ttext: {scale: \\\"stackNames\\\", field: \\\"value\\\"}\\n \\t}\\n \\t}\\n \\t}\\n\\t}\\n\\t{orient: \\\"left\\\", scale: \\\"y\\\"}\\n ]\\n marks: [\\n\\t{\\n \\t// draw the connecting line between stacks\\n \\ttype: path\\n \\tname: edgeMark\\n \\tfrom: {data: \\\"edges\\\"}\\n \\t// this prevents some autosizing issues with large strokeWidth for paths\\n \\tclip: true\\n \\tencode: {\\n \\tupdate: {\\n \\t// By default use color of the left node, except when showing traffic\\n \\t// from just one country, in which case use destination color.\\n \\tstroke: [\\n \\t{\\n \\ttest: groupSelector && groupSelector.stack=='stk1'\\n \\tscale: color\\n \\tfield: stk2\\n \\t}\\n \\t{scale: \\\"color\\\", field: \\\"stk1\\\"}\\n \\t]\\n \\tstrokeWidth: {field: \\\"strokeWidth\\\"}\\n \\tpath: {field: \\\"path\\\"}\\n \\t// when showing all traffic, and hovering over a country,\\n \\t// highlight the traffic from that country.\\n \\tstrokeOpacity: {\\n \\tsignal: !groupSelector && (groupHover.stk1 == datum.stk1 || groupHover.stk2 == datum.stk2) ? 0.9 : 0.3\\n \\t}\\n \\t// Ensure that the hover-selected edges show on top\\n \\tzindex: {\\n \\tsignal: !groupSelector && (groupHover.stk1 == datum.stk1 || groupHover.stk2 == datum.stk2) ? 1 : 0\\n \\t}\\n \\t// format tooltip string\\n \\ttooltip: {\\n \\tsignal: datum.stk1 + ' → ' + datum.stk2 + '\\t' + format(datum.size, ',.0f') + ' (' + format(datum.percentage, '.1%') + ')'\\n \\t}\\n \\t}\\n \\t// Simple mouseover highlighting of a single line\\n \\thover: {\\n \\tstrokeOpacity: {value: 1}\\n \\t}\\n \\t}\\n\\t}\\n\\t{\\n \\t// draw stack groups (countries)\\n \\ttype: rect\\n \\tname: groupMark\\n \\tfrom: {data: \\\"groups\\\"}\\n \\tencode: {\\n \\tenter: {\\n \\tfill: {scale: \\\"color\\\", field: \\\"grpId\\\"}\\n \\twidth: {scale: \\\"x\\\", band: 1}\\n \\t}\\n \\tupdate: {\\n \\tx: {scale: \\\"x\\\", field: \\\"stack\\\"}\\n \\ty: {field: \\\"scaledY0\\\"}\\n \\ty2: {field: \\\"scaledY1\\\"}\\n \\tfillOpacity: {value: 0.6}\\n \\ttooltip: {\\n \\tsignal: datum.grpId + ' ' + format(datum.total, ',.0f') + ' (' + format(datum.percentage, '.1%') + ')'\\n \\t}\\n \\t}\\n \\thover: {\\n \\tfillOpacity: {value: 1}\\n \\t}\\n \\t}\\n\\t}\\n\\t{\\n \\t// draw country code labels on the inner side of the stack\\n \\ttype: text\\n \\tfrom: {data: \\\"groups\\\"}\\n \\t// don't process events for the labels - otherwise line mouseover is unclean\\n \\tinteractive: false\\n \\tencode: {\\n \\tupdate: {\\n \\t// depending on which stack it is, position x with some padding\\n \\tx: {\\n \\tsignal: scale('x', datum.stack) + (datum.rightLabel ? bandwidth('x') + 8 : -8)\\n \\t}\\n \\t// middle of the group\\n \\tyc: {signal: \\\"(datum.scaledY0 + datum.scaledY1)/2\\\"}\\n \\talign: {signal: \\\"datum.rightLabel ? 'left' : 'right'\\\"}\\n \\tbaseline: {value: \\\"middle\\\"}\\n \\tfontWeight: {value: \\\"bold\\\"}\\n \\t// only show text label if the group's height is large enough\\n \\ttext: {signal: \\\"abs(datum.scaledY0-datum.scaledY1) > 13 ? datum.grpId : ''\\\"}\\n \\t}\\n \\t}\\n\\t}\\n\\t{\\n \\t// Create a \\\"show all\\\" button. Shown only when a country is selected.\\n \\ttype: group\\n \\tdata: [\\n \\t// We need to make the button show only when groupSelector signal is true.\\n \\t// Each mark is drawn as many times as there are elements in the backing data.\\n \\t// Which means that if values list is empty, it will not be drawn.\\n \\t// Here I create a data source with one empty object, and filter that list\\n \\t// based on the signal value. This can only be done in a group.\\n \\t{\\n \\tname: dataForShowAll\\n \\tvalues: [{}]\\n \\ttransform: [{type: \\\"filter\\\", expr: \\\"groupSelector\\\"}]\\n \\t}\\n \\t]\\n \\t// Set button size and positioning\\n \\tencode: {\\n \\tenter: {\\n \\txc: {signal: \\\"width/2\\\"}\\n \\ty: {value: 30}\\n \\twidth: {value: 80}\\n \\theight: {value: 30}\\n \\t}\\n \\t}\\n \\tmarks: [\\n \\t{\\n \\t// This group is shown as a button with rounded corners.\\n \\ttype: group\\n \\t// mark name allows signal capturing\\n \\tname: groupReset\\n \\t// Only shows button if dataForShowAll has values.\\n \\tfrom: {data: \\\"dataForShowAll\\\"}\\n \\tencode: {\\n \\tenter: {\\n \\tcornerRadius: {value: 6}\\n \\tfill: {value: \\\"#F5F7FA\\\"}\\n \\tstroke: {value: \\\"#c1c1c1\\\"}\\n \\tstrokeWidth: {value: 2}\\n \\t// use parent group's size\\n \\theight: {\\n \\tfield: {group: \\\"height\\\"}\\n \\t}\\n \\twidth: {\\n \\tfield: {group: \\\"width\\\"}\\n \\t}\\n \\t}\\n \\tupdate: {\\n \\t// groups are transparent by default\\n \\topacity: {value: 1}\\n \\t}\\n \\thover: {\\n \\topacity: {value: 0.7}\\n \\t}\\n \\t}\\n \\tmarks: [\\n \\t{\\n \\ttype: text\\n \\t// if true, it will prevent clicking on the button when over text.\\n \\tinteractive: false\\n \\tencode: {\\n \\tenter: {\\n \\t// center text in the paren group\\n \\txc: {\\n \\tfield: {group: \\\"width\\\"}\\n \\tmult: 0.5\\n \\t}\\n \\tyc: {\\n \\tfield: {group: \\\"height\\\"}\\n \\tmult: 0.5\\n \\toffset: 2\\n \\t}\\n \\talign: {value: \\\"center\\\"}\\n \\tbaseline: {value: \\\"middle\\\"}\\n \\tfontWeight: {value: \\\"bold\\\"}\\n \\ttext: {value: \\\"Show All\\\"}\\n \\t}\\n \\t}\\n \\t}\\n \\t]\\n \\t}\\n \\t]\\n\\t}\\n ]\\n signals: [\\n\\t{\\n \\t// used to highlight traffic to/from the same country\\n \\tname: groupHover\\n \\tvalue: {}\\n \\ton: [\\n \\t{\\n \\tevents: @groupMark:mouseover\\n \\tupdate: \\\"{stk1:datum.stack=='stk1' && datum.grpId, stk2:datum.stack=='stk2' && datum.grpId}\\\"\\n \\t}\\n \\t{events: \\\"mouseout\\\", update: \\\"{}\\\"}\\n \\t]\\n\\t}\\n\\t// used to filter only the data related to the selected country\\n\\t{\\n \\tname: groupSelector\\n \\tvalue: false\\n \\ton: [\\n \\t{\\n \\t// Clicking groupMark sets this signal to the filter values\\n \\tevents: @groupMark:click!\\n \\tupdate: \\\"{stack:datum.stack, stk1:datum.stack=='stk1' && datum.grpId, stk2:datum.stack=='stk2' && datum.grpId}\\\"\\n \\t}\\n \\t{\\n \\t// Clicking \\\"show all\\\" button, or double-clicking anywhere resets it\\n \\tevents: [\\n \\t{type: \\\"click\\\", markname: \\\"groupReset\\\"}\\n \\t{type: \\\"dblclick\\\"}\\n \\t]\\n \\tupdate: \\\"false\\\"\\n \\t}\\n \\t]\\n\\t}\\n ]\\n}\\n\"}}"},"id":"58a9509e-e8af-41e1-a2a4-6c8bb54992de","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2024-01-02T19:29:06.462Z","version":"WzM5NCwzXQ=="} +{"exportedCount":1,"missingRefCount":0,"missingReferences":[]} \ No newline at end of file diff --git a/visualizations/observability/http/http-network-graph.png b/visualizations/observability/http/http-network-graph.png new file mode 100644 index 0000000..0294b0d Binary files /dev/null and b/visualizations/observability/http/http-network-graph.png differ diff --git a/visualizations/observability/http/test-http-network-graph-widget.json b/visualizations/observability/http/test-http-network-graph-widget.json new file mode 100644 index 0000000..f1aeba0 --- /dev/null +++ b/visualizations/observability/http/test-http-network-graph-widget.json @@ -0,0 +1,636 @@ +{ + "$schema": "https://vega.github.io/schema/vega/v5.json", + "data": [ + { + "name": "rawData", + "format": { + "property": "aggregations.table.buckets" + }, + "transform": [ + { + "type": "formula", + "expr": "datum.key.stk1", + "as": "stk1" + }, + { + "type": "formula", + "expr": "datum.key.stk2", + "as": "stk2" + }, + { + "type": "formula", + "expr": "datum.doc_count", + "as": "size" + } + ], + "values": { + "took": 1, + "timed_out": false, + "_shards": { + "total": 1, + "successful": 1, + "skipped": 0, + "failed": 0 + }, + "hits": { + "total": 22, + "max_score": null, + "hits": [] + }, + "aggregations": { + "table": { + "after_key": { + "stk1": "162.142.125.177", + "stk2": "10.0.0.200" + }, + "buckets": [ + { + "key": { + "stk1": "10.0.0.200", + "stk2": "162.142.125.177" + }, + "doc_count": 11, + "stk1": "10.0.0.200", + "stk2": "162.142.125.177", + "size": 11 + }, + { + "key": { + "stk1": "162.142.125.177", + "stk2": "10.0.0.200" + }, + "doc_count": 11, + "stk1": "162.142.125.177", + "stk2": "10.0.0.200", + "size": 11 + } + ] + } + } + } + }, + { + "name": "nodes", + "source": "rawData", + "transform": [ + { + "type": "filter", + "expr": "!groupSelector || groupSelector.stk1 == datum.stk1 || groupSelector.stk2 == datum.stk2" + }, + { + "type": "formula", + "expr": "datum.stk1+datum.stk2", + "as": "key" + }, + { + "type": "fold", + "fields": [ + "stk1", + "stk2" + ], + "as": [ + "stack", + "grpId" + ] + }, + { + "type": "formula", + "expr": "datum.stack == 'stk1' ? datum.stk1+datum.stk2 : datum.stk2+datum.stk1", + "as": "sortField" + }, + { + "type": "stack", + "groupby": [ + "stack" + ], + "sort": { + "field": "sortField", + "order": "descending" + }, + "field": "size" + }, + { + "type": "formula", + "expr": "(datum.y0+datum.y1)/2", + "as": "yc" + } + ] + }, + { + "name": "groups", + "source": "nodes", + "transform": [ + { + "type": "aggregate", + "groupby": [ + "stack", + "grpId" + ], + "fields": [ + "size" + ], + "ops": [ + "sum" + ], + "as": [ + "total" + ] + }, + { + "type": "stack", + "groupby": [ + "stack" + ], + "sort": { + "field": "grpId", + "order": "descending" + }, + "field": "total" + }, + { + "type": "formula", + "expr": "scale('y', datum.y0)", + "as": "scaledY0" + }, + { + "type": "formula", + "expr": "scale('y', datum.y1)", + "as": "scaledY1" + }, + { + "type": "formula", + "expr": "datum.stack == 'stk1'", + "as": "rightLabel" + }, + { + "type": "formula", + "expr": "datum.total/domain('y')[1]", + "as": "percentage" + } + ] + }, + { + "name": "destinationNodes", + "source": "nodes", + "transform": [ + { + "type": "filter", + "expr": "datum.stack == 'stk2'" + } + ] + }, + { + "name": "edges", + "source": "nodes", + "transform": [ + { + "type": "filter", + "expr": "datum.stack == 'stk1'" + }, + { + "type": "lookup", + "from": "destinationNodes", + "key": "key", + "fields": [ + "key" + ], + "as": [ + "target" + ] + }, + { + "type": "linkpath", + "orient": "horizontal", + "shape": "diagonal", + "sourceY": { + "expr": "scale('y', datum.yc)" + }, + "sourceX": { + "expr": "scale('x', 'stk1') + bandwidth('x')" + }, + "targetY": { + "expr": "scale('y', datum.target.yc)" + }, + "targetX": { + "expr": "scale('x', 'stk2')" + } + }, + { + "type": "formula", + "expr": "range('y')[0]-scale('y', datum.size)", + "as": "strokeWidth" + }, + { + "type": "formula", + "expr": "datum.size/domain('y')[1]", + "as": "percentage" + } + ] + } + ], + "scales": [ + { + "name": "x", + "type": "band", + "range": "width", + "domain": [ + "stk1", + "stk2" + ], + "paddingOuter": 0.05, + "paddingInner": 0.95 + }, + { + "name": "y", + "type": "linear", + "range": "height", + "domain": { + "data": "nodes", + "field": "y1" + } + }, + { + "name": "color", + "type": "ordinal", + "range": "category", + "domain": { + "data": "rawData", + "field": "stk1" + } + }, + { + "name": "stackNames", + "type": "ordinal", + "range": [ + "Source", + "Destination" + ], + "domain": [ + "stk1", + "stk2" + ] + } + ], + "axes": [ + { + "orient": "bottom", + "scale": "x", + "encode": { + "labels": { + "update": { + "text": { + "scale": "stackNames", + "field": "value" + } + } + } + } + }, + { + "orient": "left", + "scale": "y" + } + ], + "marks": [ + { + "type": "path", + "name": "edgeMark", + "from": { + "data": "edges" + }, + "clip": true, + "encode": { + "update": { + "stroke": [ + { + "test": "groupSelector && groupSelector.stack=='stk1'", + "scale": "color", + "field": "stk2" + }, + { + "scale": "color", + "field": "stk1" + } + ], + "strokeWidth": { + "field": "strokeWidth" + }, + "path": { + "field": "path" + }, + "strokeOpacity": { + "signal": "!groupSelector && (groupHover.stk1 == datum.stk1 || groupHover.stk2 == datum.stk2) ? 0.9 : 0.3" + }, + "zindex": { + "signal": "!groupSelector && (groupHover.stk1 == datum.stk1 || groupHover.stk2 == datum.stk2) ? 1 : 0" + }, + "tooltip": { + "signal": "datum.stk1 + ' → ' + datum.stk2 + '\t' + format(datum.size, ',.0f') + ' (' + format(datum.percentage, '.1%') + ')'" + } + }, + "hover": { + "strokeOpacity": { + "value": 1 + } + } + } + }, + { + "type": "rect", + "name": "groupMark", + "from": { + "data": "groups" + }, + "encode": { + "enter": { + "fill": { + "scale": "color", + "field": "grpId" + }, + "width": { + "scale": "x", + "band": 1 + } + }, + "update": { + "x": { + "scale": "x", + "field": "stack" + }, + "y": { + "field": "scaledY0" + }, + "y2": { + "field": "scaledY1" + }, + "fillOpacity": { + "value": 0.6 + }, + "tooltip": { + "signal": "datum.grpId + ' ' + format(datum.total, ',.0f') + ' (' + format(datum.percentage, '.1%') + ')'" + } + }, + "hover": { + "fillOpacity": { + "value": 1 + } + } + } + }, + { + "type": "text", + "from": { + "data": "groups" + }, + "interactive": false, + "encode": { + "update": { + "x": { + "signal": "scale('x', datum.stack) + (datum.rightLabel ? bandwidth('x') + 8 : -8)" + }, + "yc": { + "signal": "(datum.scaledY0 + datum.scaledY1)/2" + }, + "align": { + "signal": "datum.rightLabel ? 'left' : 'right'" + }, + "baseline": { + "value": "middle" + }, + "fontWeight": { + "value": "bold" + }, + "text": { + "signal": "abs(datum.scaledY0-datum.scaledY1) > 13 ? datum.grpId : ''" + } + } + } + }, + { + "type": "group", + "data": [ + { + "name": "dataForShowAll", + "values": [ + {} + ], + "transform": [ + { + "type": "filter", + "expr": "groupSelector" + } + ] + } + ], + "encode": { + "enter": { + "xc": { + "signal": "width/2" + }, + "y": { + "value": 30 + }, + "width": { + "value": 80 + }, + "height": { + "value": 30 + } + } + }, + "marks": [ + { + "type": "group", + "name": "groupReset", + "from": { + "data": "dataForShowAll" + }, + "encode": { + "enter": { + "cornerRadius": { + "value": 6 + }, + "fill": { + "value": "#F5F7FA" + }, + "stroke": { + "value": "#c1c1c1" + }, + "strokeWidth": { + "value": 2 + }, + "height": { + "field": { + "group": "height" + } + }, + "width": { + "field": { + "group": "width" + } + } + }, + "update": { + "opacity": { + "value": 1 + } + }, + "hover": { + "opacity": { + "value": 0.7 + } + } + }, + "marks": [ + { + "type": "text", + "interactive": false, + "encode": { + "enter": { + "xc": { + "field": { + "group": "width" + }, + "mult": 0.5 + }, + "yc": { + "field": { + "group": "height" + }, + "mult": 0.5, + "offset": 2 + }, + "align": { + "value": "center" + }, + "baseline": { + "value": "middle" + }, + "fontWeight": { + "value": "bold" + }, + "text": { + "value": "Show All" + } + } + } + } + ] + } + ] + } + ], + "signals": [ + { + "name": "groupHover", + "value": {}, + "on": [ + { + "events": "@groupMark:mouseover", + "update": "{stk1:datum.stack=='stk1' && datum.grpId, stk2:datum.stack=='stk2' && datum.grpId}" + }, + { + "events": "mouseout", + "update": "{}" + } + ] + }, + { + "name": "groupSelector", + "value": false, + "on": [ + { + "events": "@groupMark:click!", + "update": "{stack:datum.stack, stk1:datum.stack=='stk1' && datum.grpId, stk2:datum.stack=='stk2' && datum.grpId}" + }, + { + "events": [ + { + "type": "click", + "markname": "groupReset" + }, + { + "type": "dblclick" + } + ], + "update": "false" + } + ] + } + ], + "config": { + "range": { + "category": [ + "#54B399", + "#6092C0", + "#D36086", + "#9170B8", + "#CA8EAE", + "#D6BF57", + "#B9A888", + "#DA8B45", + "#AA6556", + "#E7664C" + ] + }, + "arc": { + "fill": "#54B399" + }, + "area": { + "fill": "#54B399" + }, + "line": { + "stroke": "#54B399" + }, + "path": { + "stroke": "#54B399" + }, + "rect": { + "fill": "#54B399" + }, + "rule": { + "stroke": "#54B399" + }, + "shape": { + "stroke": "#54B399" + }, + "symbol": { + "fill": "#54B399" + }, + "text": { + "fill": "#dfe5ef" + }, + "trail": { + "fill": "#54B399" + }, + "title": { + "color": "#dfe3e8" + }, + "style": { + "guide-label": { + "fill": "#8d98a3" + }, + "guide-title": { + "fill": "#dfe3e8" + }, + "group-title": { + "fill": "#dfe3e8" + }, + "group-subtitle": { + "fill": "#dfe3e8" + } + }, + "axis": { + "tickColor": "#293847", + "domainColor": "#293847", + "gridColor": "#293847" + }, + "background": "transparent" + }, + "width": 1250, + "height": 800, + "autosize": { + "type": "fit", + "contains": "padding" + } +} \ No newline at end of file diff --git a/visualizations/observability/service-duration-heatmap/data/buld_data.json b/visualizations/observability/service-duration-heatmap/data/buld_data.json new file mode 100644 index 0000000..0637a08 --- /dev/null +++ b/visualizations/observability/service-duration-heatmap/data/buld_data.json @@ -0,0 +1 @@ +[] \ No newline at end of file diff --git a/visualizations/observability/service-duration-heatmap/dql-service-by-duration-widget.json b/visualizations/observability/service-duration-heatmap/dql-service-by-duration-widget.json index 6db5d8a..adee5e3 100644 --- a/visualizations/observability/service-duration-heatmap/dql-service-by-duration-widget.json +++ b/visualizations/observability/service-duration-heatmap/dql-service-by-duration-widget.json @@ -71,7 +71,7 @@ }, "transform": [ { - "calculate": "timeFormat(utcParse(datum.key_as_string,'%Y-%m-%dT%H:%M:%S.%LZ'), '%B %d, %Y %H:%M')", + "calculate": "timeFormat(utcParse(datum.key_as_string, '%Y-%m-%dT%H:%M:%S.%LZ'), '%b %d, %H:%M')", "as": "time" }, { @@ -113,7 +113,7 @@ "values": [0, 0, 1000, 5000, 10000, 25000, 5000], "title": [ "Duration SLO Index (DSI)", - "0: No Duration ; 50000: Exceptional Duration" + "0: No Duration ; 5000: Exceptional Duration" ] } }, @@ -127,20 +127,6 @@ {"field": "count"} ] } - }, - { - "data": {"values": [{}]}, - "layer": [ - { - "mark": { - "type": "text", - "text": "RED: Hourly Services Duration Heat Map ", - "font": "Outfit", - "dx": 360, - "dy": 18 - } - } - ] } ], "config": { diff --git a/visualizations/observability/service-duration-heatmap/ppl-service-by-duration-widget.json b/visualizations/observability/service-duration-heatmap/ppl-service-by-duration-widget.json index 0558169..00844bf 100644 --- a/visualizations/observability/service-duration-heatmap/ppl-service-by-duration-widget.json +++ b/visualizations/observability/service-duration-heatmap/ppl-service-by-duration-widget.json @@ -21,7 +21,7 @@ "as": "count" }, { - "calculate": "timeFormat(utcParse(datum.key_as_string,'%Y-%m-%dT%H:%M:%S.%LZ'), '%B %d, %Y %H:%M')", + "calculate": "timeFormat(utcParse(datum.key_as_string, '%Y-%m-%dT%H:%M:%S.%LZ'), '%b %d, %H:%M')", "as": "time" }, { @@ -49,7 +49,7 @@ "values": [0, 0, 1000, 5000, 10000, 25000, 5000], "title": [ "Duration SLO Index (DSI)", - "0: No Duration ; 50000: Exceptional Duration" + "0: No Duration ; 5000: Exceptional Duration" ] } }, @@ -63,20 +63,6 @@ {"field": "count"} ] } - }, - { - "data": {"values": [{}]}, - "layer": [ - { - "mark": { - "type": "text", - "text": "RED: Hourly Services Duration Heat Map ", - "font": "Outfit", - "dx": 360, - "dy": 18 - } - } - ] } ], "config": { diff --git a/visualizations/observability/service-duration-heatmap/service-duration.ndjson b/visualizations/observability/service-duration-heatmap/service-duration.ndjson new file mode 100644 index 0000000..6362f0a --- /dev/null +++ b/visualizations/observability/service-duration-heatmap/service-duration.ndjson @@ -0,0 +1,2 @@ +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Service Duration SLO heatmap","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Service Duration SLO heatmap\",\"type\":\"vega\",\"aggs\":[],\"params\":{\"spec\":\"{\\n \\\"$schema\\\": \\\"https://vega.github.io/schema/vega-lite/v5.json\\\",\\n \\\"vconcat\\\": [\\n {\\n \\\"title\\\": {\\n \\\"text\\\": \\\"SLO Services Durations\\\",\\n \\\"subtitle\\\": \\\"Hourly Service Duration SLO heatmap\\\",\\n \\\"color\\\": \\\"black\\\"\\n },\\n \\\"width\\\": 1000,\\n \\\"height\\\": {\\\"step\\\": 29},\\n \\\"data\\\": {\\n \\\"name\\\": \\\"data-raw\\\",\\n \\\"url\\\": {\\n \\\"%context%\\\": \\\"true\\\",\\n \\\"%timefield%\\\": \\\"startTime\\\",\\n \\\"index\\\": \\\"otel-v1-apm-span-*\\\",\\n \\\"body\\\": {\\n \\\"size\\\": 0,\\n \\\"aggs\\\": {\\n \\\"time_buckets\\\": {\\n \\\"date_histogram\\\": {\\n \\\"field\\\": \\\"startTime\\\",\\n \\\"interval\\\": {\\n \\\"%autointerval%\\\": true\\n },\\n \\\"extended_bounds\\\": {\\n \\\"min\\\": {\\n \\\"%timefilter%\\\": \\\"min\\\"\\n },\\n \\\"max\\\": {\\n \\\"%timefilter%\\\": \\\"max\\\"\\n }\\n },\\n \\\"min_doc_count\\\": 0\\n },\\n \\\"aggs\\\": {\\n \\\"time_buckets\\\": {\\n \\\"date_histogram\\\": {\\n \\\"field\\\": \\\"startTime\\\",\\n \\\"calendar_interval\\\": \\\"1h\\\"\\n },\\n \\\"aggs\\\": {\\n \\\"service_names\\\": {\\n \\\"terms\\\": {\\n \\\"field\\\": \\\"serviceName\\\",\\n \\\"order\\\": {\\n \\\"_count\\\": \\\"desc\\\"\\n }\\n },\\n \\\"aggs\\\": {\\n \\\"avg_duration\\\": {\\n \\\"avg\\\": {\\n \\\"field\\\": \\\"durationInNanos\\\"\\n }\\n }\\n }\\n }\\n }\\n }\\n }\\n }\\n }\\n }\\n },\\n \\\"format\\\": {\\n \\\"type\\\": \\\"json\\\",\\n \\\"property\\\": \\\"aggregations.time_buckets.buckets\\\"\\n }\\n },\\n \\\"transform\\\": [\\n {\\n \\\"calculate\\\": \\\"timeFormat(utcParse(datum.key_as_string,'%Y-%m-%dT%H:%M:%S.%LZ'), '%B %d, %Y %H:%M')\\\",\\n \\\"as\\\": \\\"time\\\"\\n },\\n {\\n \\\"calculate\\\": \\\"datum.time_buckets.buckets\\\", \\\"as\\\": \\\"servicesData\\\"\\n },\\n {\\n \\\"flatten\\\": [\\\"servicesData\\\"], \\\"as\\\": [\\\"flatten\\\"]\\n },\\n {\\n \\\"flatten\\\": [\\\"flatten.service_names.buckets\\\"], \\\"as\\\": [\\\"serviceInfo\\\"]\\n },\\n {\\n \\\"calculate\\\": \\\"datum.serviceInfo.key\\\",\\n \\\"as\\\": \\\"service\\\"\\n },\\n {\\n \\\"calculate\\\": \\\"datum.serviceInfo.avg_duration.value\\\",\\n \\\"as\\\": \\\"count\\\"\\n }\\n ],\\n \\\"params\\\": [\\n {\\\"name\\\": \\\"highlight\\\", \\\"select\\\": {\\\"type\\\": \\\"point\\\", \\\"on\\\": \\\"mouseover\\\"}}\\n ],\\n \\\"mark\\\": {\\\"type\\\": \\\"rect\\\"},\\n \\\"encoding\\\": {\\n \\\"x\\\": {\\n \\\"field\\\": \\\"time\\\",\\n \\\"type\\\": \\\"ordinal\\\"\\n },\\n \\\"y\\\": {\\\"field\\\": \\\"service\\\", \\\"type\\\": \\\"nominal\\\"},\\n \\\"fill\\\": {\\n \\\"field\\\": \\\"count\\\",\\n \\\"type\\\": \\\"quantitative\\\",\\n \\\"scale\\\": {\\n \\\"range\\\": [\\\"#FCF7E1\\\", \\\"#FCEFD5\\\", \\\"#FACC87\\\", \\\"#FA8D53\\\", \\\"#DF4B2A\\\"]\\n },\\n \\\"legend\\\": {\\n \\\"description\\\": \\\"Vega-Lite doesn't show the first value of the legend's gradient so I used zero twice in the values array\\\",\\n \\\"values\\\": [0, 0, 1000, 5000, 10000, 25000, 5000],\\n \\\"color\\\": \\\"black\\\",\\n \\\"title\\\": [\\n \\\"Duration SLO Index (DSI)\\\",\\n \\\"0: No Duration ; 50000: Exceptional Duration\\\"\\n ]\\n }\\n },\\n \\\"strokeWidth\\\": {\\n \\\"condition\\\": [{\\\"param\\\": \\\"highlight\\\", \\\"empty\\\": false, \\\"value\\\": 1}],\\n \\\"value\\\": 0\\n },\\n \\\"tooltip\\\": [\\n {\\\"field\\\": \\\"service\\\"},\\n {\\\"field\\\": \\\"time\\\", \\\"title\\\": \\\"Date\\\"},\\n {\\\"field\\\": \\\"count\\\"}\\n ]\\n }\\n },\\n {\\n \\\"data\\\": {\\\"values\\\": [{}]},\\n \\\"layer\\\": [\\n {\\n \\\"mark\\\": {\\n \\\"type\\\": \\\"text\\\",\\n \\\"text\\\": \\\"RED: Hourly Services Duration Heat Map \\\",\\n \\\"font\\\": \\\"Outfit\\\",\\n\\n \\\"dx\\\": 360,\\n \\\"dy\\\": 18\\n }\\n }\\n ]\\n }\\n ],\\n \\\"config\\\": {\\n \\\"background\\\": \\\"#f4f7ff\\\",\\n \\\"padding\\\": {\\\"top\\\": 10, \\\"bottom\\\": 10, \\\"left\\\": 30, \\\"right\\\": 30},\\n \\\"view\\\": {\\\"stroke\\\": null},\\n \\\"font\\\": \\\"Test Karbon\\\",\\n \\\"axis\\\": {\\n \\\"grid\\\": false,\\n \\\"labelColor\\\": \\\"black\\\",\\n \\\"labelFontSize\\\": 17,\\n \\\"title\\\": null\\n },\\n \\\"axisX\\\": {\\n \\\"orient\\\": \\\"bottom\\\",\\n \\\"domainColor\\\": \\\"black\\\",\\n \\\"offset\\\": 5,\\n \\\"tickCount\\\": 10,\\n \\\"tickOffset\\\": 5,\\n \\\"tickSize\\\": 8,\\n \\\"tickColor\\\": \\\"black\\\",\\n \\\"labelPadding\\\": 5,\\n \\\"labelFlush\\\": false\\n },\\n \\\"axisY\\\": {\\n \\\"domain\\\": false,\\n \\\"ticks\\\": false,\\n \\\"labelAlign\\\": \\\"left\\\",\\n \\\"labelPadding\\\": 140\\n },\\n \\\"rect\\\": {\\\"width\\\": 14.5, \\\"height\\\": 20, \\\"stroke\\\": \\\"black\\\"},\\n \\\"text\\\": {\\\"fontSize\\\": 13},\\n \\\"legend\\\": {\\n \\\"orient\\\": \\\"none\\\",\\n \\\"direction\\\": \\\"horizontal\\\",\\n \\\"gradientLength\\\": 300,\\n \\\"gradientThickness\\\": 10,\\n \\\"gradientLabelOffset\\\": 5,\\n \\\"labelFont\\\": \\\"Outfit\\\",\\n \\\"labelFontSize\\\": 13,\\n \\\"titlePadding\\\": 5,\\n \\\"titleFont\\\": \\\"Outfit\\\",\\n \\\"titleFontSize\\\": 14,\\n \\\"titleFontWeight\\\": \\\"normal\\\",\\n \\\"titleLineHeight\\\": 20,\\n \\\"titleLimit\\\": 300,\\n \\\"legendX\\\": 703,\\n \\\"legendY\\\": -140\\n },\\n \\\"title\\\": {\\n \\\"anchor\\\": \\\"start\\\",\\n \\\"offset\\\": 40,\\n \\\"fontSize\\\": 35,\\n \\\"subtitlePadding\\\": 10,\\n \\\"subtitleFontSize\\\": 25\\n },\\n \\\"concat\\\": {\\\"spacing\\\": 0}\\n }\\n}\"}}"},"id":"aa213fb0-b32d-11ee-8a87-757393a1a8e1","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2024-01-14T22:38:40.427Z","version":"WzYwNCwzXQ=="} +{"exportedCount":1,"missingRefCount":0,"missingReferences":[]} \ No newline at end of file diff --git a/visualizations/observability/service-duration-heatmap/test-service-by-duration-widget.json b/visualizations/observability/service-duration-heatmap/test-service-by-duration-widget.json index 8397feb..85afc6f 100644 --- a/visualizations/observability/service-duration-heatmap/test-service-by-duration-widget.json +++ b/visualizations/observability/service-duration-heatmap/test-service-by-duration-widget.json @@ -1,6 +1,6 @@ { "$schema": "https://vega.github.io/schema/vega-lite/v5.json", - "vconcat": [ + "vconcat": [ { "title": { "text": "SLO Services Durations", @@ -10,680 +10,527 @@ "width": 300, "height": {"step": 29}, "data": { - "name": "data-raw", - "values": { - "aggregations": { - "time_buckets": { - "buckets": [ - { - "key_as_string": "2024-01-13T07:53:30.000Z", - "key": 1705132410000, - "doc_count": 431, - "time_buckets": { - "buckets": [ - { - "key_as_string": "2024-01-13T07:00:00.000Z", - "key": 1705129200000, - "doc_count": 431, - "service_names": { - "doc_count_error_upper_bound": 0, - "sum_other_doc_count": 33, - "buckets": [ - { - "key": "frontend", - "doc_count": 151, - "avg_duration": { - "value": 5771611.549668875 - } - }, - { - "key": "checkoutservice", - "doc_count": 62, - "avg_duration": { - "value": 5149686.338709678 - } - }, - { - "key": "loadgenerator", - "doc_count": 40, - "avg_duration": { - "value": 11035119 - } - }, - { - "key": "productcatalogservice", - "doc_count": 38, - "avg_duration": { - "value": 711983.447368421 - } - }, - { - "key": "cartservice", - "doc_count": 36, - "avg_duration": { - "value": 1029000 - } - }, - { - "key": "emailservice", - "doc_count": 20, - "avg_duration": { - "value": 1094679.85 - } - }, - { - "key": "quoteservice", - "doc_count": 15, - "avg_duration": { - "value": 430928 - } - }, - { - "key": "shippingservice", - "doc_count": 15, - "avg_duration": { - "value": 1487472.1333333333 - } - }, - { - "key": "currencyservice", - "doc_count": 11, - "avg_duration": { - "value": 38003.454545454544 - } - }, - { - "key": "adservice", - "doc_count": 10, - "avg_duration": { - "value": 73330 - } - } - ] - } + "name": "data-raw", + "values": { + "aggregations": { + "time_buckets": { + "buckets": [ + { + "key_as_string": "2024-01-13T07:53:30.000Z", + "key": 1705132410000, + "doc_count": 431, + "time_buckets": { + "buckets": [ + { + "key_as_string": "2024-01-13T07:00:00.000Z", + "key": 1705129200000, + "doc_count": 431, + "service_names": { + "doc_count_error_upper_bound": 0, + "sum_other_doc_count": 33, + "buckets": [ + { + "key": "frontend", + "doc_count": 151, + "avg_duration": {"value": 5771611.549668875} + }, + { + "key": "checkoutservice", + "doc_count": 62, + "avg_duration": {"value": 5149686.338709678} + }, + { + "key": "loadgenerator", + "doc_count": 40, + "avg_duration": {"value": 11035119} + }, + { + "key": "productcatalogservice", + "doc_count": 38, + "avg_duration": {"value": 711983.447368421} + }, + { + "key": "cartservice", + "doc_count": 36, + "avg_duration": {"value": 1029000} + }, + { + "key": "emailservice", + "doc_count": 20, + "avg_duration": {"value": 1094679.85} + }, + { + "key": "quoteservice", + "doc_count": 15, + "avg_duration": {"value": 430928} + }, + { + "key": "shippingservice", + "doc_count": 15, + "avg_duration": {"value": 1487472.1333333333} + }, + { + "key": "currencyservice", + "doc_count": 11, + "avg_duration": {"value": 38003.454545454544} + }, + { + "key": "adservice", + "doc_count": 10, + "avg_duration": {"value": 73330} + } + ] } - ] - } - }, - { - "key_as_string": "2024-01-13T07:54:00.000Z", - "key": 1705132440000, - "doc_count": 629, - "time_buckets": { - "buckets": [ - { - "key_as_string": "2024-01-13T07:00:00.000Z", - "key": 1705129200000, - "doc_count": 629, - "service_names": { - "doc_count_error_upper_bound": 0, - "sum_other_doc_count": 27, - "buckets": [ - { - "key": "frontend", - "doc_count": 255, - "avg_duration": { - "value": 5246010.227450981 - } - }, - { - "key": "productcatalogservice", - "doc_count": 94, - "avg_duration": { - "value": 1379269.3617021276 - } - }, - { - "key": "loadgenerator", - "doc_count": 69, - "avg_duration": { - "value": 10756970.739130436 - } - }, - { - "key": "checkoutservice", - "doc_count": 48, - "avg_duration": { - "value": 5243597 - } - }, - { - "key": "featureflagservice", - "doc_count": 42, - "avg_duration": { - "value": 784000.880952381 - } - }, - { - "key": "cartservice", - "doc_count": 30, - "avg_duration": { - "value": 1001956.6666666666 - } - }, - { - "key": "recommendationservice", - "doc_count": 24, - "avg_duration": { - "value": 8178396.666666667 - } - }, - { - "key": "adservice", - "doc_count": 16, - "avg_duration": { - "value": 1561490.125 - } - }, - { - "key": "currencyservice", - "doc_count": 12, - "avg_duration": { - "value": 32263.416666666668 - } - }, - { - "key": "emailservice", - "doc_count": 12, - "avg_duration": { - "value": 1113677.1666666667 - } - } - ] - } + } + ] + } + }, + { + "key_as_string": "2024-01-13T07:54:00.000Z", + "key": 1705132440000, + "doc_count": 629, + "time_buckets": { + "buckets": [ + { + "key_as_string": "2024-01-13T07:00:00.000Z", + "key": 1705129200000, + "doc_count": 629, + "service_names": { + "doc_count_error_upper_bound": 0, + "sum_other_doc_count": 27, + "buckets": [ + { + "key": "frontend", + "doc_count": 255, + "avg_duration": {"value": 5246010.227450981} + }, + { + "key": "productcatalogservice", + "doc_count": 94, + "avg_duration": {"value": 1379269.3617021276} + }, + { + "key": "loadgenerator", + "doc_count": 69, + "avg_duration": {"value": 10756970.739130436} + }, + { + "key": "checkoutservice", + "doc_count": 48, + "avg_duration": {"value": 5243597} + }, + { + "key": "featureflagservice", + "doc_count": 42, + "avg_duration": {"value": 784000.880952381} + }, + { + "key": "cartservice", + "doc_count": 30, + "avg_duration": {"value": 1001956.6666666666} + }, + { + "key": "recommendationservice", + "doc_count": 24, + "avg_duration": {"value": 8178396.666666667} + }, + { + "key": "adservice", + "doc_count": 16, + "avg_duration": {"value": 1561490.125} + }, + { + "key": "currencyservice", + "doc_count": 12, + "avg_duration": {"value": 32263.416666666668} + }, + { + "key": "emailservice", + "doc_count": 12, + "avg_duration": {"value": 1113677.1666666667} + } + ] } - ] - } - }, - { - "key_as_string": "2024-01-13T07:54:30.000Z", - "key": 1705132470000, - "doc_count": 714, - "time_buckets": { - "buckets": [ - { - "key_as_string": "2024-01-13T07:00:00.000Z", - "key": 1705129200000, - "doc_count": 714, - "service_names": { - "doc_count_error_upper_bound": 0, - "sum_other_doc_count": 38, - "buckets": [ - { - "key": "frontend", - "doc_count": 285, - "avg_duration": { - "value": 5548306.863157894 - } - }, - { - "key": "productcatalogservice", - "doc_count": 96, - "avg_duration": { - "value": 1624046.1979166667 - } - }, - { - "key": "loadgenerator", - "doc_count": 80, - "avg_duration": { - "value": 10745925.05 - } - }, - { - "key": "checkoutservice", - "doc_count": 66, - "avg_duration": { - "value": 5425193.863636363 - } - }, - { - "key": "cartservice", - "doc_count": 41, - "avg_duration": { - "value": 1040795.1219512195 - } - }, - { - "key": "featureflagservice", - "doc_count": 38, - "avg_duration": { - "value": 809455.6578947369 - } - }, - { - "key": "emailservice", - "doc_count": 20, - "avg_duration": { - "value": 1099261.7 - } - }, - { - "key": "recommendationservice", - "doc_count": 20, - "avg_duration": { - "value": 8174893.9 - } - }, - { - "key": "quoteservice", - "doc_count": 15, - "avg_duration": { - "value": 508928.2 - } - }, - { - "key": "shippingservice", - "doc_count": 15, - "avg_duration": { - "value": 1640500 - } - } - ] - } + } + ] + } + }, + { + "key_as_string": "2024-01-13T07:54:30.000Z", + "key": 1705132470000, + "doc_count": 714, + "time_buckets": { + "buckets": [ + { + "key_as_string": "2024-01-13T07:00:00.000Z", + "key": 1705129200000, + "doc_count": 714, + "service_names": { + "doc_count_error_upper_bound": 0, + "sum_other_doc_count": 38, + "buckets": [ + { + "key": "frontend", + "doc_count": 285, + "avg_duration": {"value": 5548306.863157894} + }, + { + "key": "productcatalogservice", + "doc_count": 96, + "avg_duration": {"value": 1624046.1979166667} + }, + { + "key": "loadgenerator", + "doc_count": 80, + "avg_duration": {"value": 10745925.05} + }, + { + "key": "checkoutservice", + "doc_count": 66, + "avg_duration": {"value": 5425193.863636363} + }, + { + "key": "cartservice", + "doc_count": 41, + "avg_duration": {"value": 1040795.1219512195} + }, + { + "key": "featureflagservice", + "doc_count": 38, + "avg_duration": {"value": 809455.6578947369} + }, + { + "key": "emailservice", + "doc_count": 20, + "avg_duration": {"value": 1099261.7} + }, + { + "key": "recommendationservice", + "doc_count": 20, + "avg_duration": {"value": 8174893.9} + }, + { + "key": "quoteservice", + "doc_count": 15, + "avg_duration": {"value": 508928.2} + }, + { + "key": "shippingservice", + "doc_count": 15, + "avg_duration": {"value": 1640500} + } + ] } - ] - } - }, - { - "key_as_string": "2024-01-13T07:55:00.000Z", - "key": 1705132500000, - "doc_count": 646, - "time_buckets": { - "buckets": [ - { - "key_as_string": "2024-01-13T07:00:00.000Z", - "key": 1705129200000, - "doc_count": 646, - "service_names": { - "doc_count_error_upper_bound": 0, - "sum_other_doc_count": 36, - "buckets": [ - { - "key": "frontend", - "doc_count": 262, - "avg_duration": { - "value": 5117881.648854962 - } - }, - { - "key": "productcatalogservice", - "doc_count": 88, - "avg_duration": { - "value": 1118303.8181818181 - } - }, - { - "key": "loadgenerator", - "doc_count": 69, - "avg_duration": { - "value": 10462353.956521738 - } - }, - { - "key": "checkoutservice", - "doc_count": 56, - "avg_duration": { - "value": 5425841.25 - } - }, - { - "key": "cartservice", - "doc_count": 38, - "avg_duration": { - "value": 1069986.8421052631 - } - }, - { - "key": "featureflagservice", - "doc_count": 32, - "avg_duration": { - "value": 818774.40625 - } - }, - { - "key": "recommendationservice", - "doc_count": 24, - "avg_duration": { - "value": 8231048.458333333 - } - }, - { - "key": "emailservice", - "doc_count": 16, - "avg_duration": { - "value": 1103416.125 - } - }, - { - "key": "adservice", - "doc_count": 13, - "avg_duration": { - "value": 74293 - } - }, - { - "key": "currencyservice", - "doc_count": 12, - "avg_duration": { - "value": 35144.333333333336 - } - } - ] - } + } + ] + } + }, + { + "key_as_string": "2024-01-13T07:55:00.000Z", + "key": 1705132500000, + "doc_count": 646, + "time_buckets": { + "buckets": [ + { + "key_as_string": "2024-01-13T07:00:00.000Z", + "key": 1705129200000, + "doc_count": 646, + "service_names": { + "doc_count_error_upper_bound": 0, + "sum_other_doc_count": 36, + "buckets": [ + { + "key": "frontend", + "doc_count": 262, + "avg_duration": {"value": 5117881.648854962} + }, + { + "key": "productcatalogservice", + "doc_count": 88, + "avg_duration": {"value": 1118303.8181818181} + }, + { + "key": "loadgenerator", + "doc_count": 69, + "avg_duration": {"value": 10462353.956521738} + }, + { + "key": "checkoutservice", + "doc_count": 56, + "avg_duration": {"value": 5425841.25} + }, + { + "key": "cartservice", + "doc_count": 38, + "avg_duration": {"value": 1069986.8421052631} + }, + { + "key": "featureflagservice", + "doc_count": 32, + "avg_duration": {"value": 818774.40625} + }, + { + "key": "recommendationservice", + "doc_count": 24, + "avg_duration": {"value": 8231048.458333333} + }, + { + "key": "emailservice", + "doc_count": 16, + "avg_duration": {"value": 1103416.125} + }, + { + "key": "adservice", + "doc_count": 13, + "avg_duration": {"value": 74293} + }, + { + "key": "currencyservice", + "doc_count": 12, + "avg_duration": {"value": 35144.333333333336} + } + ] } - ] - } - }, - { - "key_as_string": "2024-01-13T07:55:30.000Z", - "key": 1705132530000, - "doc_count": 1008, - "time_buckets": { - "buckets": [ - { - "key_as_string": "2024-01-13T07:00:00.000Z", - "key": 1705129200000, - "doc_count": 1008, - "service_names": { - "doc_count_error_upper_bound": 0, - "sum_other_doc_count": 57, - "buckets": [ - { - "key": "frontend", - "doc_count": 403, - "avg_duration": { - "value": 4881138.660049628 - } - }, - { - "key": "checkoutservice", - "doc_count": 114, - "avg_duration": { - "value": 5139593.657894737 - } - }, - { - "key": "loadgenerator", - "doc_count": 113, - "avg_duration": { - "value": 9552697.654867256 - } - }, - { - "key": "productcatalogservice", - "doc_count": 109, - "avg_duration": { - "value": 546134.5596330275 - } - }, - { - "key": "cartservice", - "doc_count": 81, - "avg_duration": { - "value": 1070082.7160493827 - } - }, - { - "key": "emailservice", - "doc_count": 32, - "avg_duration": { - "value": 1445610.875 - } - }, - { - "key": "featureflagservice", - "doc_count": 26, - "avg_duration": { - "value": 817655.3846153846 - } - }, - { - "key": "currencyservice", - "doc_count": 25, - "avg_duration": { - "value": 34825 - } - }, - { - "key": "quoteservice", - "doc_count": 24, - "avg_duration": { - "value": 481991.7916666667 - } - }, - { - "key": "shippingservice", - "doc_count": 24, - "avg_duration": { - "value": 1557881.25 - } - } - ] - } + } + ] + } + }, + { + "key_as_string": "2024-01-13T07:55:30.000Z", + "key": 1705132530000, + "doc_count": 1008, + "time_buckets": { + "buckets": [ + { + "key_as_string": "2024-01-13T07:00:00.000Z", + "key": 1705129200000, + "doc_count": 1008, + "service_names": { + "doc_count_error_upper_bound": 0, + "sum_other_doc_count": 57, + "buckets": [ + { + "key": "frontend", + "doc_count": 403, + "avg_duration": {"value": 4881138.660049628} + }, + { + "key": "checkoutservice", + "doc_count": 114, + "avg_duration": {"value": 5139593.657894737} + }, + { + "key": "loadgenerator", + "doc_count": 113, + "avg_duration": {"value": 9552697.654867256} + }, + { + "key": "productcatalogservice", + "doc_count": 109, + "avg_duration": {"value": 546134.5596330275} + }, + { + "key": "cartservice", + "doc_count": 81, + "avg_duration": {"value": 1070082.7160493827} + }, + { + "key": "emailservice", + "doc_count": 32, + "avg_duration": {"value": 1445610.875} + }, + { + "key": "featureflagservice", + "doc_count": 26, + "avg_duration": {"value": 817655.3846153846} + }, + { + "key": "currencyservice", + "doc_count": 25, + "avg_duration": {"value": 34825} + }, + { + "key": "quoteservice", + "doc_count": 24, + "avg_duration": {"value": 481991.7916666667} + }, + { + "key": "shippingservice", + "doc_count": 24, + "avg_duration": {"value": 1557881.25} + } + ] } - ] - } - }, - { - "key_as_string": "2024-01-13T07:56:00.000Z", - "key": 1705132560000, - "doc_count": 837, - "time_buckets": { - "buckets": [ - { - "key_as_string": "2024-01-13T07:00:00.000Z", - "key": 1705129200000, - "doc_count": 837, - "service_names": { - "doc_count_error_upper_bound": 0, - "sum_other_doc_count": 45, - "buckets": [ - { - "key": "frontend", - "doc_count": 337, - "avg_duration": { - "value": 5253962.445103858 - } - }, - { - "key": "productcatalogservice", - "doc_count": 109, - "avg_duration": { - "value": 1624922.6788990826 - } - }, - { - "key": "loadgenerator", - "doc_count": 90, - "avg_duration": { - "value": 10552859.444444444 - } - }, - { - "key": "checkoutservice", - "doc_count": 76, - "avg_duration": { - "value": 5623715.815789473 - } - }, - { - "key": "cartservice", - "doc_count": 56, - "avg_duration": { - "value": 1039839.2857142857 - } - }, - { - "key": "featureflagservice", - "doc_count": 42, - "avg_duration": { - "value": 830033.8333333334 - } - }, - { - "key": "recommendationservice", - "doc_count": 24, - "avg_duration": { - "value": 8258135.083333333 - } - }, - { - "key": "adservice", - "doc_count": 20, - "avg_duration": { - "value": 75193.6 - } - }, - { - "key": "emailservice", - "doc_count": 20, - "avg_duration": { - "value": 1366047.1 - } - }, - { - "key": "currencyservice", - "doc_count": 18, - "avg_duration": { - "value": 35469.5 - } - } - ] - } + } + ] + } + }, + { + "key_as_string": "2024-01-13T07:56:00.000Z", + "key": 1705132560000, + "doc_count": 837, + "time_buckets": { + "buckets": [ + { + "key_as_string": "2024-01-13T07:00:00.000Z", + "key": 1705129200000, + "doc_count": 837, + "service_names": { + "doc_count_error_upper_bound": 0, + "sum_other_doc_count": 45, + "buckets": [ + { + "key": "frontend", + "doc_count": 337, + "avg_duration": {"value": 5253962.445103858} + }, + { + "key": "productcatalogservice", + "doc_count": 109, + "avg_duration": {"value": 1624922.6788990826} + }, + { + "key": "loadgenerator", + "doc_count": 90, + "avg_duration": {"value": 10552859.444444444} + }, + { + "key": "checkoutservice", + "doc_count": 76, + "avg_duration": {"value": 5623715.815789473} + }, + { + "key": "cartservice", + "doc_count": 56, + "avg_duration": {"value": 1039839.2857142857} + }, + { + "key": "featureflagservice", + "doc_count": 42, + "avg_duration": {"value": 830033.8333333334} + }, + { + "key": "recommendationservice", + "doc_count": 24, + "avg_duration": {"value": 8258135.083333333} + }, + { + "key": "adservice", + "doc_count": 20, + "avg_duration": {"value": 75193.6} + }, + { + "key": "emailservice", + "doc_count": 20, + "avg_duration": {"value": 1366047.1} + }, + { + "key": "currencyservice", + "doc_count": 18, + "avg_duration": {"value": 35469.5} + } + ] } - ] - } - }, - { - "key_as_string": "2024-01-13T07:56:30.000Z", - "key": 1705132590000, - "doc_count": 729, - "time_buckets": { - "buckets": [ - { - "key_as_string": "2024-01-13T07:00:00.000Z", - "key": 1705129200000, - "doc_count": 729, - "service_names": { - "doc_count_error_upper_bound": 0, - "sum_other_doc_count": 44, - "buckets": [ - { - "key": "frontend", - "doc_count": 289, - "avg_duration": { - "value": 5431514.795847751 - } - }, - { - "key": "productcatalogservice", - "doc_count": 95, - "avg_duration": { - "value": 1337376.1684210526 - } - }, - { - "key": "loadgenerator", - "doc_count": 77, - "avg_duration": { - "value": 11342439.57142857 - } - }, - { - "key": "checkoutservice", - "doc_count": 68, - "avg_duration": { - "value": 5854207.764705882 - } - }, - { - "key": "cartservice", - "doc_count": 41, - "avg_duration": { - "value": 1071419.512195122 - } - }, - { - "key": "featureflagservice", - "doc_count": 36, - "avg_duration": { - "value": 835351.8888888889 - } - }, - { - "key": "recommendationservice", - "doc_count": 24, - "avg_duration": { - "value": 7982432.625 - } - }, - { - "key": "adservice", - "doc_count": 20, - "avg_duration": { - "value": 76994.4 - } - }, - { - "key": "emailservice", - "doc_count": 20, - "avg_duration": { - "value": 1325168.15 - } - }, - { - "key": "quoteservice", - "doc_count": 15, - "avg_duration": { - "value": 559421.4666666667 - } - } - ] - } + } + ] + } + }, + { + "key_as_string": "2024-01-13T07:56:30.000Z", + "key": 1705132590000, + "doc_count": 729, + "time_buckets": { + "buckets": [ + { + "key_as_string": "2024-01-13T07:00:00.000Z", + "key": 1705129200000, + "doc_count": 729, + "service_names": { + "doc_count_error_upper_bound": 0, + "sum_other_doc_count": 44, + "buckets": [ + { + "key": "frontend", + "doc_count": 289, + "avg_duration": {"value": 5431514.795847751} + }, + { + "key": "productcatalogservice", + "doc_count": 95, + "avg_duration": {"value": 1337376.1684210526} + }, + { + "key": "loadgenerator", + "doc_count": 77, + "avg_duration": {"value": 11342439.57142857} + }, + { + "key": "checkoutservice", + "doc_count": 68, + "avg_duration": {"value": 5854207.764705882} + }, + { + "key": "cartservice", + "doc_count": 41, + "avg_duration": {"value": 1071419.512195122} + }, + { + "key": "featureflagservice", + "doc_count": 36, + "avg_duration": {"value": 835351.8888888889} + }, + { + "key": "recommendationservice", + "doc_count": 24, + "avg_duration": {"value": 7982432.625} + }, + { + "key": "adservice", + "doc_count": 20, + "avg_duration": {"value": 76994.4} + }, + { + "key": "emailservice", + "doc_count": 20, + "avg_duration": {"value": 1325168.15} + }, + { + "key": "quoteservice", + "doc_count": 15, + "avg_duration": {"value": 559421.4666666667} + } + ] } - ] - } - }] - }} - }, - "format": { - "type": "json", - "property": "aggregations.time_buckets.buckets" + } + ] + } + } + ] + } } }, + "format": { + "type": "json", + "property": "aggregations.time_buckets.buckets" + } + }, "transform": [ { - "calculate": "timeFormat(utcParse(datum.key_as_string,'%Y-%m-%dT%H:%M:%S.%LZ'), '%B %d, %Y %H:%M')", + "calculate": "timeFormat(utcParse(datum.key_as_string, '%Y-%m-%dT%H:%M:%S.%LZ'), '%b %d, %H:%M')", "as": "time" }, - { - "calculate": "datum.time_buckets.buckets", "as": "servicesData" - }, - { - "flatten": ["servicesData"], "as": ["flatten"] - }, - { - "flatten": ["flatten.service_names.buckets"], "as": ["serviceInfo"] - }, - { - "calculate": "datum.serviceInfo.key", - "as": "service" - }, - { - "calculate": "datum.serviceInfo.avg_duration.value", - "as": "count" - } + {"calculate": "datum.time_buckets.buckets", "as": "servicesData"}, + {"flatten": ["servicesData"], "as": ["flatten"]}, + {"flatten": ["flatten.service_names.buckets"], "as": ["serviceInfo"]}, + {"calculate": "datum.serviceInfo.key", "as": "service"}, + {"calculate": "datum.serviceInfo.avg_duration.value", "as": "count"} ], "params": [ {"name": "highlight", "select": {"type": "point", "on": "mouseover"}} ], "mark": {"type": "rect"}, "encoding": { - "x": { - "field": "time", - "type": "ordinal" - }, + "x": {"field": "time", "type": "ordinal"}, "y": {"field": "service", "type": "nominal"}, "fill": { "field": "count", @@ -696,7 +543,7 @@ "values": [0, 0, 1000, 5000, 10000, 25000, 5000], "title": [ "Duration SLO Index (DSI)", - "0: No Duration ; 50000: Exceptional Duration" + "0: No Duration ; 5000: Exceptional Duration" ] } }, @@ -710,20 +557,6 @@ {"field": "count"} ] } - }, - { - "data": {"values": [{}]}, - "layer": [ - { - "mark": { - "type": "text", - "text": "RED: Hourly Services Duration Heat Map ", - "font": "Outfit", - "dx": 360, - "dy": 18 - } - } - ] } ], "config": { diff --git a/visualizations/vega-visualizations.md b/visualizations/vega-visualizations.md index 1ef58fc..c254920 100644 --- a/visualizations/vega-visualizations.md +++ b/visualizations/vega-visualizations.md @@ -106,5 +106,6 @@ creating a Vega visualization. #### Additional Resources - Vega Docs: https://vega.github.io/vega/docs/ +- Vega Tutorial: https://vda-lab.github.io/2019/12/vega - OpenSearch Docs: https://opensearch.org/docs/