App Analytics dashboards

[anirudha]

Overview

Application Analytics allows OpenSearch Dashboards users to select logs, traces, and metrics to be part of an application that can be monitored for overall health and visualized on one page. It also allows users to quickly pivot between traces, metrics, and logs to dig into the source of any issues that arise.

Problem

Currently, users have no way of identifying which trace and log data belongs to which application because all trace data ends up in the same index and log data goes into different independent indices. This application is made up of user-selected logs, metrics, and traces. The current solution forces users to sift through large volumes of data to get to what they really need. Also this data, distributed across Trace Analytics, Event Analytics, Dashboards, does not provide a comprehensive view of an application’s health at a glance.

Requirements

Home Page

1. Users are able to view existing applications and some high level information (Name, Composition, Current Availability, Availability Metrics)
2. Users are able to create/rename/duplicate/delete applications

Create Page

Application Information

1. Users are able to specify name of new application
2. Users are able to optionally provide a description of their application

Composition

1. Users are able to configure base query
2. Users are able to select services and entities to include in the application
3. Users are able to constrain application to specific trace groups

Availability

1. Users are able to set custom levels of availability according to query conditions

Application Details

Overview Tab

1. Users are able to view Latency by Trace Group
2. Users are able to view default charts: Application Composition Map, Error Rate, Throughput
3. Users are able to add new charts

Services Tab

1. Users are able to view information about specific or all services in a table

Traces & Spans Tab

1. Users are able to view general information about traces and spans in two tables
2. Users are able to view more detailed information by selecting a trace or span

Log Events

1. Users are able to query and analyze logs for their application
2. Users are able to create visualizations that are saved as metrics

Panels Metrics

1. Users are able to view metrics (visualizations) created from the log events tab

Configuration

1. Users are able to edit their application composition and availability settings

UX/UI Design

Figma Link
Home Page

Overview Tab

Services Tab

Configuration Tab

Development Tasks

Home Page

• Add table to view existing applications
• • Fetch all available applications #376
• • Add selection ability to table #340
• Add availability metrics to table #489
• Add a button to create a new application
• • Redirect to Create Page if clicked
• Add a button for actions to rename/duplicate/delete selected applications #326
• • Open rename modal if rename clicked #327
• • Open delete modal if delete clicked #328
• • Add ability to delete multiple applications at once #343
• Redirect to Application Details for selected application #378

Create Page

• Add create button
• • Disable create button until required fields are filled out #319
• • Show popover indicating missing field onMouseOver #329
• • Save application to database on click #323
• • Redirect to new application page on click #324
• • Clear storage after creation #377
• Add cancel button
• • Redirect to Home page on click #325
• Make sure name is unique #418

Application Information

• Add text form fields for name(required) and description(optional)
• • Remove text "optional" #288

Composition

• Add accordion for Log Sources
• • Add autocomplete to set base query #292
• • Change to Log Source without s #289
• • Change to PPL Base Query: above search bar #290
• • Change description to "The default logs view in the application will be filtered by this query." #291
• • Add button to clear base query #305
  • • Add pop up modal to double check clear all for log source #320
• Add accordion for Services & entities
• • Add select form field where users can select services and entities
• • Add interactive service graph view where users can select services #293
• • Add button to clear services and entities #304
  • • Add pop up modal to double check clear all for services #321
• Add accordion for Trace groups
• • Add select form field where users can select trace groups or add new ones [EUI Combo Box] #295
• • Add Latency by Trace Group table #296
• • Add button to clear trace groups #308
  • • Add pop up modal to double check clear all for traces #322

Application Details

• Fetch application by ID #382
• Add description to header #387
• Change default time to 24 hours #456
• Save time for individual applications #457
• Remove default filters #458
• Add ability to add extra filters #459

Overview Tab (Dashboard from Trace Analytics)

• Modularize Overview to only include trace groups and services that are part of application #384
• Only add filter for service/trace part of application #388
• Add table to view latency of trace groups
• • Add filter for < 95th percentile or >= 95th percentile
• • Add pop up for enlarged view of 24-hour latency trend
• • Make Traces column data value a link
  • • Redirect to Traces & Spans tab with filter of trace group
• • Make Trace group name column data value a link
• Add Application Composition Map chart
• • Change name of chart from Service Map → Application Composition Map #389
• Add Error Rate chart
• Add Throughput chart
• Toggle to hide services that have no match dashboards-observability#19

Services Tab (Services from Trace Analytics)

• Only include services part of application #390
• Add filter bar
• • Change placeholder text: Service Name #391
• • Only add filter for services part of application #392
• Add table to view services
• • Make Name column data value a link
  • • Open side popup with service details #393
• • Make Traces column data value a link
  • • Redirect to Traces & Spans tab with filter of trace group #394

Traces & Spans Tab (Traces and Span Details Table from Trace Analytics)

• Add filter #309
• • Filter traces and spans by trace id or trace group name
• • Change placeholder to Trace ID or trace group name #395
• Add table to view Traces
• • Make Trace ID column a link
  • • Open slide-out of trace details #396
• Add trace details slide-out (Trace View from Trace Analytics) #397
• • Add Overview panel #398
  • • Make Span bar chart selectable #399
    • • Redirect to Span details Panel when clicked #400
• Add table to view Spans
• • Make Span ID column a link
  • • Open slide-out of span details (Span Detail Flyout from Trace Analytics) #401
    • • Include Payload Event panel #402

Log Events (Log Explorer from Event Analytics)

• Remove query tabs
• Remove save button for events
• Add PPL Query search bar
• • Add read-only Base Query #429
• • Popover with full base query on hover #430
• • Autocomplete start after base query #431
• Save visualization to panels tab

Panels Metrics (Custom Panel View from Operational Panels)

• Create panel when application is created #412
• Bring over operational panel #385
• Modularize to only include visualizations created in app analytics #386
• Add Add Visualization Button
• • Redirect to Log Events Tab #432
• Keep app panel separate #416
• Change Panels to Metrics #461
• Remove panel name #463
• Move search bar up #464
• Change Add Visualization to Add #465
• Add plus icon to Add button #466

Configuration

• Add Header with Application Composition #433
• Add Edit button #434
• • Redirect to create page in edit mode when clicked #435
• • Make base query immutable #467
• Show current Composition #436
• Show current Availability #437

Backend

• Only update mappings when needed #403
• Add appId field to custom panels object #409
• Add panelId field to application object #410
• Add appId field to savedVisualization object #411

Known Bugs

• [-] Adding two visualization deletes first visualization #468
• [-] Page focuses on search bar when service is selected which is disorienting #469
• [-] Clicking on span in Timeline of Trace Detail flyout does not update title #470
• [-] Autocomplete in log config does not complete with correct item selection #471
• Lower margin of autocomplete in log config gets cut off #484
• Autocomplete for data edge cases #485
• Regex matching for multiple indices #486
• Edit visualizations redirect to log events tab #487
• Visualizations in log events do not follow set time range #488
• Suggest span in autocomplete #499
• Only one visualization can have availability level set #537

Solution

Relationship to Other Modules

Dashboard, Trace, Span Detail, Service from Trace Analytics, Log Explorer from Event Analytics, and Panels from Operational Panels will be re-used in Application Analytics. We will need to refactor the components to become more modular. We also need to make sure only the selected services are filtered through when displaying information.

Data Model

Application

• name: User-defined name of application
• description (optional): User-defined description of the application
• baseQuery: PPL query to filter the application by
• servicesEntities: List of services and entities included in application
• traceGroups: List of trace groups included in application
  ( - availabilityLevels: User-defined thresholds that signify custom levels of availability of application
  • label
  • levelPriority
  • color
  • condition ) Temporarily not added

Model

{
   "properties": {
        "name": {
            "type": "keyword",
            },
        "description": {
            "type": "text",
            },
        "baseQuery": {
            "type": "text",
            },
        "servicesEntities": {
            "type": "nested",
        },
        "traceGroups": {
            "type": "nested",
        },
    }
}

Example

{
  "properties": {
    "name": "Application 1",
    "description": "This is my app",
    "baseQuery": "source = flight_logs",
    "servicesEntities": ["Payment"],
    "traceGroups" : ["Payment.auto"]
    }
}

Options for Structure of Availability Levels

Option 1 - List of objects with all of the fields
Pros: Simple

"availabilityLevels": [
    {
    "label": "Available",
    "color": "#50C878",
    "condition": "If otherwise",
    "levelPriority": 0,
    }, {
    "label": "Unavailable",
    "color": "#FF0000",
    "condition": "when errorRate() >= 2",
    "levelPriority": 1,
    },
]

Option 2 - List of objects with key/value pairs. (key: levelPriority) (value: label, color condition in object)
Pros: Easier to sort,

"availabilityLevels": [
    0: {
    "content": {
        "label": "Available",
        "color": "#50C878",
        "condition": "If otherwise",
        }
    }, 
    1: {
    "content": {
        "label": "Unavailable",
        "color": "#FF0000",
        "condition": "when errorRate() >= 2",
        }
    },
]

API Design

APP_ANALYTICS_API_PREFIX='/api/observability/application'

• Fetch all Applications

GET ${APP_ANALYTICS_API_PREFIX}/

RESPONSE BODY
{
  "statusCode": 200,  
  "message": "Ready", 
  "applications": [
    {
      "id": "2FF3GW3H8",
      "name": "Application 1",
      "servicesEntities": ["Payment", "Users"],
      "traceGroups" : ["Payment.auto", "Users.admin"]
    },
    {
      "id": "2FHEP953H",
      "name": "Application 2",
      "servicesEntities": ["Purchase"],
      "traceGroups" : ["Purchase.source"]
    }
  ]
}  

• Fetch Application by ID

GET ${APP_ANALYTICS_API_PREFIX}/{application_id} 
    
    RESPONSE BODY
    {
      "statusCode": 200, 
      "message": "Application Fetched", 
      "application": {
            "name": "Application 1",
            "description": "Description for the application",
            "base_query": "source = opensearch_sample_database_flights",
            "services_entities": [
                "Payment",
                "Users",
                "Purchase"
            ]
            "trace_groups": [
                "Payment.auto",
                "Users.admin",
                "Purchase.source",
            ]
        }
    }

• Create Application

POST ${APP_ANALYTICS_API_PREFIX}/
    
    REQUEST BODY
    {
       "name": "Application 1",
        "description": "Description for the application",
        "base_query": "source = opensearch_sample_database_flights",
        "services_entities": [
            "Payment",
            "Users",
            "Purchase"
        ]
        "trace_groups": [
            "Payment.auto",
            "Users.admin",
            "Purchase.source",
        ]
    }
    
    RESPONSE BODY
    {
      "statusCode": 200, 
      "message": "Application Created", 
      "newAppId": "2FG6FWGY5" // New application ID
    }

• Update Application - all fields are updatable

PATCH ${APP_ANALYTICS_API_PREFIX}/{application_id} 
    
    REQUEST BODY
    {
      "appId": "2FG6FWGY5", // application id to be renamed 
      "base_query": "source = opensearch_sample_database_flights" // fields to update
    } 
    
    RESPONSE BODY
    {
      "statusCode": 200, 
      "message": "Application Updated"
    }

• Duplicate Application

POST ${APP_ANALYTICS_API_PREFIX}/clone 
    
    REQUEST BODY
    {
      "appId": "2FG6FWGY5", // application id to be renamed 
      "name":"Application 1 (copy)" // new name for the application to be duplicated 
    } 
    
    RESPONSE BODY
    {
      "statusCode": 200, 
      "message": "Application Cloned",  
      "newAppId": "2FFAG7HAQ" // newly duplicated application id  
    }

• Delete Application

DELETE ${APP_ANALYTICS_API_PREFIX}/{application_ids}
    
    RESPONSE BODY
    {
      "statusCode": 204, 
      "message": "Application Deleted",   
    }

Autocomplete Logic

Possible PPL commands: source, dedup, eval, fields, head, rare, rename, sort, stats, top, where

for Base Query on Create Page

• Not allowed commands: stats, head, rare, top
• ex: source = opensearch_dashboard_sample_flights | where DestCityName = 'Venice'

for Availability Level Condition on Create Page (Conditions can not use PPL query)

• Allowed commands: where, eval, rename
• Condition must be evaluate to boolean

for Filter on Log Events Tab

• Not allowed commands: source
• ex: where OriginCityName = 'Rome' | stats avg( AvgTicketPrice)

for Filter on Traces & Spans Tab (pseudo) (Using filter instead of PPL query)

[ryn9]

@anirudha
Are these solution patterns being developed with multi-tenancy and various RBAC in mind?
Are there issues and/or other docs that could be linked back to on these solution patterns?

[anirudha]

yes, the observabilty plugin supports RBAC and multi -tenancy using the security plugin

[ryn9]

@anirudha additionally will the plugin support customizable index patterns, etc.. ?
The trace analytics plug-in required use of otel-v1-apm-span-* and otel-v1-apm-service-map*, which will not work in all environments, especially when multi-tenancy comes into play.

[eugenesk24]

@ryn9 Hi Ryan, I talked to Ani about your question. We are planning on supporting regex based index matching in PPL soon, I think he said next release. As to your second question, I'm a little confused how multi-tenancy will affect multiple index matching? I think the security plugin should handle any access restrictions for indices. Please feel free to elaborate.

[ryn9]

@eugenesk24

As to your second question, I'm a little confused how multi-tenancy will affect multiple index matching?

I was trying to say that making specific index patterns required for plugin use can have impact on what is, probably, the most common way to handle multi-tenancy + RBAC - role based access to specific indices. So - I am hoping that static index names are not being considered like they were with the initial trace analytics plugin.

To note - plugins haven't always been thought with multi-tenancy + RBAC in mind - so I am hoping to find out more details about how the observability plugin is being designed with this in mind.

[anirudha]

Hi @ryn9 ,
Could you please elaborate on a use-case you are looking for in some details, for now here are the details.

• Observability plugin support same tenancy structures as all of dashboards eg. link
• RBAC is i.e roles based access control will follow the rules configured by your custer admin on the opensearch setup.
• Currently we don't support index patterns for wildcard indices, this will be a supported in the next few releases. The wildcards will match any permission-ed index available to user for this roles.

The trace analytics plug-in required use of otel-v1-apm-span-* and otel-v1-apm-service-map*, which will not work in all environments, especially when multi-tenancy comes into play.

could you explain this with an example. From what i know access to these are only based on your roles and not tenancy.

[ryn9]

could you explain this with an example. From what i know access to these are only based on your roles and not tenancy.

Sorry to confuse - 'tenancy' was probably the wrong word here.

The trace analytics plug-in required use of otel-v1-apm-span-* and otel-v1-apm-service-map*, which will not work in all environments. For example - we support different groups, whom (via role access) should only have access to certain data. IE.... we wouldn't want let userA and userB both have access to these index patterns, as we don't want to allow them access to each other's data.

[WassimDhib]

Hi

Same problem here : https://discuss.opendistrocommunity.dev/t/trace-analysis-and-multi-tenant/4602/2
Trace Analysis plugin require indices:data/read/search permission on all otel-v1-apm-span-* indices
so either you user have permissions for all indices, or nothing works

I think that supporting index patterns for wildcard indices is the solution.

[ryn9]

@anirudha

Currently we don't support index patterns for wildcard indices, this will be a supported in the next few releases. The wildcards will match any permission-ed index available to user for this roles.

Does that mean the plugins would also be moving away from requiring specific index names?
Perhaps create different 'views' that, upon creation, could use different index patterns?

[ryn9]

@anirudha @eugenesk24 - wondering if you folks have confirmed index pattern support for PPL for 1.3 yet.
It doesn't answer all of the questions above, but I believe is an important step for PPL adoption.

[eugenesk24]

Sorry for the late reply @ryn9 ! We have added multiple index use through comma-separate indices and wildcard matching.

[ryn9]

@eugenesk24 .. awesome. I am excited to see it :)

[eugenesk24]

Merged to main for 1.3 release