[Feat]: Agent Connectivity Topology Map

[ss-mbrancato]

Problem

I am not sure in some cases if a node is streaming to a parent or direct to netdata cloud easily.

Description

I would like to see some sort of topology map for my netdata agents to let me know at its most basic level which agents are directly connecting to the cloud and which ones are connected to a parent and which one.

Importance

nice to have

Value proposition

1. This would help troubleshoot connectivity and ensure that configuration files are working as expected.
2. This would also let us view potential blast radius of bringing a given parent down.

Proposed implementation

leverage the existing data widget for parent, child, standalone and the network connection top function to build a network diagram

[sashwathn]

@christophidesp : Can we come up with a design to show this topology?
@kapantzak : Once the design is ready, I believe all of this information already exists in the backend.

[shyamvalsan]

@christophidesp This should probably look a radial tree chart showing all agents, parents, grandparents and netdata cloud at the center. And we can have it as an optional view on the home tab node map view.

[brodonalds]

This would be great if you could click the numbers and see which ones are which. Would help track down agents that are not reporting to parents so it could be remediated.

[brodonalds]

Thought this sounded familiar. #987

[shyamvalsan]

Sharing some work in progress from @ktsaou which is working towards this goal

[sashwathn]

@christophidesp : As discussed in our meeting, these are the set of requirements:

• Topology Map: Hierarchical view
  • Group by parents
• Option to visualise online agents
• Search for specific agents
  • Highlight identified agents
  • Highlight all the agents involved in streaming (parents, grandparents etc)
• Agent Data Flow (Map)
  • For each agent, show the streaming flow
  • Parents, Grandparents etc and then the Cloud

@shyamvalsan @christophidesp : Please add if I have missed any details.

[christophidesp]

You can find here the mockups. The hierarchy is depicted by nesting and creating a tree/drilldown view.
Below attaching a short video of a prototype (note that some elements, like tooltips and dropdowns, are missing, and what is examined here is the nodes depiction and the drill-down/up functionalities)

topology-proto.mov

[sashwathn]

Summary of our initial meeting on parent/child topology visualization:

• Panos' design was applauded and considered sufficient for up to 3 levels. This is also the maximum we recommend to customers.
• There are questions on how to know the topology of the agents:
  • The Agent is the most likely entity for knowing the topology around it.
  • Strong preference for explicit topology information communicated by the Agent, rather than Cloud-side heuristics.
  • Unsure how to Agents can know their (potential) connections in the face of NATs, Load Balancers, etc.
  • What happens if the topology changes?
• Agreement to reconvene with a representative of the Agent team (@stelfrag )

cc: @christophidesp @ralphm @papazach @car12o @juacker

[sashwathn]

@stelfrag : Do we have any update on the information that is needed by cloud to make a meaningful representation of the topology - especially for active-active parents?

[ktsaou]

The information needed is already exposed by /api/v3/node_instances.

The response includes the stream path of all nodes, across all parents and it is available on every node, children and parents.