Sync dependencies #347
Comments
|
Another idea: don't sync dependencies to the relational database at all but instead only keep them in Redis. Whenever Web wants to display something about dependencies, the required graph algorithms are implemented as Lua scripts and executed on the Redis server. Only downside with this: you couldn't use the dependency information in filters then (trying to support this would probably require some nasty work merging result sets from Redis and SQL), but things like showing all (reverse) dependencies of a checkable or rendering nice dependency graphs starting from a checkable should work nicely. There could also be some combined solution where we think about what filtering on dependencies should be supported, write information specifically for these queries to the relational database and use Lua for everything else. |
|
My proposal, which fits nicely with what is planned in Web, uses primarily recursive common table expressions to query connected nodes in either direction. Required database versions:
The related tables consist of the following structures:
Schema AdditionsTablesdependency
dependency_state
redundancy_group
redundancy_group_state
dependency_node
dependency_edge
RelationserDiagram
dependency_node ||--|| host : is
dependency_node ||--|| service : is
dependency_node ||--|| redundancy_group : is
dependency_edge }|--|{ dependency_node : "refers to / referred by"
dependency_edge ||--|| dependency : "declared by / declares"
dependency ||--|| dependency_state : has
dependency }|--|| redundancy_group : "member of/has members"
dependency }|--|| timeperiod : "uses / used by"
redundancy_group ||--|| redundancy_group_state : has
Example QueriesRoot ProblemsWITH RECURSIVE UnreachableNodes AS (
SELECT id as node_id,
0 as height,
cast('' as binary(20)) as ref_node_id,
dependency_node.host_id,
cast('' as binary(20)) as service_id,
cast('' as binary(20)) as dependency_id,
cast('' as binary(20)) as redundancy_group_id
FROM dependency_node
WHERE dependency_node.host_id = '<host-id>'
and dependency_node.service_id is null
UNION ALL
SELECT e.to_node_id,
r.height + 1,
e.from_node_id,
tn.host_id,
tn.service_id,
e.dependency_id,
tn.redundancy_group_id
FROM dependency_edge e
INNER JOIN UnreachableNodes r ON e.from_node_id = r.node_id
INNER JOIN dependency_node tn ON e.to_node_id = tn.id
LEFT JOIN host_state hs on hs.host_id = tn.host_id
LEFT JOIN service_state ss on ss.service_id = tn.service_id
LEFT JOIN dependency_state ds ON e.dependency_id = ds.dependency_id
LEFT JOIN redundancy_group_state rgs on rgs.redundancy_group_id = tn.redundancy_group_id
WHERE ds.failed = 'y'
or rgs.failed = 'y'
or hs.is_reachable = 'n'
or ss.is_reachable = 'n')
SELECT rn.node_id,
rn.height,
rn.ref_node_id,
h.name as host_name,
s.name as service_name,
d.name as dependency_name,
rg.name as redundancy_group_name
FROM UnreachableNodes rn
LEFT JOIN host h on rn.host_id = h.id
LEFT JOIN service s on rn.service_id = s.id
LEFT JOIN dependency d on rn.dependency_id = d.id
LEFT JOIN redundancy_group rg on rn.redundancy_group_id = rg.id;Affected ChildrenWITH RECURSIVE UnreachableNodes AS (
SELECT id as node_id,
0 as depth,
cast('' as binary(20)) as ref_node_id,
dependency_node.host_id,
cast('' as binary(20)) as service_id,
cast('' as binary(20)) as dependency_id,
cast('' as binary(20)) as redundancy_group_id
FROM dependency_node
WHERE dependency_node.host_id = '<host-id>'
and dependency_node.service_id is null
UNION ALL
SELECT e.from_node_id,
r.depth + 1,
e.to_node_id,
fn.host_id,
fn.service_id,
e.dependency_id,
fn.redundancy_group_id
FROM dependency_edge e
INNER JOIN UnreachableNodes r ON e.to_node_id = r.node_id
INNER JOIN dependency_node fn ON e.from_node_id = fn.id
LEFT JOIN host_state hs on hs.host_id = fn.host_id
LEFT JOIN service_state ss on ss.service_id = fn.service_id
LEFT JOIN dependency_state ds ON e.dependency_id = ds.dependency_id
LEFT JOIN redundancy_group_state rgs on rgs.redundancy_group_id = fn.redundancy_group_id
WHERE ds.failed = 'y'
or rgs.failed = 'y'
or hs.is_reachable = 'n'
or ss.is_reachable = 'n')
SELECT rn.node_id,
rn.depth,
rn.ref_node_id,
h.name as host_name,
s.name as service_name,
d.name as dependency_name,
rg.name as redundancy_group_name
FROM UnreachableNodes rn
LEFT JOIN host h on rn.host_id = h.id
LEFT JOIN service s on rn.service_id = s.id
LEFT JOIN dependency d on rn.dependency_id = d.id
LEFT JOIN redundancy_group rg on rn.redundancy_group_id = rg.id;Direct ParentsSELECT e.to_node_id,
h.name as host_name,
s.name as service_name,
d.name as dependency_name,
rg.name as redundancy_group_name
FROM dependency_edge e
INNER JOIN dependency_node tn ON e.to_node_id = tn.id
INNER JOIN dependency_node fn ON e.from_node_id = fn.id
LEFT JOIN host h on tn.host_id = h.id
LEFT JOIN service s on tn.service_id = s.id
LEFT JOIN dependency d on e.dependency_id = d.id
LEFT JOIN redundancy_group rg on tn.redundancy_group_id = rg.id
WHERE fn.host_id = '<host-id>'
and fn.service_id is null;Direct ChildrenSELECT e.from_node_id,
h.name as host_name,
s.name as service_name,
d.name as dependency_name,
rg.name as redundancy_group_name
FROM dependency_edge e
INNER JOIN dependency_node tn ON e.to_node_id = tn.id
INNER JOIN dependency_node fn ON e.from_node_id = fn.id
LEFT JOIN host h on fn.host_id = h.id
LEFT JOIN service s on fn.service_id = s.id
LEFT JOIN dependency d on e.dependency_id = d.id
LEFT JOIN redundancy_group rg on fn.redundancy_group_id = rg.id
WHERE tn.host_id = '<host-id>'
and tn.service_id is null; |
That's redundant with the host/service state information and I believe it also doesn't provide the information you hope for. I think you might want a Similarly for redundancy groups. After all, they are just a set of dependencies that counts as failed if all individual dependencies are failed.
This table can become huge. Imagine a dependency structure like the following (boxes are hosts, arrows are dependencies), scale up the top and bottom level and the size of this table will grow quadratically in the size of the Icinga 2 config. graph TD;
A1-->B;
A2-->B;
A3-->B;
A4-->B;
B-->C1;
B-->C2;
B-->C3;
B-->C4;
So if this table is necessary, I see three options:
Why not simply stick to "child" and "parent"?
How do these map to the tables given? Also, can you please give an example how you'd expect these additional virtual nodes to be generated? At least with redundancy groups, that's not obvious. |
Yeah, I thought of this, but figured this would be noted sooner or later, as it happend now ;)
Before this becomes a problem (for the database or our queries), doesn't Icinga already suffer from this anyway? I'd go with 3 for now.
Hah, I knew this question comes up. Because the table is called _edge, not _relation, and left or right were not neutral enough to me. 😅
Eh. There is no simple closure table. Added the names to the other two.
Done. |
To be honest, I don't know what would happen exactly, it's quite possible that things will get slower with lots of dependencies. But so far, it does not expand all indirect dependencies, so it's not obvious that this problem already exists.
Just to be sure: we're still talking about a directed graph here (otherwise, I don't know how this should work)? And the edges are aligned in the direction between a child and a parent, the edge just doesn't necessarily directly connect to something that's a host/service, i.e. the parent/child of a dependency in Icinga 2.
That's not what I meant. More like which of these options do you have in mind? Or something totally different? graph LR;
ChildHost-->RedundancyGroup;
RedundancyGroup-->ParentHostA;
RedundancyGroup-->ParentHostB;
ChildHost-->Dependency;
Dependency-->ParentHostC;
graph LR;
ChildHost-->RedundancyGroup;
RedundancyGroup-->DependencyA;
DependencyA-->ParentHostA;
RedundancyGroup-->DependencyB;
DependencyB-->ParentHostB;
ChildHost-->Dependency;
Dependency-->ParentHostC;
|
|
As Julian have already pointed out, the
I'm not sure at the moment why this is going to be useful, but technically redundancy groups only define a way to construct alternate dependencies and don't provide any actual states by themselves. So, why not just link the
Looking at this where clause, I'm wondering why the nodes are being expanded to themselves in the first place! Isn't that an indication of a cyclic graph (dependency), which Icinga 2 clearly prohibits, since Icinga 2 Footnotes
|
It is not required to re-construct any relationships, yes. But it stores information crucial for a very specific use-case more efficiently. At least compared to work required to fetch the same information by traversing the edges each time. This table is only there to easily aggregate the number of affected children, as in the example. Without this table, this information cannot be shown in the UI in lists. (e.g. host and service list)
It's this alternation, which is abstracted away by this table. I don't want to have to express this in a query. Icinga knows about this, and can evaluate/update it in case it changes.
This doesn't check for cyclic dependencies. It's used to know in advance that the recursion will stop with this row. |
|
I've updated the proposal now with what I've discussed with @julianbrost yesterday. This is mainly:
|
The total number of affected children, which Web would like to show in a host's or service's list item, is better be calculated by Icinga (DB) in advance and stored in table The list of affected children will either be not shown, be a simple link to the zoomed in map or only be visible in case the parent has a problem while the list then only includes unreachable nodes. This is because, traversing the hierarchy top down (with a CTE) is considered to be very expensive, compared to a bottom up approach which is done when querying root problems. |
Models for the new schema additions described here: Icinga/icingadb#347 (comment)
At the moment dependencies are not synchronized in Redis and the database and there is no idea for the storage scheme yet.
Since dependencies represent trees (or even graphs?), They can be difficult to store in the database.
At the moment I can think of the following ways to store the dependencies in the database:
I'm sure there's more.
All schemes must be compared in terms of complexity and performance when writing and reading.
I think it also makes sense to store the parent dependencies directly for the objects. This would allow us to display these in the detail areas in Web without much effort.
The text was updated successfully, but these errors were encountered: