Dealing with removal of interfaces table from supplied snmp.yml

[candlerb]

I've just noticed that the walking of ifTable has been removed from the model-specific sample configs in commit 4dc6101, e.g.

   mikrotik:
     walk:
-      - interfaces
-      - ifMIB
       - laIndex
-      - sysUpTime
       - sysDescr
       - host
       - mikrotik

This means that for a Mikrotik router, which I currently poll using module=mikrotik, I no longer get interface stats :-(

I am wondering what's the recommended way to deal with this?

• I could have multiple SNMP scrape jobs, one for each MIB. Then I'd scrape the device once with module=if_mib and again with module=mikrotik.
  • The scrapes would not happen at the same time
  • Every time I deploy a new MIB I'd have to add a new scrape job, with the appropriate subset of targets
• I could ignore the sample config entirely, and build custom configs using generator.yml for every single device type
  • I would have to combine MIB sections to build them in the way the old samples worked, e.g. take the "if_mib" section and the "mikrotik" section and combine them into a new "if_mib_mikrotik". This would require a YAML preprocessor.

In an ideal world, I'd scrape once with e.g. module=if_mib,mikrotik but I don't believe this is doable today. Nor is it possible to use module=if_mib&module=mikrotik; and in any case, since I use relabelling to configure the target scraping, it's only possible to put one value in __param_module

Any other possibilities I'm missing?

[SuperQ]

There is this issue about adding support for multiple modules in a single request. #731

It's something I would like to implement, but don't have time currently to do it myself.

The scrapes would not happen at the same time

This is actually an advantage, since scrapes typically cause CPU usage spikes on devices. Many network engineers see CPU use as a "problem" rather than simply a side effect of the device doing work. IMO this is a non-problem, but I can't really change some people's minds. By having the data collected in multiple scrapes, this spreads the load out over time, which reduces the perceived CPU problem.

What we've done is tried to split the metrics into different groups, such that precise time correlation is not necessary as the data is correlated with each other. IE, system information and traffic information don't need exact time correlation and within the same scrape interval is good enough.

Also note, the "official" generator and snmp yaml configs are meant as examples. They are not meant to be canonical configurations for every site.

Right now the exporter can only load one config file at a time. I want to add multiple-file support eventually.

Longer term, I've wanted to build a library repo of more canonical device configurations. This way users can more easily download pre-canned device modules. Now that we've separated auths from walks, we're a step closer to this being easier to do. Next up is multi-file support.

[candlerb]

It's something I would like to implement, but don't have time currently to do it myself.

It sounds like it ought to be straightforward: split 'module' param into a list, iterate over the list. If I get a chance I'll take look.

By having the data collected in multiple scrapes, this spreads the load out over time, which reduces the perceived CPU problem

I guess it depends how the scrape interval compares to the CPU load interval that it's being averaged over. The long term average is obviously going to be the same, and I agree that shorter spikes where the CPU is doing work are perfectly normal.

If you wanted to smear out the spikes, I think putting a short configurable delay after each SNMP exchange would be the way to go.

[candlerb]

It sounds like it ought to be straightforward: split 'module' param into a list, iterate over the list. If I get a chance I'll take look.

There's one wrinkle here. Collector() is split into two phases: first ScrapeTarget, then buildMetricTree which emits the desired metrics. It generates some low-level scrape metrics from the first part, and the total snmp_scrape_duration_seconds at the end.

I could attempt to merge all the module trees together (which would also de-duplicate any tables being walked), but this would become very messy when it comes to filtering being done on those tables.

For simplicity and consistency, I think that module=foo,bar should do the same work and return exactly the same results as two scrapes, one with module=foo and one with module=bar

However, then I have to modify the scrape stats. Either I could generate multiple instances:

snmp_scrape_walk_duration_seconds{module="foo"}
snmp_scrape_walk_duration_seconds{module="bar"}
snmp_scrape_packets_sent{module="foo"}
snmp_scrape_packets_sent{module="bar"}
... etc

but that would be a non-backwards-compatible change (and for consistency probably should be done on all metrics; that also solves the problem of what happens if two modules return the same metric).

Or else I could merge the stats to give a total amount of time walking, total packets sent and so on. I guess that's what I should do for now, to make it a minimally-intrusive change.

[SuperQ]

If you wanted to smear out the spikes, I think putting a short configurable delay after each SNMP exchange would be the way to go.

I was just thinking this as well. Basically we would need a sleep between packets. This would need to be a feature of the upstream gosnmp library.

[RichiH]

Side note: I don't think we should cater to "too much CPU" as a default use case. Plus, I would sometimes need to see if e.g. a route change lead to CPU spike or reduced forwarding, so "precisely what he same time" has value. Sent by mobile; please excuse my brevity.

…

Message ID: ***@***.*** com>