Improve update-l2database.pl

[rowanthorpe]

• add & use adapted version of Net::SNMP::Mixin::Util::normalize_mac()
  (needed because one stray MAC was outputing as ASCII, untranslated)
• delete old copy of update-l2database.pl from sflow directory
• copy comment about testing status on switches from old sflow version,
  and add Extreme BD-8806 to list
• apply $do_nothing to all DB-affecting actions
• improve debug output layout and increase output
• replace a printf occurrence with print
• standardise whitespace

[barryo]

@nickhilliard - this one's for you (thanks to your superior Perl foo!). Please note:

• I made some fixes last month which aren't recorded in the commit history as I just inserted my fixed version. @rowanthorpe is deleting the correct old file.
• the fixes were related to incorrect mapping of interfaces (works well on most switches but not all). The correction was to use the OID defined by @interface2ifindex.
• it should be noted that this script works well on Extreme models: X460-48t, X650-24x(SSns) and X670V-48x also.

[rowanthorpe]

By the way: the snmp "bug" (not translating the output to hex if it decides the packed-string MAC looks like valid ASCII) also applies to the cli snmp tools and to all the language bindings I've seen - including PHP. So there needs to be an equivalent fix/workaround in the PHP controller which snmp-queries the switch in realtime. I haven't zeroed-in on location of that code yet.

Also, it's not exactly an snmptools bug, more of a documentation-bug due to it being a nasty gotcha with almost no text explaining that this can happen when the heuristics occasionally misfire. With lower-level lib-interfaces it is possible to pass a parameter to force the output format to "translated-to-hex" from the outset (rather than deciding heuristically), but this perl script uses a more high-level wrapping library which doesn't pass the necessary parameters for that, so I just added a "normalize" function after the fact instead. I think I remember seeing that the used PHP libs can pass those parameters, so they should be able to do the workaround in a less circuitous manner.

For the past month one of the "macaddresses" I've been looking at (on commandline and in IXP-Manager's interface) has looked like this \st....!! For a while it seemed it was a switch-bug. Luckily one of our rockstar server-admins was able to diagnose it for me the other day.

[rowanthorpe]

(I just added a commit to add the models mentioned by @barryo to the comments)

[rowanthorpe]

Is there anything I can/must do to make this PR (more) acceptable? It doesn't make any very invasive changes, but makes one absolutely vital fix (for those unfortunate enough to have a switch with a MAC in the doomed range). Before applying this I was unable to get graphs or anything because they were all borking on the garbage value. Also, I am still not sure whether an equivalent fix is needed for the php snmp calls, but I remember before this fix I was still receiving garbage values even when probing from within the interface (i.e. via php). Strangely now with this PR applied on my fork it apparently has no problem via perl or php (even though I only patched the perl).

[bcix]

For me, it seems update-l2database.pl doesn't work with Dell Force10 S4810 Switches

[rowanthorpe]

@bcix Does it not work even with this PR applied? Or does it not work in its present state? It didn't work for me (with an Extreme switch) because of the heuristic translation problem until I applied this normalisation change. The OCTET_STRING representation of the MACs of some switch models fall in a range where the first bytes can be treated as valid ASCII, so the heuristics make a wrong guess and they get turned into weird short strings. If memory serves, running update-l2database.pl manually (having edited it so $debug = 1) should show if any of the returned MACs are being converted in this way.

[bcix]

@rowanthorpe I just use your latest version and I still only see the MACs from my Brocade BigIron RX, but not from the Force10 S4810 in the IXP-Manager frontend.

With your $debug =1 I'll see "undef" for my force10 switches:

./tools/runtime/l2database/update-l2database_2.pl

$l2mapping = {
'f10-n15' => undef,
'f10-g60' => undef,
'f10-k15' => undef,
'rx8-l105' => {
'10GigabitEthernet6/1' => [
'001cb0b54e00'
..

[rowanthorpe]

@bcix I posted this pull request two months ago and now my memory is a bit hazy about the sequence of problems I encountered, but I think I also hit a few "undefs" along the way. When you say "my latest version" - do you mean you pulled the "update-l2database-fixes" branch from my fork (which this PR references)? I am not a part of the project, I just submit PRs as I hit bugs myself :-) Although @nickhilliard is probably the one to answer authoritatively about it, I suspect your problem may not be related to this PR - possibly something like firewalling?, SNMP "permissions"? (I struck all of those problems too...).

[barryo]

@rowanthorpe I was leaving the PR to @nickhilliard but I may review it myself soon if he doesn't.

@bcix Hi Thorlief and welcome to the project! Can you post the snmpwalk output of the following OIDs for us?

snmpwalk -v 2c -c xxxxxx $switchip .1.3.6.1.2.1.2.2.1.2
snmpwalk -v 2c -c xxxxxx $switchip .1.3.6.1.2.1.17.1.4.1.2
snmpwalk -v 2c -c xxxxxx $switchip .1.3.6.1.2.1.17.4.3.1.2
snmpwalk -v 2c -c xxxxxx $switchip .1.3.6.1.2.1.17.4.3.1.1

Probably best to post it in a Gist ( https://gist.github.com/ ).

[bcix]

@barryo here we go with the SNMP OID output https://gist.github.com/bcix/6592380

@rowanthorpe No SNMP-Permission problem, connection is working.

[nickhilliard]

@rowanthorpe i mis-read the patch slightly on day one and decided that it needed a little work, but having gone through it carefully it's actually fine and I'll commit as-is.

@bcix re: https://gist.github.com/bcix/6592380, oh dear. This patch won't fix that problem. Can you take the problem to ixpmanager@inex.ie and we can discuss it there because I'm going to close this pull request.

[barryo]

@bcix well there's the problem.

• .1.3.6.1.2.1.2.2.1.2' (interface descriptions) is fine;
• .1.3.6.1.2.1.17.1.4.1.2 (ifindex of the interfaces) should have an equal number of entries as there are physical ports (at least, may also contain virtual ports and others);
• .1.3.6.1.2.1.17.4.3.1.2 is used to map MAC address entries to ports - doesn;t seem to be available on your box;
• .1.3.6.1.2.1.17.4.3.1.1 is used to get the MAC addresses - also doesn't seem to be available on your box.

We may not necessarily be using the best possible OIDs (although they work fine on Brocade, Cisco and Extreme at least).

I look into this further, I'd need a complete SNMP dump (snmpwalk -On -v 2c -c xxxxxx $switchip .) which you can email to barry.odonovan at inex.ie. But I won't get a chance to look at this this week.

[pdxmaverick]

I too am running into the same issue where using a Juniper switch.
https://gist.github.com/pdxmaverick/6597601

Brian Thompson
Senior Infrastructure Engineer // Senior Second Guesser

Direct: 503.943.6779
Mobile: 503.707.9018 // Twitter: iovation
www.iovation.com

On Tue, Sep 17, 2013 at 3:22 AM, barryo notifications@github.com wrote:

@bcix https://github.com/bcix well there's the problem.

.1.3.6.1.2.1.2.2.1.2' (interface descriptions) is fine;

.1.3.6.1.2.1.17.1.4.1.2 (ifindex of the interfaces) should have an
equal number of entries as there are physical ports (at least, may also
contain virtual ports and others);

.1.3.6.1.2.1.17.4.3.1.2 is used to map MAC address entries to ports -
doesn;t seem to be available on your box;

.1.3.6.1.2.1.17.4.3.1.1 is used to get the MAC addresses - also
doesn't seem to be available on your box.

We may not necessarily be using the best possible OIDs (although they work
fine on Brocade, Cisco and Extreme at least).

I look into this further, I'd need a complete SNMP dump (snmpwalk -On -v
2c -c xxxxxx $switchip .) which you can email to barry.odonovan at inex.ie.
But I won't get a chance to look at this this week.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/47#issuecomment-24577556
.

[barryo]

Hi Brian,

We pushed this discussion to the mailing list.

Archives: https://www.inex.ie/pipermail/ixpmanager/2013-September/thread.html
Subscription: https://www.inex.ie/mailman/listinfo/ixpmanager

Can you check of any of the posts help and take it up there?

[pdxmaverick]

Hi Barry,

I read through the archive, I am sorry I am not sure how to join that conversation.
https://www.inex.ie/pipermail/ixpmanager/2013-September/000136.html is simmilar

I also can:
root@portal:/usr/local/ixp# snmpwalk -v 2c -c xxxxxx exxxxxxxt .1.3.6.1.2.1.17.7.1.2.2.1.2
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.0.0.0.0.0 = INTEGER: 0
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.1.99.142.92.0 = INTEGER: 535
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.3.50.175.76.25 = INTEGER: 541
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.3.108.234.188.27 = INTEGER: 513
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.11.69.10.72.0 = INTEGER: 521
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.12.133.209.66.16 = INTEGER: 544
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.13.102.237.202.102 = INTEGER: 525
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.18.30.196.16.219 = INTEGER: 534
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.18.242.244.163.0 = INTEGER: 544
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.20.246.141.48.31 = INTEGER: 518
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.20.246.246.80.62 = INTEGER: 545
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.22.156.108.125.0 = INTEGER: 540
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.22.156.111.100.64 = INTEGER: 551
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.23.203.164.21.252 = INTEGER: 523
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.25.7.170.156.128 = INTEGER: 513
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.26.162.236.136.64 = INTEGER: 561
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.27.33.22.177.48 = INTEGER: 565
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.27.237.177.206.0 = INTEGER: 520
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.27.237.229.201.96 = INTEGER: 529
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.28.15.92.152.64 = INTEGER: 542
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.28.87.210.184.132 = INTEGER: 544
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.29.181.160.143.240 = INTEGER: 513
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.30.19.228.244.64 = INTEGER: 533
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.31.18.218.248.2 = INTEGER: 537
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.38.152.20.145.5 = INTEGER: 528
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.39.12.237.251.129 = INTEGER: 534
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.39.13.253.182.0 = INTEGER: 536
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.80.11.56.180.25 = INTEGER: 532
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.0.208.43.25.65.0 = INTEGER: 552
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.48.247.13.147.186.177 = INTEGER: 566
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.64.85.57.28.233.187 = INTEGER: 539
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.92.94.171.54.51.15 = INTEGER: 517
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.92.94.171.209.216.101 = INTEGER: 519
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.92.94.171.210.66.120 = INTEGER: 562
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.92.94.171.214.216.120 = INTEGER: 530
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.92.94.171.220.126.121 = INTEGER: 515
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.108.156.237.41.204.205 = INTEGER: 531
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.120.254.61.15.112.164 = INTEGER: 524
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.120.254.61.239.22.68 = INTEGER: 545
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.124.32.100.230.236.203 = INTEGER: 527
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.136.224.243.40.30.1 = INTEGER: 516
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.136.224.243.122.196.100 = INTEGER: 543
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.136.224.243.125.121.193 = INTEGER: 547
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.196.100.19.201.3.32 = INTEGER: 538
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.248.192.1.216.148.136 = INTEGER: 549
iso.3.6.1.2.1.17.7.1.2.2.1.2.2.252.251.251.67.161.3 = INTEGER: 566

And did verify, those are the correct mac addresses with the INTEGER being the interface index.

Unfortunately this coding is above my head. I am using a Juniper EX4500. Any help would be greatly appreciated. Our exchange is going through big changes for the better. www.nwax.net

Cheers,
Brian

[barryo]

See "Subscribing to ixpmanager" on https://www.inex.ie/mailman/listinfo/ixpmanager

Nick is taking a look at the code soon for Thorlief in BCIX (same issue as you) so hopefully we'll have something soon.