[opennms-discuss] ifInOctets / ifOutOctets counter wrap problem :^(

[Michael Hale]

Hello,

I was hoping some of you folks could help me out with a problem I'm
having using the graphing functionality of OpenNMS.

Basically, we're pushing a ton of traffic, about 400 Mbit / sec and
we're trying to monitor the bandwidth utilization on our Cisco 7609.
Since the Cisco MIB seems to define ifInOctets and ifOutOctets as 32-
bit values, even with a polling interval set down to 1 minute, the
counters are wrapping several times between collection intervals - as
you can imagine, this wrecks havok on the graphing :^)

Have any of you encountered this problem? If so, how did you address
it? My first hunch would be to try and configure OpenNMS / rrdtool
to store the value as a 64-bit value, but I'm not quite sure how to
go about this.

Any help would be greatly appreciated!
--
Michael Hale
***@transcomus.com

[Jean-Sebastien Morisset]

Post by Michael Hale
Hello,
I was hoping some of you folks could help me out with a problem I'm
having using the graphing functionality of OpenNMS.
Basically, we're pushing a ton of traffic, about 400 Mbit / sec and
we're trying to monitor the bandwidth utilization on our Cisco 7609.
Since the Cisco MIB seems to define ifInOctets and ifOutOctets as 32-
bit values, even with a polling interval set down to 1 minute, the
counters are wrapping several times between collection intervals - as
you can imagine, this wrecks havok on the graphing :^)
Have any of you encountered this problem? If so, how did you address
it? My first hunch would be to try and configure OpenNMS / rrdtool
to store the value as a 64-bit value, but I'm not quite sure how to
go about this.
Any help would be greatly appreciated!

Try polling every 30 secs. I can monitor a 1Gb interface running flat
out with a 30 second frequency.

LateR!
js.

[Aaron Glenn]

Post by Jean-Sebastien Morisset
Try polling every 30 secs. I can monitor a 1Gb interface running flat
out with a 30 second frequency.

Or you can try polling the proper OID for the 64bit counter (-:

1.3.6.1.2.1.31.1.1.1.6 for ifHCInOctets
1.3.6.1.2.1.31.1.1.1.10 for ifHCOutOctets.

[Jean-Sebastien Morisset]

Post by Aaron Glenn

Post by Jean-Sebastien Morisset
Try polling every 30 secs. I can monitor a 1Gb interface running flat
out with a 30 second frequency.

1.3.6.1.2.1.31.1.1.1.6 for ifHCInOctets
1.3.6.1.2.1.31.1.1.1.10 for ifHCOutOctets.

If you have access to a 64 bit counter. :-) The current "stable" release
of Net-SNMP doesn't.

js.

[Andy Ellsworth]

Post by Michael Hale
Have any of you encountered this problem? If so, how did you address
it? My first hunch would be to try and configure OpenNMS / rrdtool
to store the value as a 64-bit value, but I'm not quite sure how to
go about this.

Your hunch is correct - you want the 64-bit mib-2 counters ifHCInOctets
and ifHCOutOctets. These are standard counters that should certainly be
available on the high speed interfaces of the 6500/7600.

These 64 bit counters are already defined in the OpenNMS data collection
config, and in fact are probably being collected right under your nose.

[Michael Hale]

Save the file, and restart tomcat, which will reread the
snmp-graph.properties file. You don't need to restart OpenNMS (I just
tested it). Then view the SNMP performance for a particular
interface on
the 7600. You should see the 64-bit counters below the normal
counters.
Hope that helps.
-Andy

Thanks - that helps a lot.

I found, as you mentioned, the OIDs, ifHCInOctets and ifHCOutOctets.
If I do an snmpwallk of the router, it shows up there as well. As a
matter of fact, in snmp-graph.properties, in the reports variable,
there *is* a report called mib2.HCbits:

IF-MIB::ifHCOutOctets.12 = Counter64: 428998337713566
IF-MIB::ifHCInOctets.12 = Counter64: 429965289930891


Here's the report:

report.mib2.HCbits.name=Bits In/Out (High Speed)
report.mib2.HCbits.columns=ifHCInOctets,ifHCOutOctets
report.mib2.HCbits.type=interface
report.mib2.HCbits.command=--title="Bits In/Out (High Speed)" \
DEF:octIn={rrd1}:ifHCInOctets:AVERAGE \
DEF:octOut={rrd2}:ifHCOutOctets:AVERAGE \
CDEF:bitsIn=octIn,8,* \
CDEF:bitsOut=octOut,8,* \
LINE2:bitsIn#000000:"Bits In" \
GPRINT:bitsIn:AVERAGE:" Avg \\: %8.2lf %s" \
GPRINT:bitsIn:MIN:"Min \\: %8.2lf %s" \
GPRINT:bitsIn:MAX:"Max \\: %8.2lf %s\\n" \
LINE2:bitsOut#cc0000:"Bits Out" \
GPRINT:bitsOut:AVERAGE:"Avg \\: %8.2lf %s" \
GPRINT:bitsOut:MIN:"Min \\: %8.2lf %s" \
GPRINT:bitsOut:MAX:"Max \\: %8.2lf %s\\n"

But for some reason, it doesn't show up in the web GUI, where I only
have the following reports available under Network Performance Data:

Bits In/Out
Discards In/Out
Errors, In/Out
TCP Open Connections
TCP Errors and Failures
TCP Segments
InOut Traffic
CPU Utilization (Cisco)
Buffer Failures (Cisco)
Cisco Packets Dropped In
Cisco Packets Dropped Out
Available Memory (Cisco)
Cisco Memory
Cisco Packets In/Out
Chassis Temperatures (Cisco)

I followed your suggestion and added mib2.bits-hc to the report
variable, and added the report itself and restarted, but it didn't
show up either.

Is there some other place I have to enable the report?
--
Michael Hale
***@transcomus.com

[Michael Hale]

Post by Michael Hale
I found, as you mentioned, the OIDs, ifHCInOctets and ifHCOutOctets.
If I do an snmpwallk of the router, it shows up there as well. As a
matter of fact, in snmp-graph.properties, in the reports variable,

Sorry - here's the reports variable:

reports=mib2.bits, mib2.bits-hc, mib2.discards, mib2.errors,
mib2.HCbits, mib2.packets, \ etc...

below is the SNMP walk of the MIB on the router, so I know the OID
exists on the router

Post by Michael Hale
IF-MIB::ifHCOutOctets.12 = Counter64: 428998337713566
IF-MIB::ifHCInOctets.12 = Counter64: 429965289930891

--
Michael Hale
***@transcomus.com

[Andy Ellsworth]

Post by Michael Hale
I found, as you mentioned, the OIDs, ifHCInOctets and ifHCOutOctets.
If I do an snmpwalk of the router, it shows up there as well. As a
matter of fact, in snmp-graph.properties, in the reports variable,
But for some reason, it doesn't show up in the web GUI
Is there some other place I have to enable the report?

Hmm. You shouldn't need to make any additional changes - if the RRD
files are present for that particular interface, then ONMS should
automatically put the corresponding graphs in the performance page for
that interface.

A couple of things to check:

- Look in /var/opennms/rrd/snmp/<node number of 7609>/<interface name>/
for files called ifHCInOctets.rrd and ifHCOutOctets.rrd (among many
others). If you have those files under that node and interface, then you
have RRD data.

- If you have RRD data, force ONMS to draw a graph with it. Click
"Performance" from the front page, then choose your 7609 under "Custom
Performance Reports" (the bottom list) and click Start. Pick your
interface, click Next, then on the "Choose the Data Sources" screen,
choose "ifHCInOctets" for Data Source 1 and "ifHCOutOctets" for Data
Source 2. "Next" until you get to a graph. If you see a graph there,
then your RRD files have good data and ONMS can build a graph from them.

- Stop tomcat, rm -rf the contents of $TOMCAT_HOME/work/*
(/var/tomcat4/work/* on my system), and start tomcat. This will clean
out the tomcat 'work' directory.

Try that.

-Andy

[Tarus Balog]

Gang:

Anyone here a Cisco device SNMP guru? I'm looking for a way to retrieve a
counter from the command "show counters", specifically "rxBadCodes" via
SNMP.

-T

[Tarus Balog]

Post by Tarus Balog
Anyone here a Cisco device SNMP guru? I'm looking for a way to retrieve a
counter from the command "show counters", specifically "rxBadCodes" via
SNMP.
-T

Err, make that "rxBadCode" - no "s".

-T

[Michael Hale]

Post by Andy Ellsworth
- Look in /var/opennms/rrd/snmp/<node number of 7609>/<interface name>/
for files called ifHCInOctets.rrd and ifHCOutOctets.rrd (among many
others). If you have those files under that node and interface, then you
have RRD data.

Well, looking under the interface name, it doesn't appear that
information is being collected for ifHCInOctets or ifHCOutOctets - I
don't see the rrd files in any case.

Looking in /opt/OpenNMS/etc/datacollection-config.xml, I see:

<mibObj oid=".1.3.6.1.2.1.31.1.1.1.6" instance="ifIndex"
alias="ifHCInOctets" type="counter"/>
<mibObj oid=".1.3.6.1.2.1.31.1.1.1.10" instance="ifIndex"
alias="ifHCOutOctets" type="counter"/>

I've verified these are the correct OIDs according to Cisco - they're
part of IF-MIB.

Is there something that I'm missing configuration-wise to get RRD to
grab these values from the router? Perhaps a place to specify what
OIDs that RRD stores?

Thanks for your help - I think we're getting closer to figuring this
one out!
--
Michael Hale
***@transcomus.com

[Michael Hale]

Post by Michael Hale
Well, looking under the interface name, it doesn't appear that
information is being collected for ifHCInOctets or ifHCOutOctets - I
don't see the rrd files in any case.

FYI, here's the rrd files in /var/opennms/rrd/snmp/26/
Gi4_4-000b45e0abca :

ifInDiscards.rrd ifOutDiscards.rrd locIfInQueueDrops.rrd
ifInErrors.rrd ifOutErrors.rrd locIfOutPktsSec.rrd
ifInNUcastpkts.rrd ifOutNUcastPkts.rrd locIfOutQueueDrops.rrd
ifInOctets.rrd ifOutOctets.rrd
ifInUcastpkts.rrd locIfInPktsSec.rrd

--
Michael Hale
***@transcomus.com

[Tarus Balog]

Post by Michael Hale
Well, looking under the interface name, it doesn't appear that
information is being collected for ifHCInOctets or ifHCOutOctets - I
don't see the rrd files in any case.

The SNMP gods sayeth unto thee: thou shall use SNMP version 2 when seekest
thou the 64-bit counter. Go unto the snmp-config.xml file a setteth the
version to 2c. Not 3, and not 1, unless thou continuest on to 2c.

-T

[Graeme Fowler]

Post by Tarus Balog
The SNMP gods sayeth unto thee: thou shall use SNMP version 2 when seekest
thou the 64-bit counter. Go unto the snmp-config.xml file a setteth the
version to 2c. Not 3, and not 1, unless thou continuest on to 2c.

1... 2c... 3! No, 2c!

Arrrrrgh!

(damn you, Python)

[Tarus Balog]

Post by Graeme Fowler

Post by Tarus Balog
The SNMP gods sayeth unto thee: thou shall use SNMP version 2 when seekest
thou the 64-bit counter. Go unto the snmp-config.xml file a setteth the
version to 2c. Not 3, and not 1, unless thou continuest on to 2c.

1... 2c... 3! No, 2c!
Arrrrrgh!
(damn you, Python)

Actually, 3 would work too (in trunk), but then the joke wouldn't. I went
with the joke.

-T

[Michael Hale]

Post by Tarus Balog

Post by Michael Hale
Well, looking under the interface name, it doesn't appear that
information is being collected for ifHCInOctets or ifHCOutOctets - I
don't see the rrd files in any case.

The SNMP gods sayeth unto thee: thou shall use SNMP version 2 when seekest
thou the 64-bit counter. Go unto the snmp-config.xml file a setteth the
version to 2c. Not 3, and not 1, unless thou continuest on to 2c.

Yep! I queried the OID with a v1 query and it failed. Then I queried
with v2c and it worked. I looked in snmp-config.xml, changed the
setting to 2c, and what do you know? :^) Fixed it, came back to the
list to report a success - now everything shows up properly!

So thank you everybody for being so helpful! I like OpenNMS quite a
bit more than nagios so far :^)

--
Michael Hale
***@transcomus.com

[Chris Beck]

Cisco's SNMP object navigator doesn't list that counter as being
available via a SNMP OID. None of my devices have rxBadCode in their
show counters output for any of their interface types, so I can't check
it for you. Cisco's knowledgebase only has 2 references to rxBadCode
which gives the description as "rxBadCode - A count of received frames
for which the preamble has a bad code." Searching their SNMP MIBs for
"preamble" doesn't show anything similar either. Searching for "bad
code" yields one MIB related to their VPN concentrators which doesn't
look like the right OID either. I'm guessing it might not be available
via SNMP.

-Chris

-----Original Message-----
From: opennms-discuss-***@lists.sourceforge.net
[mailto:opennms-discuss-***@lists.sourceforge.net] On Behalf Of
Tarus Balog
Sent: Tuesday, October 03, 2006 8:15 AM
To: General OpenNMS Discussion
Subject: Re: [opennms-discuss] OT: Cisco SNMP Gurus

Post by Tarus Balog
Anyone here a Cisco device SNMP guru? I'm looking for a way to
retrieve a counter from the command "show counters", specifically
"rxBadCodes" via SNMP.
-T

Err, make that "rxBadCode" - no "s".

-T


------------------------------------------------------------------------
-
Take Surveys. Earn Cash. Influence the Future of IT Join
SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDE
V
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of
this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

[Tarus Balog]

Post by Chris Beck
Cisco's SNMP object navigator doesn't list that counter as being
available via a SNMP OID. None of my devices have rxBadCode in their
show counters output for any of their interface types, so I can't check
it for you. Cisco's knowledgebase only has 2 references to rxBadCode
which gives the description as "rxBadCode - A count of received frames
for which the preamble has a bad code." Searching their SNMP MIBs for
"preamble" doesn't show anything similar either. Searching for "bad
code" yields one MIB related to their VPN concentrators which doesn't
look like the right OID either. I'm guessing it might not be available
via SNMP.

Yeah, that's the conclusion I came up with as well. Thanks for looking out
for me.

-T

[John Blake]

Looks like it may be part of FlowControl on a 6509 Gig interface.

http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/sw_5_2/comdr
ef/sh_d_i.htm#28329

I noticed it says "Flowcontrol Version 1".

http://puck.nether.net/pipermail/cisco-nsp/2005-May/020856.html


John

-----Original Message-----
From: opennms-discuss-***@lists.sourceforge.net
[mailto:opennms-discuss-***@lists.sourceforge.net] On Behalf Of
Tarus Balog
Sent: Tuesday, October 03, 2006 12:37 PM
To: General OpenNMS Discussion
Subject: Re: [opennms-discuss] OT: Cisco SNMP Gurus

Post by Chris Beck
Cisco's SNMP object navigator doesn't list that counter as being
available via a SNMP OID. None of my devices have rxBadCode in their
show counters output for any of their interface types, so I can't

check

Post by Chris Beck
it for you. Cisco's knowledgebase only has 2 references to rxBadCode
which gives the description as "rxBadCode - A count of received frames
for which the preamble has a bad code." Searching their SNMP MIBs for
"preamble" doesn't show anything similar either. Searching for "bad
code" yields one MIB related to their VPN concentrators which doesn't
look like the right OID either. I'm guessing it might not be available
via SNMP.

Yeah, that's the conclusion I came up with as well. Thanks for looking
out
for me.

-T


------------------------------------------------------------------------
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share
your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDE
V
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of
this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

[John Blake]

So if the device has a gigabit interface in it, and you're using
net-snmp, how do ya deal with the counters rolling?
I'm asking cause it looks like I'm in that situation (Solaris and linux
servers). I thought net-snmp 5.3.1 had support and I asked our guy to
make sure its there in our next upgrade.
I'll have to re-look at this.

John

-----Original Message-----
From: opennms-discuss-***@lists.sourceforge.net
[mailto:opennms-discuss-***@lists.sourceforge.net] On Behalf Of
Jean-Sebastien Morisset
Sent: Wednesday, October 04, 2006 8:33 AM
To: General OpenNMS Discussion
Subject: Re: [opennms-discuss] ifInOctets / ifOutOctets counter wrap
problem:^(

Post by Aaron Glenn

Post by Jean-Sebastien Morisset
Try polling every 30 secs. I can monitor a 1Gb interface running flat
out with a 30 second frequency.

1.3.6.1.2.1.31.1.1.1.6 for ifHCInOctets
1.3.6.1.2.1.31.1.1.1.10 for ifHCOutOctets.

If you have access to a 64 bit counter. :-) The current "stable" release
of Net-SNMP doesn't.

js.
--
Jean-Sebastien Morisset, Sr. UNIX Administrator <***@mvlan.net>
Personal Home Page <http://jsmoriss.mvlan.net/>
Underwater and Travel Photographs <http://www.mvpix.com/>

------------------------------------------------------------------------
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share
your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDE
V
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of
this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

[Jean-Sebastien Morisset]

Post by John Blake
So if the device has a gigabit interface in it, and you're using
net-snmp, how do ya deal with the counters rolling?

Poll every 30 secs.

Post by John Blake
I'm asking cause it looks like I'm in that situation (Solaris and linux
servers). I thought net-snmp 5.3.1 had support and I asked our guy to
make sure its there in our next upgrade.

The next stable release should support 64 bit counters.

LateR!
js.

Post by John Blake
I'll have to re-look at this.
John
-----Original Message-----
Jean-Sebastien Morisset
Sent: Wednesday, October 04, 2006 8:33 AM
To: General OpenNMS Discussion
Subject: Re: [opennms-discuss] ifInOctets / ifOutOctets counter wrap
problem:^(

Post by Aaron Glenn

Post by Jean-Sebastien Morisset
Try polling every 30 secs. I can monitor a 1Gb interface running flat
out with a 30 second frequency.

1.3.6.1.2.1.31.1.1.1.6 for ifHCInOctets
1.3.6.1.2.1.31.1.1.1.10 for ifHCOutOctets.

If you have access to a 64 bit counter. :-) The current "stable" release
of Net-SNMP doesn't.
js.
--
Personal Home Page <http://jsmoriss.mvlan.net/>
Underwater and Travel Photographs <http://www.mvpix.com/>
------------------------------------------------------------------------
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share
your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDE
V
opennms-discuss mailing list
To *unsubscribe* or change your subscription options, see the bottom of
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
opennms-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/opennms-discuss