[opennms-discuss] Log4j SNMP Monitoring

[Dylan Hansen]

Greetings everyone,

We are currently having an issue setting up SNMP monitoring using
OpenNMS, and I have decided to post to this list to see if anyone out
there has a similar configuration as we do, and if you could give me
any pointers. Here is what we'd like to do:

Currently, we have our applications running on a distributed system
using JBoss 4. JBoss uses Log4j, much like OpenNMS does. We currently
use the SMTP appender in our JBoss Log4j configuration, which will fire
off an e-mail to our on-call phone when an error occurs. This works
fine for intermittent error messages, but when there is a major problem
with our system, we get fired off hundreds of e-mails to the on-call
phone, overloading it. A better solution in our eyes would have
OpenNMS page the on-call phone when the amount of JBoss errors exceeds
a certain threshold. Also, it would be nice to have SNMP data reports
on the amount of warnings and errors that are in our logs.

Log4j does provide a SNMP trap appender, which uses the JoeSNMP
library, that will fire off a trap to the specified management host.
The issue I am having is that OpenNMS does not receive these traps when
they occur. I have not gotten to the point where I have to configure
OpenNMS to build graphs based on the SNMP data, I am first trying to
get OpenNMS to recognize the fact that it received the trap.

So, I'm curious as to what sort of configuration on the OpenNMS side is
required. I notice in the event log of the node I am sending the traps
from, that it has received traps before as an "unformatted enterprise
event". Although these aren't the traps being sent from Log4j, I would
assume that the Log4j traps would be received in the same way. Then,
there would have to be some configuring on the OpenNMS side to parse
the event OID and gather data upon that. Am I correct in this? Or am
I missing a step here in having OpenNMS recognize an event, whether it
is formatted or not?

If anyone subscribed to this list has successfully had OpenNMS receive
Log4j traps, I would be interested in hearing from you. A snipped of
my log4j.xml file is included below (in this configuration, I have the
threshold set to INFO so I don't have to generate an error for
testing):

<appender name="TRAP_LOG"
class="org.apache.log4j.ext.SNMPTrapAppender">
<errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
<param name="ImplementationClassName"
value="org.apache.log4j.ext.JoeSNMPTrapSender"/>
<param name="ManagementHost" value="192.*.*.1"/>
<param name="ManagementHostTrapListenPort" value="162"/>
<param name="EnterpriseOID" value="1.3.6.1.4.1.24.0"/>
<param name="LocalIPAddress" value="192.*.*.50"/>
<param name="LocalTrapSendPort" value="161"/>
<param name="GenericTrapType" value="6"/>
<param name="SpecificTrapType" value="12345678"/>
<param name="CommunityString" value="public"/>
<param name="ForwardStackTraceWithTrap" value="true"/>
<param name="Threshold" value="INFO"/>
<param name="ApplicationTrapOID"
value="1.3.6.1.4.1.24.12.10.22.64"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern"
value="%d,%p,[%t],[%c],%m%n"/>
</layout>
</appender>

<category name="com.how2share">
<priority value="INFO"/>
<appender-ref ref="TRAP_LOG"/>
</category>

<root>
<appender-ref ref="CONSOLE"/>
<appender-ref ref="FILE"/>
<appender-ref ref="TRAP_LOG"/>
<appender-ref ref="SMTP"/>
</root>

If anyone else is looking for a similar solution, the website to the
SNMPTrapAppender is:
http://www.m2technologies.net/asp/snmpTrapAppender.asp

Any input is appreciated, thank you very much for your time!

Dylan Hansen
How2Share Technologies Inc.

PS. A couple of my colleagues that went to JBoss World in Atlanta said
they had the opportunity to meet some of you nice OpenNMS folks.
Hopefully I too will have the pleasure sometime in the future!

[Gerald Turner]

Hello Dylan, this is a very interesting subject to me since I am working
on projects utilizing JBoss 3.2 and OpenNMS, although unrelated to
monitoring our JBoss servers with OpenNMS - you got me side tracked for
a few hours trying to get log4j to send traps :P

Did you have any luck getting SNMPTrapAppender to send traps to OpenNMS?
I'm having trouble even getting that far - it seems as though the
appender does nothing. I've even tried using the
NetSnmpCommandLineTrapSender implementation which has quite a bit of
diagnostics (enabled via System properties -Dlog4j.debug=true
-Dlog4j.ext.snmpTrapAppender.netSnmp.diagnostic=true) but it has been
utterly silent.

However the JBoss SNMP Adaptor does work - I get 'coldstart' trap sent
to OpenNMS whenever JBoss starts. This feature of JBoss is probably not
what you're looking for since it taps into JMX Notifications, not
logging in general. Looks like JBoss is sending "127.0.0.1" in the trap
which is unfortunate because OpenNMS assigns an event to the wrong node.

As for your question about how to configure OpenNMS to
collect/report/threshold these traps: Well I've only begun to grok the
OpenNMS source a few weeks ago so I'm still a newbie but intuition tells
me that this is not something OpenNMS will be able to do entirely. The
traps get converted to events, the event could be configured to have a
certain severity which would page you, however this is quite similar to
your Log4j SMTP flood. I don't think that a trap could be collected to
an RRD, though if that were possible some interval were taken into
account (traps per second) then it does seem that the threshold system
could page you at the appropriate time. I could be completely mistaken
though!

Post by Dylan Hansen
Greetings everyone,
We are currently having an issue setting up SNMP monitoring using
OpenNMS, and I have decided to post to this list to see if anyone out
there has a similar configuration as we do, and if you could give me
Currently, we have our applications running on a distributed system
using JBoss 4. JBoss uses Log4j, much like OpenNMS does. We currently
use the SMTP appender in our JBoss Log4j configuration, which will
fire off an e-mail to our on-call phone when an error occurs. This
works fine for intermittent error messages, but when there is a major
problem with our system, we get fired off hundreds of e-mails to the
on-call phone, overloading it. A better solution in our eyes would
have OpenNMS page the on-call phone when the amount of JBoss errors
exceeds a certain threshold. Also, it would be nice to have SNMP data
reports on the amount of warnings and errors that are in our logs.
Log4j does provide a SNMP trap appender, which uses the JoeSNMP
library, that will fire off a trap to the specified management host.
The issue I am having is that OpenNMS does not receive these traps
when they occur. I have not gotten to the point where I have to
configure OpenNMS to build graphs based on the SNMP data, I am first
trying to get OpenNMS to recognize the fact that it received the
trap.
So, I'm curious as to what sort of configuration on the OpenNMS side
is required. I notice in the event log of the node I am sending the
traps from, that it has received traps before as an "unformatted
enterprise event". Although these aren't the traps being sent from
Log4j, I would assume that the Log4j traps would be received in the
same way. Then, there would have to be some configuring on the OpenNMS
side to parse the event OID and gather data upon that. Am I correct in
this? Or am I missing a step here in having OpenNMS recognize an
event, whether it is formatted or not?
If anyone subscribed to this list has successfully had OpenNMS receive
Log4j traps, I would be interested in hearing from you. A snipped of
my log4j.xml file is included below (in this configuration, I have the
threshold set to INFO so I don't have to generate an error for
<appender name="TRAP_LOG"
class="org.apache.log4j.ext.SNMPTrapAppender">
<errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
<param name="ImplementationClassName"
value="org.apache.log4j.ext.JoeSNMPTrapSender"/>
<param name="ManagementHost" value="192.*.*.1"/>
<param name="ManagementHostTrapListenPort" value="162"/>
<param name="EnterpriseOID" value="1.3.6.1.4.1.24.0"/>
<param name="LocalIPAddress" value="192.*.*.50"/>
<param name="LocalTrapSendPort" value="161"/>
<param name="GenericTrapType" value="6"/>
<param name="SpecificTrapType" value="12345678"/>
<param name="CommunityString" value="public"/>
<param name="ForwardStackTraceWithTrap" value="true"/>
<param name="Threshold" value="INFO"/>
<param name="ApplicationTrapOID" value="1.3.6.1.4.1.24.12.10.22.64"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d,%p,[%t],[%c],%m%n"/>
</layout>
</appender>
<category name="com.how2share">
<priority value="INFO"/>
<appender-ref ref="TRAP_LOG"/>
</category>
<root>
<appender-ref ref="CONSOLE"/>
<appender-ref ref="FILE"/>
<appender-ref ref="TRAP_LOG"/>
<appender-ref ref="SMTP"/>
</root>
If anyone else is looking for a similar solution, the website to the
http://www.m2technologies.net/asp/snmpTrapAppender.asp
Any input is appreciated, thank you very much for your time!
Dylan Hansen
How2Share Technologies Inc.
PS. A couple of my colleagues that went to JBoss World in Atlanta said
they had the opportunity to meet some of you nice OpenNMS folks.
Hopefully I too will have the pleasure sometime in the future!

[Dylan Hansen]

Hello Gerald, David, and list,

Post by David Hustace
This looks like the problem to me. SNMP traps are sent on port 162.

I tried changing the "LocalTrapSendPort" parameter to 162, and still
had no luck. I find it strange that the default configuration of the
SNMPTrapAppender would have this parameter set to 161, even though
traps should be sent on port 162. Thank you for the pointer though!

Post by David Hustace
Did you have any luck getting SNMPTrapAppender to send traps to OpenNMS?

No, I have not had any success using the SNMPTrapAppender. It is my
understanding that there are 3 implementations of this appender:
JoeSNMPTrapSender, WengsoftSNMPTrapSender and
NetSnmpCommandLineTrapSender. You said that you haven't had any luck
with the NetSnmpCommandLineTrapSender, and I was hesitant to try it
because on the website it says this class is a "proof of concept" and
is not recommended to be used on production systems. After reading
your e-mail I decided to try the WengsoftSNMPTrapSender, but had
nothing but ClassNotFound errors in my JBoss logs. A search for the
missing classes in the source and online turned up nothing. In the
end, I would think that the JoeSNMPTrapSender would be the right
choice, because it was taken from OpenNMS.

I've tried tweaking many things this morning, including:
- Changing the EnterpriseOID and ApplicationTrapOID to different values
- Setting the SpecificTrapType equal to the last value in the
ApplicationTrapOID
- Changing the snmpd.conf file on the client to have snmpd run as the
JBoss user

All of these attempts have come up with nothing. I don't see any
action in OpenNMS, or even when I attempt to send the traps to
snmptrapd on another machine.

I've done some poking around on the Log4j mailing list archives, and
for most of the posts that I see there, it seems like the
SNMPTrapAppender just works for most of the people who have used it.
This is frustrating for me, because I've continually been re-jigging my
log4j.xml file for the past few days and had no luck. I've also made a
post to that mailing list explaining my situation, but haven't received
any replies.

Post by David Hustace
As for your question about how to configure OpenNMS to
collect/report/threshold these traps: Well I've only begun to grok the
OpenNMS source a few weeks ago so I'm still a newbie but intuition tells
me that this is not something OpenNMS will be able to do entirely. The
traps get converted to events, the event could be configured to have a
certain severity which would page you, however this is quite similar to
your Log4j SMTP flood. I don't think that a trap could be collected to
an RRD, though if that were possible some interval were taken into
account (traps per second) then it does seem that the threshold system
could page you at the appropriate time. I could be completely mistaken
though!

Thank you for the explanation! I think you might be correct there. I
sometimes get confused whenever I see an OID and think "hey, you can
get data from that!". I see how OpenNMS acts like an snmpwalk on nodes
and gathers information that way. That is much different from having a
node send a trap to OpenNMS and then act upon that.

Again, if anyone out there has successfully gotten Log4j to send traps
to OpenNMS, I would be interested in hearing from you.

Thank you very much!

Dylan Hansen
How2Share Technologies Inc.

[Gerald Turner]

Post by Dylan Hansen
Hello Gerald, David, and list,

Post by David Hustace
This looks like the problem to me. SNMP traps are sent on port 162.

I tried changing the "LocalTrapSendPort" parameter to 162, and still
had no luck. I find it strange that the default configuration of the
SNMPTrapAppender would have this parameter set to 161, even though
traps should be sent on port 162. Thank you for the pointer though!

I just debugged this thoroughly and found that JoeSNMPTrapSender is
failing to construct an SnmpTrapSession due to "Bind Exception:
Permission Denied". I'm running the JBoss VM as a regular user on unix
without permission to bind to ports <1000. Maybe you're running JBoss
in a similar environment.

Good news is changing the "LocalTrapSendPort" parameter to 0 got it to
work. Upon receiving the first trap OpenNMS did a discovery of my JBoss
server.

[Matt Brozowski]

Post by Gerald Turner

Post by Dylan Hansen
Hello Gerald, David, and list,

Post by David Hustace
This looks like the problem to me. SNMP traps are sent on port 162.

I tried changing the "LocalTrapSendPort" parameter to 162, and still
had no luck. I find it strange that the default configuration of the
SNMPTrapAppender would have this parameter set to 161, even though
traps should be sent on port 162. Thank you for the pointer though!

I just debugged this thoroughly and found that JoeSNMPTrapSender is
Permission Denied". I'm running the JBoss VM as a regular user on unix
without permission to bind to ports <1000. Maybe you're running JBoss
in a similar environment.
Good news is changing the "LocalTrapSendPort" parameter to 0 got it to
work. Upon receiving the first trap OpenNMS did a discovery of my JBoss
server.

For those of you that are using both JBoss and OpenNMS, Mike Jamison of
ZenoData has contributed the beginnings of a JMX Collector. It is
currently in a branch called feature-jmx-collector-branch. It would be
great of some of you could check it out and try out. As soon as we
have it working well I can merge into 1.3.

Thanks!
Matt
________________________________________________________________________
__
Matt Brozowski, OpenNMS Maintainer Main: [phone]
The OpenNMS Group, Inc. Fax: [phone]
Email: ***@opennms.org URL: http://www.opennms.com

[DJ Gregor]

Post by Dylan Hansen
All of these attempts have come up with nothing. I don't see any
action in OpenNMS, or even when I attempt to send the traps to
snmptrapd on another machine.

Traps are sent with a source port on 161, and this traditionally
requires root privileges since the port is below 1024, unless your
system allows otherwise (some modern UNIXes allow this with appropriate
configuration). Is your application running as root?


- djg

[Dylan Hansen]

Post by DJ Gregor
Traps are sent with a source port on 161, and this traditionally
requires root privileges since the port is below 1024, unless your
system allows otherwise (some modern UNIXes allow this with
appropriate configuration). Is your application running as root?
Good news is changing the "LocalTrapSendPort" parameter to 0 got it to
work. Upon receiving the first trap OpenNMS did a discovery of my
JBoss
server.

Thanks to both of you!

Yes, JBoss was running as an unpriveledged user, and therefore, wasn't
able to send traps. I've done as Gerald said and set the
LocalTrapSendPort to "0", and all is well.

Thank you very much for your help!

Dylan Hansen
How2Share Technologies Inc.

[David Hustace]

Post by Dylan Hansen
Greetings everyone,

Greetings, Dylan.

Post by Dylan Hansen
<param name="LocalTrapSendPort" value="161"/>

This looks like the problem to me. SNMP traps are sent on port 162.

-David