[opennms-discuss] Requirement for Parameters defined at the node level for use in collectd-configuration and pollerd-configuration

[JohnD Blackburn]

Due to the number of client databases we are monitoring, we have an ever increasing number of packages in collectd-configuration and pollerd-configuration.

The problem is that for every database we need a different jdbc connection string due to differences in port number and service name.

The ONLY difference between all these packages is the PORT and SERVICE_NAME tags:

<parameter key="url" value="jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=OPENNMS_JDBC_HOSTNAME)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=devdb01)))"/>

There is a variable for the HOST tag, but due to having to hard code the port number and service name, we have had to replicate the database monitoring packages for every database.

Is it possible to variablize the PORT and SERVICE_NAME parameters (or the whole "URL" parameter key), so that we can reuse the same package for monitoring all our databases (or at least significantly reduce the number of required packages)?

Regards,
John Blackburn.

[David Hustace]

Is it possible to variablize the PORT and SERVICE_NAME parameters (or the whole “URL” parameter key), so that we can reuse the same package for monitoring all our databases (or at least significantly reduce the number of required packages)?

John,

This is something that I've wanted to improve in OpenNMS for quite some time. One way to address it would be to storing the attributes determined from the poller package with the monitored service when they're provisioned. That way, we don't require complex package configurations, possibly one for every service, to handle variations on the way we want to monitor a resource with any of the protocol monitors. A simple URL formatted string field stored with the service and a "URL Handler" for each of the ServiceMonitor classes would get us what we need. DevJam is coming up, perhaps if we add this as an enhancement someone will pick it up or possibly some organization will sponsor it. :)


For example:

SELECT ifs.id,
s.servicename,
ip.ipaddr,
ip.iphostname,
ifs.url
FROM ifservices ifs
JOIN service s
ON s.serviceid = ifs.serviceid
JOIN ipinterface ip
ON ip.id = ifs.ipinterfaceid
WHERE ip.id = 9999999;

id | servicename | ipaddr | iphostname | url
---------+-------------+---------------+---------------+------------------------------------------------------------------
353553 | ICMP | 10.100.58.116 | 10.100.58.116 | icmp://10.100.58.116?retries=1&timeout=800
1075355 | HTTPS | 10.100.58.116 | 10.100.58.116 | https://10.100.58.116:443/opennms/alarm/list.htm?display=long
353555 | SNMP | 10.100.58.116 | 10.100.58.116 | snmp://10.100.58.116?retries=3&timeout=800&read-community=public


So, in this case we just create a URL handlers for the IcmpMonitor, HttpsMonitor, and SnmpMonitor classes.


Cheers,
David


David Hustace
The OpenNMS Group, Inc.

[Ronald Roskens]

Post by David Hustace

Is it possible to variablize the PORT and SERVICE_NAME parameters (or the whole “URL” parameter key), so that we can reuse the same package for monitoring all our databases (or at least significantly reduce the number of required packages)?

John,
This is something that I've wanted to improve in OpenNMS for quite some time. One way to address it would be to storing the attributes determined from the poller package with the monitored service when they're provisioned. That way, we don't require complex package configurations, possibly one for every service, to handle variations on the way we want to monitor a resource with any of the protocol monitors. A simple URL formatted string field stored with the service and a "URL Handler" for each of the ServiceMonitor classes would get us what we need. DevJam is coming up, perhaps if we add this as an enhancement someone will pick it up or possibly some organization will sponsor it. :)
SELECT ifs.id,
s.servicename,
ip.ipaddr,
ip.iphostname,
ifs.url
FROM ifservices ifs
JOIN service s
ON s.serviceid = ifs.serviceid
JOIN ipinterface ip
ON ip.id = ifs.ipinterfaceid
WHERE ip.id = 9999999;
id | servicename | ipaddr | iphostname | url
---------+-------------+---------------+---------------+------------------------------------------------------------------
353553 | ICMP | 10.100.58.116 | 10.100.58.116 | icmp://10.100.58.116?retries=1&timeout=800 <icmp://10.100.58.116?retries=1&timeout=800>
1075355 | HTTPS | 10.100.58.116 | 10.100.58.116 | https://10.100.58.116:443/opennms/alarm/list.htm? <https://10.100.58.116/opennms/alarm/list.htm?>display=long
353555 | SNMP | 10.100.58.116 | 10.100.58.116 | snmp://10.100.58.116?retries=3&timeout=800&read-community=public <snmp://10.100.58.116?retries=3&timeout=800&read-community=public>
So, in this case we just create a URL handlers for the IcmpMonitor, HttpsMonitor, and SnmpMonitor classes.

I have to say this would be pretty nice to have. I’m also having to create multiple service definitions each time I spin up a new group of systems just because the port # changes. Having a way to store this at the node / group / category / provisioning requisition would be nice.

I’ve made an entry for this under this years dev-jam page.

https://wiki.opennms.org/wiki/Dev-Jam_2017/ServiceUrls <https://wiki.opennms.org/wiki/Dev-Jam_2017/ServiceUrls>
Ron

[Ronny Trommer]

Hi there,

I have used a few other monitoring tools before and they had the possibility to set test parameters in a monitor or a data collection from from generic to specific. Translated to OpenNMS it would mean you define a generic behaviour in a package and you are able to set test configuration parameters based on a category membership, node membership and last not least on an IP interface. It would be useful to set node and IP interface specific parameter sets during provisioning.

We started something similar in Google Summer of Code 2013 for Pollerd [1]. There where some issues how Pollerd is designed, Christian and Dustin can explain that more in detail. It would make sense to discuss the conceptional goals for pollerd and collectd in the Wiki [2] to build a shared understanding.

[1] https://wiki.opennms.org/wiki/Ideas_for_Google_Summer_of_Code_2013/Key-Value_store_for_service_assurance_with_Pollerd <https://wiki.opennms.org/wiki/Ideas_for_Google_Summer_of_Code_2013/Key-Value_store_for_service_assurance_with_Pollerd>
[2] https://wiki.opennms.org/wiki/DevProjects

tl;dr

Personally I’m not a friend of misusing URLs and overwriting URL handlers. Every implementation has to build own error prone parser which are hard to validate, with JVM wide side effects and we have a key value pairs in our configs already. They are handled well and can be easily persisted, validated and are easy to understood. We have a similar place where URL handlers are used when you implement a Provisioning adapter We had to use to build the VMware importer. Instead of implementing against APIs where your IDE is telling you what you have to do, you have to build custom URL string parser and you have to do some workarounds which ends up in expensive nasty glue code. Most of the time you need additional configuration parameters beside the URL anyways cause they are limited, e.g. authentication.

Post by Ronald Roskens

Post by David Hustace

Is it possible to variablize the PORT and SERVICE_NAME parameters (or the whole “URL” parameter key), so that we can reuse the same package for monitoring all our databases (or at least significantly reduce the number of required packages)?

John,
This is something that I've wanted to improve in OpenNMS for quite some time. One way to address it would be to storing the attributes determined from the poller package with the monitored service when they're provisioned. That way, we don't require complex package configurations, possibly one for every service, to handle variations on the way we want to monitor a resource with any of the protocol monitors. A simple URL formatted string field stored with the service and a "URL Handler" for each of the ServiceMonitor classes would get us what we need. DevJam is coming up, perhaps if we add this as an enhancement someone will pick it up or possibly some organization will sponsor it. :)
SELECT ifs.id,
s.servicename,
ip.ipaddr,
ip.iphostname,
ifs.url
FROM ifservices ifs
JOIN service s
ON s.serviceid = ifs.serviceid
JOIN ipinterface ip
ON ip.id = ifs.ipinterfaceid
WHERE ip.id = 9999999;
id | servicename | ipaddr | iphostname | url
---------+-------------+---------------+---------------+------------------------------------------------------------------
353553 | ICMP | 10.100.58.116 | 10.100.58.116 | icmp://10.100.58.116?retries=1&timeout=800 <icmp://10.100.58.116?retries=1&timeout=800>
1075355 | HTTPS | 10.100.58.116 | 10.100.58.116 | https://10.100.58.116:443/opennms/alarm/list.htm? <https://10.100.58.116/opennms/alarm/list.htm?>display=long
353555 | SNMP | 10.100.58.116 | 10.100.58.116 | snmp://10.100.58.116?retries=3&timeout=800&read-community=public <snmp://10.100.58.116?retries=3&timeout=800&read-community=public>
So, in this case we just create a URL handlers for the IcmpMonitor, HttpsMonitor, and SnmpMonitor classes.

I have to say this would be pretty nice to have. I’m also having to create multiple service definitions each time I spin up a new group of systems just because the port # changes. Having a way to store this at the node / group / category / provisioning requisition would be nice.
I’ve made an entry for this under this years dev-jam page.
https://wiki.opennms.org/wiki/Dev-Jam_2017/ServiceUrls <https://wiki.opennms.org/wiki/Dev-Jam_2017/ServiceUrls>
Ron
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
http://www.opennms.org/index.php/Mailing_List_FAQ
opennms-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

[David Hustace]

Post by Ronny Trommer
Personally I’m not a friend of misusing URLs and overwriting URL handlers. Every implementation has to build own error prone parser which are hard to validate, with JVM wide side effects and we have a key value pairs in our configs already.

I know that, Ronny. It's just the easiest way to explain it. I just didn't want to get into the technical details but we have already improved upon that strategy in Provisiond with the release of H20. I would suggest we would do the same type of thing without having to using the Java official URL handler API.


David Hustace
The OpenNMS Group, Inc.

PS
I knew this was coming from you sooner or later. :)

[Ronny Trommer]

Post by David Hustace

Post by Ronny Trommer
Personally I’m not a friend of misusing URLs and overwriting URL handlers. Every implementation has to build own error prone parser which are hard to validate, with JVM wide side effects and we have a key value pairs in our configs already.

I know that, Ronny. It's just the easiest way to explain it. I just didn't want to get into the technical details but we have already improved upon that strategy in Provisiond with the release of H20. I would suggest we would do the same type of thing without having to using the Java official URL handler API.

If it means you have to configure a connection through something like vmware://user:***@vmware-center/foobar?importVMware=true <vmware://user:***@vmware-center/foobar?importVMware=true> using an in-official homebrew URL handler API does not solve the problems I’ve described beside the JVM wide side effect.

Just following the German paradigm something smells fishy to me.

- complain early, complain often :D