Discussion:
[opennms-discuss] OpenNMS broken after 20.0.0 --> 20.0.1, won't start
Reinhard Kuss
2017-07-11 18:47:24 UTC
Permalink
Hi folks,

did an update recently via debian-repo (debian.opennms.org) to 20.0.1,
now ONMS won't start.

Physical environment, debian jessie x64, Oracle-Java JRE 1.8.0_131,
output.log shows

Caused by: java.lang.StackOverflowError
at java.lang.Throwable.getCause(Throwable.java:415)
at
org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:187)
at
org.opennms.netmgt.model.OnmsNode_$$_jvst50a_33.getId(OnmsNode_$$_jvst50a_33.java)
at
org.opennms.netmgt.model.topology.SharedSegment.getBridgeIdsOnSegment(SharedSegment.java:245)
at
org.opennms.netmgt.model.topology.BroadcastDomain.getSharedSegmentOnTopologyForBridge(BroadcastDomain.java:297)
at
org.opennms.netmgt.model.topology.BroadcastDomain.loadTopologyRoot(BroadcastDomain.java:240)
at
org.opennms.netmgt.model.topology.BroadcastDomain.loadTopologyRoot(BroadcastDomain.java:242)
at
org.opennms.netmgt.model.topology.BroadcastDomain.loadTopologyRoot(BroadcastDomain.java:242)
at
org.opennms.netmgt.model.topology.BroadcastDomain.loadTopologyRoot(BroadcastDomain.java:242)

[...last line repeating hundreds of lines...]

at
org.opennms.netmgt.model.topology.BroadcastDomain.loadTopologyRoot(BroadcastDomain.java:242)

and at last:

at
org.opennms.netmgt.model.topology.BroadcastDomain.loadTopologyRoot(BroadcastDomain.java:242)
at
org.opennms.netmgt.model.topology.BroadcastDomain.loadTopologyRoot(BroadcastDomain.java:242)
2017-07-10 21:10:07,256 ActiveMQ BrokerService[localhost] Task-2 WARN
Unable to register Log4j shutdown hook because JVM is shutting down.
Using SimpleLogger


I've already tried to disable enlinkd in service-configuration.xml:

<name>OpenNMS:Name=EnhancedLinkd</name>
<class-name>org.opennms.netmgt.enlinkd.jmx.EnhancedLinkd</class-name>
<invoke at="start" pass="0" method="init"/>
<invoke at="start" pass="0" method="start"/>
<invoke at="status" pass="0" method="status"/>
<invoke at="stop" pass="0" method="stop"/>

No luck, enlinkd starts despite pass="0" (probably missed something
here...)


JVM-Parameters:

# heap was 8192, tried to increase, no luck
JAVA_HEAP_SIZE=16384
ADDITIONAL_MANAGER_OPTIONS="${ADDITIONAL_MANAGER_OPTIONS} -XX:+UseG1GC"


Have 8GB RAM, 22GB swap...


Other relevant parts from output.log are:

An error occurred while attempting to start the
"OpenNMS:Name=EnhancedLinkd" service (class
org.opennms.netmgt.enlinkd.jmx.EnhancedLinkd). Shutting down and exiting.
javax.management.RuntimeMBeanException:
org.springframework.beans.factory.access.BootstrapException: Unable to
return specified BeanFactory instance: factory key [enhancedLinkdContext],
from group with resource name [classpath*:beanRefContext.xml]; nested
exception is org.springframework.beans.factory.BeanCreationException:
Error creating bean with name 'enhancedLinkdContext' defined in URL
[jar:file:/usr/share/opennms/lib/opennms-services-20.0.1.jar!/beanRefContext.xml]:
Bean instantiation
via constructor failed; nested exception is
org.springframework.beans.BeanInstantiationException: Failed to
instantiate [org.springframework.context.support
.ClassPathXmlApplicationContext]: Constructor threw exception; nested
exception is org.springframework.beans.factory.BeanCreationException:
Error creating bean with name
'daemon' defined in class path resource
[META-INF/opennms/applicationContext-enhancedLinkd.xml]: Invocation of
init method failed; nested exception is java.lang.StackOverflowError
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:821)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at org.opennms.netmgt.vmmgr.Invoker.invoke(Invoker.java:277)
at org.opennms.netmgt.vmmgr.Invoker.invokeMethods(Invoker.java:206)
at org.opennms.netmgt.vmmgr.Starter.start(Starter.java:154)
at org.opennms.netmgt.vmmgr.Starter.startDaemon(Starter.java:94)
at org.opennms.netmgt.vmmgr.Controller.start(Controller.java:173)
at org.opennms.netmgt.vmmgr.Controller.main(Controller.java:150)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.opennms.bootstrap.Bootstrap$4.run(Bootstrap.java:525)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.beans.factory.access.BootstrapException:
Unable to return specified BeanFactory instance: factory key
[enhancedLinkdContext], from group with
resource name [classpath*:beanRefContext.xml]; nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name
'enhancedLinkdContext' defined in URL
[jar:file:/usr/share/opennms/lib/opennms-services-20.0.1.jar!/beanRefContext.xml]:
Bean instantiation via constructor failed;
nested exception is
org.springframework.beans.BeanInstantiationException: Failed to
instantiate
[org.springframework.context.support.ClassPathXmlApplicationContext]:
Constructor threw exception; nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'daemon' defined in class
path resource [META-INF/opennms/applicationContext-enhancedLinkd.xml]:
Invocation of init method failed; nested exception is
java.lang.StackOverflowError
at
org.springframework.beans.factory.access.SingletonBeanFactoryLocator.useBeanFactory(SingletonBeanFactoryLocator.java:404)
at
org.opennms.core.spring.BeanUtils.getBeanFactory(BeanUtils.java:80)
at org.opennms.core.spring.BeanUtils.getFactory(BeanUtils.java:132)
at
org.opennms.netmgt.daemon.AbstractSpringContextJmxServiceDaemon$1.run(AbstractSpringContextJmxServiceDaemon.java:103)
at org.opennms.core.logging.Logging.withPrefix(Logging.java:71)
at
org.opennms.netmgt.daemon.AbstractSpringContextJmxServiceDaemon.init(AbstractSpringContextJmxServiceDaemon.java:96)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)


And finally enlinkd.log:

2017-07-10 21:10:05,049 DEBUG [Main] o.o.c.s.BeanUtils:
m_locationAwareSnmpClient is not null
2017-07-10 21:10:05,050 INFO [Main] o.o.n.e.EnhancedLinkd: init:
Loading nodes.....
2017-07-10 21:10:05,164 DEBUG [Main] o.o.n.d.h.AbstractDaoHibernate:
criteria = Criteria [class=class
org.opennms.netmgt.model.OnmsNode,aliases=[Alias [association
Path=ipInterfaces, alias=iface, type=LEFT_JOIN,
joinCondition=null]],restrictions=[EqRestriction [attribute=type,
value=A], EqRestriction [attribute=iface.isSnmpPr
imary, value=P]],distinct=false]
2017-07-10 21:10:05,584 INFO [Main] o.o.n.e.EnhancedLinkd: init: Nodes
loaded.
2017-07-10 21:10:05,584 INFO [Main] o.o.n.e.EnhancedLinkd: init:
Loading Bridge Topology.....
2017-07-10 21:10:05,745 WARN [Main]
o.s.c.s.ClassPathXmlApplicationContext: Exception encountered during
context initialization - cancelling refresh attempt
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'daemon' defined in class path resource
[META-INF/opennms/applicationContext
-enhancedLinkd.xml]: Invocation of init method failed; nested exception
is java.lang.StackOverflowError
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1574)
~[org.apache.s
ervicemix.bundles.spring-beans-4.1.6.RELEASE_1.jar:?]
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)
~[org.apache.serv
icemix.bundles.spring-beans-4.1.6.RELEASE_1.jar:?]
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
~[org.apache.servic
emix.bundles.spring-beans-4.1.6.RELEASE_1.jar:?]
at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)
~[org.apache.servicemix.bundles.spring-beans-4.1
.6.RELEASE_1.jar:?]
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
~[org.apache.servicemix.bundl
es.spring-beans-4.1.6.RELEASE_1.jar:?]
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)
~[org.apache.servicemix.bundles.spring-beans-4.1.6
.RELEASE_1.jar:?]
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
~[org.apache.servicemix.bundles.spring-beans-4.1.6.R
ELEASE_1.jar:?]
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:755)
~[org.apache.servicem
ix.bundles.spring-beans-4.1.6.RELEASE_1.jar:?]
at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757)
~[org.apache.service
mix.bundles.spring-context-4.1.6.RELEASE_1.jar:?]
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)
[org.apache.servicemix.bundles.spring-contex
t-4.1.6.RELEASE_1.jar:?]
at
org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
[org.apache.servicemix.bundles.spring
-context-4.1.6.RELEASE_1.jar:?]
at
org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:105)
[org.apache.servicemix.bundles.spring
-context-4.1.6.RELEASE_1.jar:?]
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
~[?:1.8.0_131]
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[?:1.8.0_131]
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[?:1.8.0_131]
at
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
[?:1.8.0_131]
at
org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:147)
[org.apache.servicemix.bundles.spring-beans-4.1.6.RELEASE_1.jar:?]
at
org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:122)
[org.apache.servicemix.bundles.s
pring-beans-4.1.6.RELEASE_1.jar:?]
at
org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:267)
[org.apache.servicemix.bundles.spring-be




I'm stuck here, pls could someone gve a hint..?




Regards
Reinhard
--
Reinhard Kuss, office <at> edv-profi24 <dot> at


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Alejandro Galue
2017-07-11 19:33:49 UTC
Permalink
Hi,
Post by Reinhard Kuss
<name>OpenNMS:Name=EnhancedLinkd</name>
<class-name>org.opennms.netmgt.enlinkd.jmx.EnhancedLinkd</class-name>
<invoke at="start" pass="0" method="init"/>
<invoke at="start" pass="0" method="start"/>
<invoke at="status" pass="0" method="status"/>
<invoke at="stop" pass="0" method="stop"/>
No luck, enlinkd starts despite pass="0" (probably missed something here
)
To successfully disable it, it should look like this:

<service enabled="false">
<name>OpenNMS:Name=EnhancedLinkd</name>
<class-name>org.opennms.netmgt.enlinkd.jmx.EnhancedLinkd</class-name>
<invoke method="init" pass="0" at="start"/>
<invoke method="start" pass="1" at="start"/>
<invoke method="status" pass="0" at="status"/>
<invoke method="stop" pass="0" at="stop"/>
</service>

If you run the following, it should not be listed:

/opt/opennms/bin/opennms -v status
Post by Reinhard Kuss
# heap was 8192, tried to increase, no luck
JAVA_HEAP_SIZE=16384
ADDITIONAL_MANAGER_OPTIONS="${ADDITIONAL_MANAGER_OPTIONS} -XX:+UseG1GC"
Have 8GB RAM, 22GB swap...
The above doesn’t sound right.

If I were you, I would to turn off swap and add more RAM to the machine.

Also,i f you have 8GB of RAM, you should never assign more than that to the heap size (that alone should explain the problem). In fact, if PostgreSQL is also running on the same machine, you should not give it more than 4GB (assuming postgresql.conf is properly tuned).

As a matter of fact, I would put PostgreSQL on its own machine.

Alejandro
Reinhard Kuss
2017-07-11 20:41:53 UTC
Permalink
Hi Alejandro,

success!

After disabling service enlinkd ONMS starts successful, thanks for the hint.

There is another post on the list reporting exactly the same behaviour,
Norbert posted the full log. Have compared it with my logfiles, they are
almost equal.

Adjusted heap size, too (6GB).
Post by Reinhard Kuss
<name>OpenNMS:Name=EnhancedLinkd</name>
<class-name>org.opennms.netmgt.enlinkd.jmx.EnhancedLinkd</class-name>
<invoke at="start" pass="0" method="init"/>
<invoke at="start" pass="0" method="start"/>
<invoke at="status" pass="0" method="status"/>
<invoke at="stop" pass="0" method="stop"/>
No luck, enlinkd starts despite pass="0" (probably missed something
here…)
<service *enabled="false"*>
<name>OpenNMS:Name=EnhancedLinkd</name>
<class-name>org.opennms.netmgt.enlinkd.jmx.EnhancedLinkd</class-name>
<invoke method="init" pass="0" at="start"/>
<invoke method="start" pass="1" at="start"/>
<invoke method="status" pass="0" at="status"/>
<invoke method="stop" pass="0" at="stop"/>
</service>
Regards
Reinhard
--
Reinhard Kuss, office <at> edv-profi24 <dot> at


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourcefor
Loading...