I can't explain this in any short way so my apologies!
We run WAS 6 and 6.1 in a Network Deployment (i.e. managed by ndgmr)
Two identical nodes suddenly behave totally differently with one node failing to get metrics with errors in the agent log like:
2010-03-31 08:11:00,743 ERROR [ScheduleThread] [WebsphereCollector] ADMC0053E: The system cannot create a SOAP connector to connect to host pndmgrnode01 at port 8879 with SOAP connector security enabled.
org.hyperic.hq.product.MetricUnreachableException: ADMC0053E: The system cannot create a SOAP connector to connect to host pndmgrnode01 at port 8879 with SOAP connector security enabled.
at org.hyperic.hq.plugin.websphere.WebsphereUtil.getMBeanServer(WebsphereUtil.java:120)
at org.hyperic.hq.plugin.websphere.WebsphereCollector.getMBeanServer(WebsphereCollector.java:120)
at org.hyperic.hq.plugin.websphere.WebsphereCollector.init(WebsphereCollector.java:86)
at org.hyperic.hq.product.Collector.getValue(Collector.java:512)
at org.hyperic.hq.product.MeasurementPlugin.getValue(MeasurementPlugin.java:445)
at org.hyperic.hq.plugin.websphere.WebsphereMeasurementPlugin.getValue(WebsphereMeasurementPlugin.java:51)
at org.hyperic.hq.product.MeasurementPluginManager.getPluginValue(MeasurementPluginManager.java:176)
at org.hyperic.hq.product.MeasurementPluginManager.getValue(MeasurementPluginManager.java:274)
at org.hyperic.hq.measurement.agent.server.ScheduleThread.getValue(ScheduleThread.java:298)
at org.hyperic.hq.measurement.agent.server.ScheduleThread.collect(ScheduleThread.java:387)
at org.hyperic.hq.measurement.agent.server.ScheduleThread.collect(ScheduleThread.java:344)
at org.hyperic.hq.measurement.agent.server.ScheduleThread.collect(ScheduleThread.java:490)
at org.hyperic.hq.measurement.agent.server.ScheduleThread.run(ScheduleThread.java:512)
at java.lang.Thread.run(Thread.java:810)
Caused by:
com.ibm.websphere.management.exception.ConnectorException: ADMC0053E: The system cannot create a SOAP connector to connect to host pndmgrnode01 at port 8879 with SOAP connector security enabled.
at com.ibm.websphere.management.AdminClientFactory.createAdminClient(AdminClientFactory.java:414)
at org.hyperic.hq.plugin.websphere.WebsphereUtil.getMBeanServer(WebsphereUtil.java:118)
... 13 more
Caused by:
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor8.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:522)
at com.ibm.websphere.management.AdminClientFactory.createAdminClient(AdminClientFactory.java:305)
... 14 more
Caused by:
com.ibm.websphere.management.exception.ConnectorNotAvailableException: [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.h: PKIX path building failed: java.security.cert.CertPathBuilderException: unable to find valid certification path to requested target; targetException=java.lang.IllegalArgumentException: Error opening socket: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.h: PKIX path building failed: java.security.cert.CertPathBuilderException: unable to find valid certification path to requested target]
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.reconnect(SOAPConnectorClient.java:295)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.<init>(SOAPConnectorClient.java:190)
... 18 more
Caused by:
[SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.h: PKIX path building failed: java.security.cert.CertPathBuilderException: unable to find valid certification path to requested target; targetException=java.lang.IllegalArgumentException: Error opening socket: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.h: PKIX path building failed: java.security.cert.CertPathBuilderException: unable to find valid certification path to requested target]
at org.apache.soap.transport.http.SOAPHTTPConnection.send(Unknown Source)
at org.apache.soap.rpc.Call.invoke(Unknown Source)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient$2.run(SOAPConnectorClient.java:266)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.reconnect(SOAPConnectorClient.java:259)
... 19 more
Repeated for every metric it is trying to collect.
The setup is that the dmgr is WAS6.1
It manages a cluster of which 12 nodes have a single WAS6.0 node whereas the 2 nodes in question have both a 6.0 and a 6.1 node.
Between the two identical nodes, the keystores are identical, the soap.client.props files are identical, the ssl.client.props files and the agent.properties files are identical
Yet one works and the other doesnt! They both worked for a long time (6 months plus) until last week when they were restarted. As nothing differs between the nodes or in their configuration in hyperic..... I am totally stumped!
The errors suggest it is a SOAP security issue, yet if I execute wsadmin.sh from the problem node (which gets a SOAP connection to the DMGR node just as the agent would do) it works without problem. I can also run synchNode.sh from the node and sync with the dmgr.
Soooooo where the hell to look next :)
- One question I have which may or may not be related is the agent.properties file has the following set
websphere.installpath=/opt/IBM/WebSphere/AppServer
However in this example of course we have 2 versions of websphere installed. Should I need to specify both paths somehow?
websphere.installpath=/opt/IBM/WebSphere/AppServer
and
websphere.installpath=/opt/IBM/WebSphere/AppServer61
It can' be the cause if the other node works though right?
We run WAS 6 and 6.1 in a Network Deployment (i.e. managed by ndgmr)
Two identical nodes suddenly behave totally differently with one node failing to get metrics with errors in the agent log like:
2010-03-31 08:11:00,743 ERROR [ScheduleThread] [WebsphereCollector] ADMC0053E: The system cannot create a SOAP connector to connect to host pndmgrnode01 at port 8879 with SOAP connector security enabled.
org.hyperic.hq.product.MetricUnreachableException: ADMC0053E: The system cannot create a SOAP connector to connect to host pndmgrnode01 at port 8879 with SOAP connector security enabled.
at org.hyperic.hq.plugin.websphere.WebsphereUtil.getMBeanServer(WebsphereUtil.java:120)
at org.hyperic.hq.plugin.websphere.WebsphereCollector.getMBeanServer(WebsphereCollector.java:120)
at org.hyperic.hq.plugin.websphere.WebsphereCollector.init(WebsphereCollector.java:86)
at org.hyperic.hq.product.Collector.getValue(Collector.java:512)
at org.hyperic.hq.product.MeasurementPlugin.getValue(MeasurementPlugin.java:445)
at org.hyperic.hq.plugin.websphere.WebsphereMeasurementPlugin.getValue(WebsphereMeasurementPlugin.java:51)
at org.hyperic.hq.product.MeasurementPluginManager.getPluginValue(MeasurementPluginManager.java:176)
at org.hyperic.hq.product.MeasurementPluginManager.getValue(MeasurementPluginManager.java:274)
at org.hyperic.hq.measurement.agent.server.ScheduleThread.getValue(ScheduleThread.java:298)
at org.hyperic.hq.measurement.agent.server.ScheduleThread.collect(ScheduleThread.java:387)
at org.hyperic.hq.measurement.agent.server.ScheduleThread.collect(ScheduleThread.java:344)
at org.hyperic.hq.measurement.agent.server.ScheduleThread.collect(ScheduleThread.java:490)
at org.hyperic.hq.measurement.agent.server.ScheduleThread.run(ScheduleThread.java:512)
at java.lang.Thread.run(Thread.java:810)
Caused by:
com.ibm.websphere.management.exception.ConnectorException: ADMC0053E: The system cannot create a SOAP connector to connect to host pndmgrnode01 at port 8879 with SOAP connector security enabled.
at com.ibm.websphere.management.AdminClientFactory.createAdminClient(AdminClientFactory.java:414)
at org.hyperic.hq.plugin.websphere.WebsphereUtil.getMBeanServer(WebsphereUtil.java:118)
... 13 more
Caused by:
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor8.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:522)
at com.ibm.websphere.management.AdminClientFactory.createAdminClient(AdminClientFactory.java:305)
... 14 more
Caused by:
com.ibm.websphere.management.exception.ConnectorNotAvailableException: [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.h: PKIX path building failed: java.security.cert.CertPathBuilderException: unable to find valid certification path to requested target; targetException=java.lang.IllegalArgumentException: Error opening socket: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.h: PKIX path building failed: java.security.cert.CertPathBuilderException: unable to find valid certification path to requested target]
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.reconnect(SOAPConnectorClient.java:295)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.<init>(SOAPConnectorClient.java:190)
... 18 more
Caused by:
[SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.h: PKIX path building failed: java.security.cert.CertPathBuilderException: unable to find valid certification path to requested target; targetException=java.lang.IllegalArgumentException: Error opening socket: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.h: PKIX path building failed: java.security.cert.CertPathBuilderException: unable to find valid certification path to requested target]
at org.apache.soap.transport.http.SOAPHTTPConnection.send(Unknown Source)
at org.apache.soap.rpc.Call.invoke(Unknown Source)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient$2.run(SOAPConnectorClient.java:266)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.reconnect(SOAPConnectorClient.java:259)
... 19 more
Repeated for every metric it is trying to collect.
The setup is that the dmgr is WAS6.1
It manages a cluster of which 12 nodes have a single WAS6.0 node whereas the 2 nodes in question have both a 6.0 and a 6.1 node.
Between the two identical nodes, the keystores are identical, the soap.client.props files are identical, the ssl.client.props files and the agent.properties files are identical
Yet one works and the other doesnt! They both worked for a long time (6 months plus) until last week when they were restarted. As nothing differs between the nodes or in their configuration in hyperic..... I am totally stumped!
The errors suggest it is a SOAP security issue, yet if I execute wsadmin.sh from the problem node (which gets a SOAP connection to the DMGR node just as the agent would do) it works without problem. I can also run synchNode.sh from the node and sync with the dmgr.
Soooooo where the hell to look next :)
- One question I have which may or may not be related is the agent.properties file has the following set
websphere.installpath=/opt/IBM/WebSphere/AppServer
However in this example of course we have 2 versions of websphere installed. Should I need to specify both paths somehow?
websphere.installpath=/opt/IBM/WebSphere/AppServer
and
websphere.installpath=/opt/IBM/WebSphere/AppServer61
It can' be the cause if the other node works though right?