Hey,
We have been using Hyperic extensively for monitoring more than 150 platforms for the past 3-4 months. We have recently upgraded the version from 4.4 to 4.6.6.
The issue is with the Hyperic Agent Startup and it is coming intermittently on few of the VM's. Steps to reproduce this:
1) Stop the Hyperic Agent
2) Start the Hyperic Agent
What I see on the console is:
Error:Error contacting agent: Error sending argument: Unable to connect to 127.0.0.1:2144: Connection refused, retried 5 times, cmd=agent
ing
The logs in log/agent,log:
19-02-2013 13:12:01,000 UTC INFO [AgentDaemonMain] [AgentDaemon@1050] Attempting to rollback agent bundle
19-02-2013 13:12:01,001 UTC INFO [AgentDaemonMain] [AgentDaemon@1059] Rollback of agent bundle was successful
19-02-2013 13:12:01,001 UTC INFO [AgentDaemonMain] [AgentDaemon@1064] Restarting JVM...
19-02-2013 13:12:01,002 UTC INFO [AgentDaemonMain] [AgentUpgradeManager@67] restart requested
19-02-2013 13:12:20,748 UTC ERROR [AgentDaemonMain] [AgentDaemon@1004] Agent configuration failed:
org.jasypt.exceptions.EncryptionOperationNotPossibleException
at org.jasypt.encryption.pbe.StandardPBEStringEncryptor.decrypt(StandardPBEStringEncryptor.java:653)
at org.hyperic.util.security.MarkedStringEncryptor.decrypt(MarkedStringEncryptor.java:74)
at org.jasypt.properties.PropertyValueEncryptionUtils.decrypt(PropertyValueEncryptionUtils.java:72)
at org.hyperic.util.security.SecurityUtil.decrypt(SecurityUtil.java:101)
at org.hyperic.hq.agent.server.AgentDListProvider.init(AgentDListProvider.java:371)
at org.hyperic.hq.agent.server.AgentDaemon.createStorageProvider(AgentDaemon.java:384)
at org.hyperic.hq.agent.server.AgentDaemon.configure(AgentDaemon.java:430)
at org.hyperic.hq.agent.server.AgentDaemon.newInstance(AgentDaemon.java:184)
at org.hyperic.hq.agent.server.AgentDaemon$RunnableAgent.run(AgentDaemon.java:1001)
at java.lang.Thread.run(Thread.java:662)
While the logs in log/wrapper.log are as follows:
STATUS | wrapper | 2013/02/19 13:11:21 | JVM requested a restart.
INFO | jvm 1 | 2013/02/19 13:11:22 | Stopping agent ...
INFO | jvm 1 | 2013/02/19 13:11:34 | Success -- agent is stopped!
STATUS | wrapper | 2013/02/19 13:11:39 | Reloading Wrapper configuration...
STATUS | wrapper | 2013/02/19 13:11:39 | Launching a JVM...
INFO | jvm 2 | 2013/02/19 13:11:39 | WrapperManager: Initializing...
INFO | jvm 2 | 2013/02/19 13:11:41 | - Invoking agent
INFO | jvm 2 | 2013/02/19 13:11:41 | - Agent thread running
INFO | jvm 2 | 2013/02/19 13:11:41 | - Verifying if agent is running...
INFO | jvm 2 | 2013/02/19 13:11:41 | Error invoking agent: Agent reported an error while starting up
STATUS | wrapper | 2013/02/19 13:11:41 | JVM requested a restart.
INFO | jvm 2 | 2013/02/19 13:11:42 | Stopping agent ...
INFO | jvm 2 | 2013/02/19 13:11:54 | Success -- agent is stopped!
STATUS | wrapper | 2013/02/19 13:11:59 | Reloading Wrapper configuration...
STATUS | wrapper | 2013/02/19 13:11:59 | Launching a JVM...
INFO | jvm 3 | 2013/02/19 13:11:59 | WrapperManager: Initializing...
INFO | jvm 3 | 2013/02/19 13:12:00 | - Invoking agent
INFO | jvm 3 | 2013/02/19 13:12:00 | - Agent thread running
INFO | jvm 3 | 2013/02/19 13:12:00 | - Verifying if agent is running...
INFO | jvm 3 | 2013/02/19 13:12:01 | Error invoking agent: Agent reported an error while starting up
STATUS | wrapper | 2013/02/19 13:12:01 | JVM requested a restart.
INFO | jvm 3 | 2013/02/19 13:12:02 | Stopping agent ...
INFO | jvm 3 | 2013/02/19 13:12:14 | Success -- agent is stopped!
STATUS | wrapper | 2013/02/19 13:12:19 | Reloading Wrapper configuration...
STATUS | wrapper | 2013/02/19 13:12:19 | Launching a JVM...
INFO | jvm 4 | 2013/02/19 13:12:19 | WrapperManager: Initializing...
INFO | jvm 4 | 2013/02/19 13:12:20 | - Invoking agent
INFO | jvm 4 | 2013/02/19 13:12:20 | - Agent thread running
INFO | jvm 4 | 2013/02/19 13:12:20 | - Verifying if agent is running...
INFO | jvm 4 | 2013/02/19 13:12:20 | Error invoking agent: Agent reported an error while starting up
STATUS | wrapper | 2013/02/19 13:12:20 | JVM requested a restart.
INFO | jvm 4 | 2013/02/19 13:12:21 | Stopping agent ...
INFO | jvm 4 | 2013/02/19 13:12:33 | Success -- agent is stopped!
STATUS | wrapper | 2013/02/19 13:12:38 | Reloading Wrapper configuration...
STATUS | wrapper | 2013/02/19 13:12:38 | Launching a JVM...
INFO | jvm 5 | 2013/02/19 13:12:39 | WrapperManager: Initializing...
INFO | jvm 5 | 2013/02/19 13:12:40 | - Invoking agent
INFO | jvm 5 | 2013/02/19 13:12:40 | - Agent thread running
INFO | jvm 5 | 2013/02/19 13:12:40 | - Verifying if agent is running...
INFO | jvm 5 | 2013/02/19 13:12:40 | Error invoking agent: Agent reported an error while starting up
INFO | jvm 5 | 2013/02/19 13:12:41 | Stopping agent ...
INFO | jvm 5 | 2013/02/19 13:12:53 | Success -- agent is stopped!
ERROR | wrapper | 2013/02/19 13:13:14 | Shutdown failed: Timed out waiting for signal from JVM.
ERROR | wrapper | 2013/02/19 13:13:14 | JVM did not exit on request, terminated
STATUS | wrapper | 2013/02/19 13:13:15 | JVM received a signal SIGKILL (9).
STATUS | wrapper | 2013/02/19 13:13:15 | on_exit trigger matched. Restarting the JVM. (Exit code: 1)
FATAL | wrapper | 2013/02/19 13:13:15 | There were 5 failed launches in a row, each lasting less than 60 seconds. Giving up.
FATAL | wrapper | 2013/02/19 13:13:15 | There may be a configuration problem:please check the logs.
STATUS | wrapper | 2013/02/19 13:13:15 | <-- Wrapper Stopped
Any help on this would be really appreciated.
Regards,
Yashika