Quantcast
Channel: VMware Communities : Popular Discussions - Hyperic User Forums
Viewing all articles
Browse latest Browse all 52618

jboss-4.2.3.GA agent disconnect

$
0
0
This puzzle is past my understanding at the moment.

OS is RHEL 5.4 Server,  official version, not Centos.
JBoss 4.2.3.GA is running   Agent is installed. 
JMX monitoring is installed but JBoss isn't exporting JMX yet so it isn't connecting.

Monitoring is set on JBoss itself.  Just the simplest thing.  Send me an e-mail.

JBoss is bounced several times in the past 3 weeks, no problems. 
Yesterday it was bounced at 18:10. 
The agent gripes a fair bit but seemed to try to reconnect, then

Silence of the logs.  Until I restart the agent almost 24 hours later.

However, the monitor at the HQ server keeps on indicating all is well until about 0040 the next day.  THEN it sends out an e-mail to say that it is down and that is when the HQ server says JBoss is down, went down, and indicates that the agent is unavailable.  

JBoss however, is up and has been up since 18:12 or there-about. 

Agent leaves no breadcrumbs in the log at all.  When checked is apparently still running (Containing JVM is anyway).   

No indications in wrapper log either. 

So what happened here.  Agent was not recording in the log at all.  Hyperic HQ was unaware of the problem until something triggered the next day.  ? 

agent.log looks  like this (leaving out a heap of normalcy at the beginning).

2010-05-10 11:45:10,645 WARN  [ScheduleThread] [ScheduleThread]
Collection of metric: 'JBoss 4.2:jboss.system:
type=ServerInfo:ActiveThreadCount:java.naming.provider.url=jnp%3A//0.0.0.0%3A1099,java.naming.security.princip
al=%java.naming.security.principal%,java.naming.security.credentials='
took: 10629ms
2010-05-10 14:05:08,159 WARN  [ScheduleThread] [ScheduleThread]
Collection of metric: 'JBoss 4.2:jboss.system:
type=ServerInfo:ActiveThreadCount:java.naming.provider.url=jnp%3A//0.0.0.0%3A1099,java.naming.security.princip
al=%java.naming.security.principal%,java.naming.security.credentials='
took: 8144ms
2010-05-10 14:10:09,903 WARN  [ScheduleThread] [ScheduleThread]
Collection of metric: 'JBoss 4.2:jboss.system:
type=ServerInfo:ActiveThreadCount:java.naming.provider.url=jnp%3A//0.0.0.0%3A1099,java.naming.security.princip
al=%java.naming.security.principal%,java.naming.security.credentials='
took: 9876ms
2010-05-10 14:20:08,858 WARN  [ScheduleThread] [ScheduleThread]
Collection of metric: 'JBoss 4.2:jboss.system:
type=ServerInfo:ActiveThreadCount:java.naming.provider.url=jnp%3A//0.0.0.0%3A1099,java.naming.security.princip
al=%java.naming.security.principal%,java.naming.security.credentials='
took: 8824ms
2010-05-10 16:23:58,241 INFO  [Thread-2] [RuntimeAutodiscoverer] Running
runtime autodiscovery for NetworkServ
er
2010-05-10 16:23:58,434 INFO  [Thread-2] [RuntimeAutodiscoverer]
NetworkServer discovery took 0
2010-05-10 16:23:58,435 INFO  [Thread-2] [RuntimeAutodiscoverer] Running
runtime autodiscovery for HQ Agent
2010-05-10 16:23:58,622 INFO  [Thread-2] [RuntimeAutodiscoverer] HQ
Agent discovery took 0
2010-05-10 16:23:58,622 INFO  [Thread-2] [RuntimeAutodiscoverer] Running
runtime autodiscovery for ProcessServ
er
2010-05-10 16:23:58,823 INFO  [Thread-2] [RuntimeAutodiscoverer]
ProcessServer discovery took 0
2010-05-10 16:23:58,823 INFO  [Thread-2] [RuntimeAutodiscoverer] Running
runtime autodiscovery for Tomcat 6.0
2010-05-10 16:24:07,476 INFO  [Thread-2] [RuntimeAutodiscoverer] Tomcat
6.0 discovery took 8.43
2010-05-10 16:24:07,476 INFO  [Thread-2] [RuntimeAutodiscoverer] Running
runtime autodiscovery for NTP 4.x
2010-05-10 16:24:07,689 INFO  [Thread-2] [RuntimeAutodiscoverer] NTP 4.x
discovery took 0
2010-05-10 16:24:07,689 INFO  [Thread-2] [RuntimeAutodiscoverer] Running
runtime autodiscovery for FileServer
2010-05-10 16:24:07,902 INFO  [Thread-2] [RuntimeAutodiscoverer]
FileServer discovery took 0
2010-05-10 16:24:07,902 INFO  [Thread-2] [RuntimeAutodiscoverer] Running
runtime autodiscovery for JBoss 4.2
2010-05-10 16:24:08,262 INFO  [Thread-2] [RuntimeAutodiscoverer] JBoss
4.2 discovery took 0.16
2010-05-10 16:24:08,262 INFO  [Thread-2] [RuntimeAutodiscoverer] Running
runtime autodiscovery for Net Service
s
2010-05-10 16:24:08,465 INFO  [Thread-2] [RuntimeAutodiscoverer] Net
Services discovery took 0
2010-05-10 16:25:08,054 WARN  [ScheduleThread] [ScheduleThread]
Collection of metric: 'JBoss 4.2:jboss.system:
type=ServerInfo:ActiveThreadCount:java.naming.provider.url=jnp%3A//0.0.0.0%3A1099,java.naming.security.princip
al=%java.naming.security.principal%,java.naming.security.credentials='
took: 8039ms
2010-05-10 18:05:00,144 ERROR [ScheduleThread] [ScheduleThread] Metric
unreachable: Can't connect to MBeanServ
er
[java.naming.provider.url=jnp%3A//0.0.0.0%3A1099,java.naming.security.principal=%java.naming.security.princ
ipal%,java.naming.security.credentials=%java.naming.security.credentials%]:
javax.naming.NameNotFoundException
: jmx not bound
2010-05-10 18:05:00,144 WARN  [ScheduleThread] [ScheduleThread]
Disabling metrics for: 2:10024
2010-05-10 18:05:00,154 ERROR [ScheduleThread] [ScheduleThread] Metric
unreachable: Error connecting with moni
tored server, make sure it is started and check http://0.0.0.0:34010/?*
Message=Connection refused
2010-05-10 18:05:00,154 WARN  [ScheduleThread] [ScheduleThread]
Disabling metrics for: 3:10133
2010-05-10 18:05:00,156 ERROR [ScheduleThread] [ScheduleThread] Metric
unreachable: Error connecting with moni
tored server, make sure it is started and check http://0.0.0.0:34010/?*
Message=Connection refused
2010-05-10 18:05:00,156 WARN  [ScheduleThread] [ScheduleThread]
Disabling metrics for: 3:10132
2010-05-10 18:05:00,158 ERROR [ScheduleThread] [ScheduleThread] Metric
unreachable: Error connecting with moni
tored server, make sure it is started and check http://0.0.0.0:34010/?*
Message=Connection refused
2010-05-10 18:05:00,158 WARN  [ScheduleThread] [ScheduleThread]
Disabling metrics for: 3:10131
2010-05-10 18:05:00,161 ERROR [ScheduleThread] [ScheduleThread] Metric
unreachable: Error connecting with moni
tored server, make sure it is started and check http://0.0.0.0:34010/?*
Message=Connection refused
2010-05-10 18:05:00,161 WARN  [ScheduleThread] [ScheduleThread]
Disabling metrics for: 3:10130
2010-05-10 18:05:00,167 ERROR [ScheduleThread] [ScheduleThread] Metric
unreachable: Can't connect to MBeanServ
er
[java.naming.provider.url=jnp%3A//0.0.0.0%3A1099,java.naming.security.principal=%java.naming.security.princ
ipal%,java.naming.security.credentials=%java.naming.security.credentials%]:
javax.naming.NameNotFoundException
: jmx not bound
2010-05-10 18:05:00,167 WARN  [ScheduleThread] [ScheduleThread]
Disabling metrics for: 3:10062
2010-05-10 18:05:00,171 ERROR [ScheduleThread] [ScheduleThread] Metric
unreachable: Can't connect to MBeanServ
er
[java.naming.provider.url=jnp%3A//0.0.0.0%3A1099,java.naming.security.principal=%java.naming.security.princ
ipal%,java.naming.security.credentials=%java.naming.security.credentials%]:
javax.naming.NameNotFoundException
: jmx not bound
2010-05-10 18:05:00,171 WARN  [ScheduleThread] [ScheduleThread]
Disabling metrics for: 3:10061
2010-05-10 18:05:00,175 ERROR [ScheduleThread] [ScheduleThread] Metric
unreachable: Error connecting with moni
tored server, make sure it is started and check http://0.0.0.0:34010/?*
Message=Connection refused

2010-05-10 18:05:00,175 WARN  [ScheduleThread] [ScheduleThread]
Disabling metrics for: 3:10120
2010-05-10 18:05:00,177 ERROR [ScheduleThread] [ScheduleThread] Metric
unreachable: Error connecting with moni
tored server, make sure it is started and check http://0.0.0.0:34010/?*
Message=Connection refused
2010-05-10 18:05:00,177 WARN  [ScheduleThread] [ScheduleThread]
Disabling metrics for: 3:10122
2010-05-10 18:05:00,179 ERROR [ScheduleThread] [ScheduleThread] Metric
unreachable: Error connecting with moni
tored server, make sure it is started and check http://0.0.0.0:34010/?*
Message=Connection refused
2010-05-10 18:05:00,179 WARN  [ScheduleThread] [ScheduleThread]
Disabling metrics for: 3:10121
2010-05-10 18:05:00,180 ERROR [ScheduleThread] [ScheduleThread] Metric
unreachable: Error connecting with moni
tored server, make sure it is started and check http://0.0.0.0:34010/?*
Message=Connection refused
2010-05-10 18:05:00,180 WARN  [ScheduleThread] [ScheduleThread]
Disabling metrics for: 3:10124
2010-05-10 18:05:00,182 ERROR [ScheduleThread] [ScheduleThread] Metric
unreachable: Error connecting with moni
tored server, make sure it is started and check http://0.0.0.0:34010/?*
Message=Connection refused
2010-05-10 18:05:00,182 WARN  [ScheduleThread] [ScheduleThread]
Disabling metrics for: 3:10123
2010-05-10 18:05:00,184 ERROR [ScheduleThread] [ScheduleThread] Metric
unreachable: Error connecting with moni
tored server, make sure it is started and check http://0.0.0.0:34010/?*
Message=Connection refused
2010-05-10 18:05:00,184 WARN  [ScheduleThread] [ScheduleThread]
Disabling metrics for: 3:10126
2010-05-10 18:05:00,187 ERROR [ScheduleThread] [ScheduleThread] Metric
unreachable: Error connecting with moni
tored server, make sure it is started and check http://0.0.0.0:34010/?*
Message=Connection refused
2010-05-10 18:05:00,187 WARN  [ScheduleThread] [ScheduleThread]
Disabling metrics for: 3:10125
2010-05-10 18:05:00,189 ERROR [ScheduleThread] [ScheduleThread] Metric
unreachable: Error connecting with moni
tored server, make sure it is started and check http://0.0.0.0:34010/?*
Message=Connection refused
2010-05-10 18:05:00,189 WARN  [ScheduleThread] [ScheduleThread]
Disabling metrics for: 3:10127
2010-05-10 18:05:00,191 ERROR [ScheduleThread] [ScheduleThread] Metric
unreachable: Error connecting with moni
tored server, make sure it is started and check http://0.0.0.0:34010/?*
Message=Connection refused
2010-05-10 18:05:00,191 WARN  [ScheduleThread] [ScheduleThread]
Disabling metrics for: 3:10128
2010-05-10 18:05:00,193 ERROR [ScheduleThread] [ScheduleThread] Metric
unreachable: Error connecting with moni
tored server, make sure it is started and check http://0.0.0.0:34010/?*
Message=Connection refused
2010-05-10 18:05:00,193 WARN  [ScheduleThread] [ScheduleThread]
Disabling metrics for: 3:10129
2010-05-10 18:05:00,200 ERROR [ScheduleThread] [ScheduleThread] Metric
unreachable: Error connecting with moni
tored server, make sure it is started and check http://0.0.0.0:34010/?*
Message=Connection refused
2010-05-10 18:05:00,200 WARN  [ScheduleThread] [ScheduleThread]
Disabling metrics for: 3:10118
2010-05-10 18:05:00,202 ERROR [ScheduleThread] [ScheduleThread] Metric
unreachable: Error connecting with moni
tored server, make sure it is started and check http://0.0.0.0:34010/?*
Message=Connection refused
2010-05-10 18:05:00,202 WARN  [ScheduleThread] [ScheduleThread]
Disabling metrics for: 3:10119
2010-05-10 18:05:00,204 ERROR [ScheduleThread] [ScheduleThread] Metric
unreachable: Error connecting with moni
tored server, make sure it is started and check http://0.0.0.0:34010/?*
Message=Connection refused
2010-05-10 18:05:00,204 WARN  [ScheduleThread] [ScheduleThread]
Disabling metrics for: 3:10116
2010-05-10 18:05:00,206 ERROR [ScheduleThread] [ScheduleThread] Metric
unreachable: Error connecting with moni
tored server, make sure it is started and check http://0.0.0.0:34010/?*
Message=Connection refused
2010-05-10 18:05:00,206 WARN  [ScheduleThread] [ScheduleThread]
Disabling metrics for: 3:10117
2010-05-10 18:10:00,265 INFO  [ScheduleThread] [ScheduleThread]
Re-enabling metrics for: 3:10132
2010-05-10 18:10:00,265 INFO  [ScheduleThread] [ScheduleThread]
Re-enabling metrics for: 3:10131
2010-05-10 18:10:00,267 INFO  [ScheduleThread] [ScheduleThread]
Re-enabling metrics for: 3:10130
2010-05-10 18:10:00,271 INFO  [ScheduleThread] [ScheduleThread]
Re-enabling metrics for: 3:10062
2010-05-10 18:10:00,282 INFO  [ScheduleThread] [ScheduleThread]
Re-enabling metrics for: 3:10061
2010-05-10 18:10:00,318 INFO  [ScheduleThread] [ScheduleThread]
Re-enabling metrics for: 3:10120
2010-05-10 18:10:00,355 INFO  [ScheduleThread] [ScheduleThread]
Re-enabling metrics for: 3:10122
2010-05-10 18:10:00,377 INFO  [ScheduleThread] [ScheduleThread]
Re-enabling metrics for: 3:10121
2010-05-10 18:10:00,378 INFO  [ScheduleThread] [ScheduleThread]
Re-enabling metrics for: 3:10124
2010-05-10 18:10:00,378 INFO  [ScheduleThread] [ScheduleThread]
Re-enabling metrics for: 3:10123
2010-05-10 18:10:00,379 INFO  [ScheduleThread] [ScheduleThread]
Re-enabling metrics for: 3:10126
2010-05-10 18:10:00,382 INFO  [ScheduleThread] [ScheduleThread]
Re-enabling metrics for: 3:10125
2010-05-10 18:10:00,382 INFO  [ScheduleThread] [ScheduleThread]
Re-enabling metrics for: 3:10127
2010-05-10 18:10:00,383 INFO  [ScheduleThread] [ScheduleThread]
Re-enabling metrics for: 3:10128
2010-05-10 18:10:00,383 INFO  [ScheduleThread] [ScheduleThread]
Re-enabling metrics for: 3:10129
2010-05-10 18:10:00,426 INFO  [ScheduleThread] [ScheduleThread]
Re-enabling metrics for: 3:10118
2010-05-10 18:10:00,427 INFO  [ScheduleThread] [ScheduleThread]
Re-enabling metrics for: 3:10119
2010-05-10 18:10:00,427 INFO  [ScheduleThread] [ScheduleThread]
Re-enabling metrics for: 3:10116
2010-05-10 18:10:00,428 INFO  [ScheduleThread] [ScheduleThread]
Re-enabling metrics for: 3:10117
2010-05-10 18:11:00,010 INFO  [ScheduleThread] [ScheduleThread]
Re-enabling metrics for: 2:10024
2010-05-10 18:11:00,011 INFO  [ScheduleThread] [ScheduleThread]
Re-enabling metrics for: 3:10133
2010-05-11 16:24:09,230 INFO  [Thread-2] [RuntimeAutodiscoverer] Running
runtime autodiscovery for NetworkServ
er
2010-05-11 16:24:09,443 INFO  [Thread-2] [RuntimeAutodiscoverer]
NetworkServer discovery took 0
2010-05-11 16:24:09,443 INFO  [Thread-2] [RuntimeAutodiscoverer] Running
runtime autodiscovery for HQ Agent
2010-05-11 16:24:09,666 INFO  [Thread-2] [RuntimeAutodiscoverer] HQ
Agent discovery took 0.04
2010-05-11 16:24:09,666 INFO  [Thread-2] [RuntimeAutodiscoverer] Running
runtime autodiscovery for ProcessServ
er
2010-05-11 16:24:09,881 INFO  [Thread-2] [RuntimeAutodiscoverer]
ProcessServer discovery took 0.01
2010-05-11 16:24:09,881 INFO  [Thread-2] [RuntimeAutodiscoverer] Running
runtime autodiscovery for Tomcat 6.0
2010-05-11 16:24:10,131 INFO  [Thread-2] [RuntimeAutodiscoverer] Tomcat
6.0 discovery took 0.08
2010-05-11 16:24:10,132 INFO  [Thread-2] [RuntimeAutodiscoverer] Running
runtime autodiscovery for NTP 4.x
2010-05-11 16:24:10,318 INFO  [Thread-2] [RuntimeAutodiscoverer] NTP 4.x
discovery took 0
2010-05-11 16:24:10,319 INFO  [Thread-2] [RuntimeAutodiscoverer] Running
runtime autodiscovery for FileServer
2010-05-11 16:24:10,538 INFO  [Thread-2] [RuntimeAutodiscoverer]
FileServer discovery took 0
2010-05-11 16:24:10,538 INFO  [Thread-2] [RuntimeAutodiscoverer] Running
runtime autodiscovery for JBoss 4.2
2010-05-11 16:24:11,017 INFO  [Thread-2] [RuntimeAutodiscoverer] JBoss
4.2 discovery took 0.29
2010-05-11 16:24:11,018 INFO  [Thread-2] [RuntimeAutodiscoverer] Running
runtime autodiscovery for Net Service
s
2010-05-11 16:24:11,188 INFO  [Thread-2] [RuntimeAutodiscoverer] Net
Services discovery took 0
2010-05-11 19:57:16,486 INFO  [WrapperListener_stop_runner]
[AgentConnection] 127.0.0.1:2144 -> agent:die
2010-05-11 19:57:16,618 INFO  [WrapperListener_stop_runner]
[AgentConnection] 127.0.0.1:2144 -> agent:ping
2010-05-11 19:57:17,745 INFO  [Thread-1] [AgentCommandsServer] Agent
commands shut down
2010-05-11 19:57:17,750 INFO  [Thread-1] [CommandsServer] Commands
Server shut down
2010-05-11 19:57:17,751 INFO  [Thread-1] [ControlCommandsServer] Control
Commands Server shut down
2010-05-11 19:57:17,749 INFO  [WrapperListener_stop_runner]
[AgentConnection] 127.0.0.1:2144 -> agent:ping
2010-05-11 19:57:17,759 INFO  [Thread-1] [AutoinventoryCommandsServer]
Autoinventory Commands Server shutting
down
2010-05-11 19:57:18,262 INFO  [Thread-1] [AutoinventoryCommandsServer]
Autoinventory Commands Server shut down
2010-05-11 19:57:18,263 INFO  [Thread-1] [MeasurementCommandsServer]
Measurement Commands Server shutting down
2010-05-11 19:57:18,263 INFO  [SenderThread] [SenderThread] Measurement
sender interrupted
2010-05-11 19:57:28,267 WARN  [Thread-1] [MeasurementCommandsServer]
ScheduleThread did not die within the tim
eout period.  Killing it
2010-05-11 19:57:28,267 INFO  [Thread-1] [MeasurementCommandsServer]
Measurement Commands Server shut down
2010-05-11 19:57:28,273 INFO  [Thread-1] [RunnableTrackThread]
RunnableLogThread stopped
2010-05-11 19:57:28,273 INFO  [Thread-1] [CollectorThread]
CollectorThread stopped
2010-05-11 19:57:28,337 INFO  [Thread-1] [AgentDaemon] Agent shut down
2010-05-11 19:57:31,845 INFO  [main] [AgentConnection] 127.0.0.1:2144 ->
agent:ping
2010-05-11 19:57:32,637 INFO  [WrapperStartStopAppMain]
[AgentConnection] 127.0.0.1:2144 -> agent:ping
2010-05-11 19:57:34,324 INFO  [main] [AgentConnection] 127.0.0.1:2144 ->
agent:ping
2010-05-11 19:57:34,938 INFO  [Thread-1] [AgentCommandsServer] Agent
commands loaded
2010-05-11 19:57:35,102 INFO  [Thread-1] [AgentDaemon] Agent starting
up, bundle name=agent-4.2.0-1260
2010-05-11 19:57:35,151 INFO  [Thread-1] [ProductPluginManager]
agent.pdkDir=../../bundles/agent-4.2.0-1260/pd
k
2010-05-11 19:57:35,151 INFO  [Thread-1] [ProductPluginManager]
agent.pdkPluginDir=../../bundles/agent-4.2.0-1
260/pdk/plugins
2010-05-11 19:57:35,151 INFO  [Thread-1] [ProductPluginManager]
agent.pdkWorkDir=../../bundles/agent-4.2.0-126
0/pdk/work
2010-05-11 19:57:35,273 INFO  [Thread-1] [ProductPluginManager] Loading
plugin: groovy-scripting-plugin.jar
2010-05-11 19:57:35,328 INFO  [main] [AgentConnection] 127.0.0.1:2144 ->
agent:ping
2010-05-11 19:57:35,781 INFO  [Thread-1] [ProductPluginManager] Loading
plugin: system-plugin.jar
2010-05-11 19:57:36,015 INFO  [Thread-1] [ProductPluginManager] Loading
plugin: sqlquery-plugin.jar
2010-05-11 19:57:36,073 INFO  [Thread-1] [SQLQueryMeasurementPlugin]
Registered proxy for domain: sql
2010-05-11 19:57:36,074 INFO  [Thread-1] [ProductPluginManager] Loading
plugin: netservices-plugin.jar
2010-05-11 19:57:36,146 INFO  [Thread-1] [MeasurementPlugin] Register
TCP Socket proxy for domain: socket
2010-05-11 19:57:36,146 INFO  [Thread-1] [MeasurementPlugin] Register
HTTP proxy for domain: http

Somehow got some duplicated lines in there. Sorry.

   
Message was edited by: bjchip

Viewing all articles
Browse latest Browse all 52618

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>