Hi,
I have a trouble on a server where the snmp agent don't collect data where the load go up. The problem seems that the snmp take too long to response and the agent suspend the data collecting.
The server is a CentOS 5.5 64bit and is a clone (apache config) of a server with the same software version but in 32bit mode that work as aspected also on hight load.
I have made a simple script to check the snmp response time from shell and max time to get all snmp tree was 500ms while for the agent is over 3000ms.
In the agent.log i see this line:
2011-02-02 17:50:09,022 WARN [pool-3-thread-1] [ScheduleThread] Disabling metrics for: 3:11600
2011-02-02 17:50:09,022 WARN [pool-3-thread-1] [ScheduleThread] Collection of metric: 'Apache 2.2 VHost:apache:snmpIp=127.0.0.1,snmpPort=1610,snmpTransport=udp,snmpVersion=v2c,snmpCommunity=public,snmpUser=username,snmpSecurityContext=%snmpSecurityContext%,snmpAuthType=none,snmpPassword=****************,snmpPrivacyType=none,snmpPrivacyPassPhrase=*************************:wwwSummaryOutLowBytes:snmpIndexName=wwwServiceName->wwwServiceProtocol,snmpIndexValue=xxxxx.it->1.3.6.1.2.1.6.80__RATE__=1m' took: 3004ms
2011-02-02 17:50:12,027 WARN [pool-3-thread-1] [ScheduleThread] Disabling metrics for: 3:11602
2011-02-02 17:50:12,027 WARN [pool-3-thread-1] [ScheduleThread] Collection of metric: 'Apache 2.2 VHost:apache:snmpIp=127.0.0.1,snmpPort=1610,snmpTransport=udp,snmpVersion=v2c,snmpCommunity=public,snmpUser=username,snmpSecurityContext=%snmpSecurityContext%,snmpAuthType=none,snmpPassword=****************,snmpPrivacyType=none,snmpPrivacyPassPhrase=*************************:wwwSummaryOutLowBytes:snmpIndexName=wwwServiceName->wwwServiceProtocol,snmpIndexValue=localhost->1.3.6.1.2.1.6.80__RATE__=1m' took: 3004ms
2011-02-02 17:50:15,032 WARN [pool-3-thread-1] [ScheduleThread] Disabling metrics for: 3:11601
2011-02-02 17:50:15,032 WARN [pool-3-thread-1] [ScheduleThread] Collection of metric: 'Apache 2.2 VHost:apache:snmpIp=127.0.0.1,snmpPort=1610,snmpTransport=udp,snmpVersion=v2c,snmpCommunity=public,snmpUser=username,snmpSecurityContext=%snmpSecurityContext%,snmpAuthType=none,snmpPassword=****************,snmpPrivacyType=none,snmpPrivacyPassPhrase=*************************:wwwSummaryOutLowBytes:snmpIndexName=wwwServiceName->wwwServiceProtocol,snmpIndexValue=www.xxxxx.it->1.3.6.1.2.1.6.80__RATE__=1m' took: 3004ms
2011-02-02 17:50:18,037 WARN [pool-3-thread-1] [ScheduleThread] Disabling metrics for: 3:11598
2011-02-02 17:50:18,039 WARN [pool-3-thread-1] [ScheduleThread] Collection of metric: 'Apache 2.2 VHost:apache:snmpIp=127.0.0.1,snmpPort=1610,snmpTransport=udp,snmpVersion=v2c,snmpCommunity=public,snmpUser=username,snmpSecurityContext=%snmpSecurityContext%,snmpAuthType=none,snmpPassword=****************,snmpPrivacyType=none,snmpPrivacyPassPhrase=*************************:wwwSummaryOutLowBytes:snmpIndexName=wwwServiceName->wwwServiceProtocol,snmpIndexValue=192.168.3.140->1.3.6.1.2.1.6.80__RATE__=1m' took: 3004ms
2011-02-02 17:50:21,043 WARN [pool-3-thread-1] [ScheduleThread] Disabling metrics for: 2:10055
2011-02-02 17:50:21,043 WARN [pool-3-thread-1] [ScheduleThread] Collection of metric: 'Apache 2.2:apache:snmpIp=127.0.0.1,snmpPort=1610,snmpTransport=udp,snmpVersion=v2c,snmpCommunity=public,snmpUser=username,snmpSecurityContext=%snmpSecurityContext%,snmpAuthType=none,snmpPassword=****************,snmpPrivacyType=none,snmpPrivacyPassPhrase=*************************:wwwSummaryOutLowBytes:snmpIndexName=wwwServiceName->wwwServiceProtocol,snmpIndexValue=marvel->1.3.6.1.2.1.6.80__RATE__=1m' took: 3003ms
2011-02-02 17:56:00,007 INFO [ScheduleThread] [ScheduleThread] Re-enabling metrics for: 3:11599
2011-02-02 17:56:00,007 INFO [ScheduleThread] [ScheduleThread] Re-enabling metrics for: 3:11597
2011-02-02 17:56:00,007 INFO [ScheduleThread] [ScheduleThread] Re-enabling metrics for: 3:11600
2011-02-02 17:56:00,007 INFO [ScheduleThread] [ScheduleThread] Re-enabling metrics for: 3:11602
2011-02-02 17:56:00,007 INFO [ScheduleThread] [ScheduleThread] Re-enabling metrics for: 3:11601
2011-02-02 17:56:00,007 INFO [ScheduleThread] [ScheduleThread] Re-enabling metrics for: 3:11598
2011-02-02 17:56:00,007 INFO [ScheduleThread] [ScheduleThread] Re-enabling metrics for: 2:10055
2011-02-02 18:00:03,015 WARN [pool-3-thread-1] [ScheduleThread] Disabling metrics for: 3:11599
Regards
Lucio
I have a trouble on a server where the snmp agent don't collect data where the load go up. The problem seems that the snmp take too long to response and the agent suspend the data collecting.
The server is a CentOS 5.5 64bit and is a clone (apache config) of a server with the same software version but in 32bit mode that work as aspected also on hight load.
I have made a simple script to check the snmp response time from shell and max time to get all snmp tree was 500ms while for the agent is over 3000ms.
In the agent.log i see this line:
2011-02-02 17:50:09,022 WARN [pool-3-thread-1] [ScheduleThread] Disabling metrics for: 3:11600
2011-02-02 17:50:09,022 WARN [pool-3-thread-1] [ScheduleThread] Collection of metric: 'Apache 2.2 VHost:apache:snmpIp=127.0.0.1,snmpPort=1610,snmpTransport=udp,snmpVersion=v2c,snmpCommunity=public,snmpUser=username,snmpSecurityContext=%snmpSecurityContext%,snmpAuthType=none,snmpPassword=****************,snmpPrivacyType=none,snmpPrivacyPassPhrase=*************************:wwwSummaryOutLowBytes:snmpIndexName=wwwServiceName->wwwServiceProtocol,snmpIndexValue=xxxxx.it->1.3.6.1.2.1.6.80__RATE__=1m' took: 3004ms
2011-02-02 17:50:12,027 WARN [pool-3-thread-1] [ScheduleThread] Disabling metrics for: 3:11602
2011-02-02 17:50:12,027 WARN [pool-3-thread-1] [ScheduleThread] Collection of metric: 'Apache 2.2 VHost:apache:snmpIp=127.0.0.1,snmpPort=1610,snmpTransport=udp,snmpVersion=v2c,snmpCommunity=public,snmpUser=username,snmpSecurityContext=%snmpSecurityContext%,snmpAuthType=none,snmpPassword=****************,snmpPrivacyType=none,snmpPrivacyPassPhrase=*************************:wwwSummaryOutLowBytes:snmpIndexName=wwwServiceName->wwwServiceProtocol,snmpIndexValue=localhost->1.3.6.1.2.1.6.80__RATE__=1m' took: 3004ms
2011-02-02 17:50:15,032 WARN [pool-3-thread-1] [ScheduleThread] Disabling metrics for: 3:11601
2011-02-02 17:50:15,032 WARN [pool-3-thread-1] [ScheduleThread] Collection of metric: 'Apache 2.2 VHost:apache:snmpIp=127.0.0.1,snmpPort=1610,snmpTransport=udp,snmpVersion=v2c,snmpCommunity=public,snmpUser=username,snmpSecurityContext=%snmpSecurityContext%,snmpAuthType=none,snmpPassword=****************,snmpPrivacyType=none,snmpPrivacyPassPhrase=*************************:wwwSummaryOutLowBytes:snmpIndexName=wwwServiceName->wwwServiceProtocol,snmpIndexValue=www.xxxxx.it->1.3.6.1.2.1.6.80__RATE__=1m' took: 3004ms
2011-02-02 17:50:18,037 WARN [pool-3-thread-1] [ScheduleThread] Disabling metrics for: 3:11598
2011-02-02 17:50:18,039 WARN [pool-3-thread-1] [ScheduleThread] Collection of metric: 'Apache 2.2 VHost:apache:snmpIp=127.0.0.1,snmpPort=1610,snmpTransport=udp,snmpVersion=v2c,snmpCommunity=public,snmpUser=username,snmpSecurityContext=%snmpSecurityContext%,snmpAuthType=none,snmpPassword=****************,snmpPrivacyType=none,snmpPrivacyPassPhrase=*************************:wwwSummaryOutLowBytes:snmpIndexName=wwwServiceName->wwwServiceProtocol,snmpIndexValue=192.168.3.140->1.3.6.1.2.1.6.80__RATE__=1m' took: 3004ms
2011-02-02 17:50:21,043 WARN [pool-3-thread-1] [ScheduleThread] Disabling metrics for: 2:10055
2011-02-02 17:50:21,043 WARN [pool-3-thread-1] [ScheduleThread] Collection of metric: 'Apache 2.2:apache:snmpIp=127.0.0.1,snmpPort=1610,snmpTransport=udp,snmpVersion=v2c,snmpCommunity=public,snmpUser=username,snmpSecurityContext=%snmpSecurityContext%,snmpAuthType=none,snmpPassword=****************,snmpPrivacyType=none,snmpPrivacyPassPhrase=*************************:wwwSummaryOutLowBytes:snmpIndexName=wwwServiceName->wwwServiceProtocol,snmpIndexValue=marvel->1.3.6.1.2.1.6.80__RATE__=1m' took: 3003ms
2011-02-02 17:56:00,007 INFO [ScheduleThread] [ScheduleThread] Re-enabling metrics for: 3:11599
2011-02-02 17:56:00,007 INFO [ScheduleThread] [ScheduleThread] Re-enabling metrics for: 3:11597
2011-02-02 17:56:00,007 INFO [ScheduleThread] [ScheduleThread] Re-enabling metrics for: 3:11600
2011-02-02 17:56:00,007 INFO [ScheduleThread] [ScheduleThread] Re-enabling metrics for: 3:11602
2011-02-02 17:56:00,007 INFO [ScheduleThread] [ScheduleThread] Re-enabling metrics for: 3:11601
2011-02-02 17:56:00,007 INFO [ScheduleThread] [ScheduleThread] Re-enabling metrics for: 3:11598
2011-02-02 17:56:00,007 INFO [ScheduleThread] [ScheduleThread] Re-enabling metrics for: 2:10055
2011-02-02 18:00:03,015 WARN [pool-3-thread-1] [ScheduleThread] Disabling metrics for: 3:11599
Regards
Lucio