Hello,
Does anyone know why my Hyperic Server process continuously gets killed by the Linux kernel's oom-killer?
I am running Hyperic HQ Server 4.6 on a 64-bit Ubutnu 11.10 virtual server hosted by VMware. The server has 1GB RAM and I have allocated 512MB RAM to the Hyperic server process with the execution options set in ./hyperic/server-4.6/conf/hq-server.conf:
server.java.opts=-Djava.awt.headless=true -XX:MaxPermSize=192m -Xmx512m -Xms512m -XX:+HeapDumpOnOutOfMemoryError
This is a dedicated server running only the Hyperic server, it uses a stand alone PostgreSQL 8.4 database. I don't have the Hyperic Agent client running, only the Hyperic Server.
Both the java process and the wrapper process (wrapper-linux-x) are killed off by the oom-killer. I noticed that the swap memory was totally used when oom-killer happens. I noticed that it happens often when the daily Cron jobs are executing, but also at other times during the day when there was no other activity. One time it happened when viewing an event list in the Hyperic GUI.
Find below some of the kernel log output:
Thanks.
----------------------------
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375303] wrapper-linux-x invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375337] wrapper-linux-x cpuset=/ mems_allowed=0
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375345] Pid: 14256, comm: wrapper-linux-x Not tainted 3.0.0-12-server #20-Ubuntu
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375506] Mem-Info:
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375509] Node 0 DMA per-cpu:
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375513] CPU 0: hi: 0, btch: 1 usd: 0
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375517] CPU 1: hi: 0, btch: 1 usd: 0
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375519] Node 0 DMA32 per-cpu:
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375523] CPU 0: hi: 186, btch: 31 usd: 31
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375526] CPU 1: hi: 186, btch: 31 usd: 5
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375532] active_anon:94694 inactive_anon:94791 isolated_anon:0
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375534] active_file:171 inactive_file:216 isolated_file:0
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375535] unevictable:0 dirty:6 writeback:0 unstable:0
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375537] free:11750 slab_reclaimable:2360 slab_unreclaimable:3080
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375538] mapped:3855 shmem:3808 pagetables:2161 bounce:0
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375542] Node 0 DMA free:4104kB min:776kB low:968kB high:1164kB active_anon:5040kB inactive_anon:5176kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15688kB mlocked:0kB dirty:0kB writeback:0kB mapped:924kB shmem:924kB slab_reclaimable:128kB slab_unreclaimable:468kB kernel_stack:208kB pagetables:576kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375555] lowmem_reserve[]: 0 844 844 844
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375561] Node 0 DMA32 free:42896kB min:42948kB low:53684kB high:64420kB active_anon:373736kB inactive_anon:373988kB active_file:684kB inactive_file:864kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:864496kB mlocked:0kB dirty:24kB writeback:0kB mapped:14496kB shmem:14308kB slab_reclaimable:9312kB slab_unreclaimable:11852kB kernel_stack:1976kB pagetables:8068kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2329 all_unreclaimable? yes
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375574] lowmem_reserve[]: 0 0 0 0
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375579] Node 0 DMA: 8*4kB 18*8kB 22*16kB 20*32kB 14*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 4112kB
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375596] Node 0 DMA32: 294*4kB 531*8kB 426*16kB 286*32kB 130*64kB 43*128kB 6*256kB 2*512kB 1*1024kB 0*2048kB 1*4096kB = 42896kB
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375610] 17193 total pagecache pages
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375613] 12967 pages in swap cache
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375616] Swap cache stats: add 974728, delete 961761, find 1400113/1467316
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375619] Free swap = 0kB
Aug 28 06:41:29 ubuntuserver11 kernel: [327021.375621] Total swap = 522236kB
Aug 28 06:41:30 ubuntuserver11 kernel: [327021.380070] 223216 pages RAM