linux system slab cache proc_node_cache usage too large

Question

My system run java program only, at some time the memory usage increase very quickly to a very hight percentage:

memory usage of java :

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                       
23046 admin     18   0 2115m 427m  13m S 14.7 10.8   1609:58 java

but the system memory usage is :

[root@AY130716093217935dfdZ ~]# free -m
             total       used       free     shared    buffers     cached
Mem:          3959       3709        249          0         93        122
-/+ buffers/cache:       3493        465
Swap:            0          0          0

then I know the usage exclude java are system kernel’s slab :

[root@AY130716093217935dfdZ ~]# cat /proc/meminfo | grep slab -i
Slab:          3092688 kB

so, I check the detail usage of slab use slabtop :

 Active / Total Objects (% used)    : 9509520 / 9545651 (99.6%)
 Active / Total Slabs (% used)      : 771723 / 771725 (100.0%)
 Active / Total Caches (% used)     : 94 / 138 (68.1%)
 Active / Total Size (% used)       : 2751436.50K / 2761816.41K (99.6%)
 Minimum / Average / Maximum Object : 0.02K / 0.29K / 128.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
3140784 3140358  99%    0.08K  65433       48    261732K selinux_inode_security
3127590 3127260  99%    0.21K 173755       18    695020K dentry_cache
3108264 3108182  99%    0.58K 518044        6   2072176K proc_inode_cache

You can found , the proc_node_cache, entry_cache very large.

so, I suppose there are some program scan the /proc directory ?

but the java program should not do this things.

when I restart the java program, the slab shrink to low.

btw, my system mount an nfs and the java program write log to the file on nfs:

[root@AY130716093217935dfdZ ~]# mount
/dev/hda1 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
//42.121.111.155/server23 on /mnt/server23 type cifs (rw,mand)
//10.161.143.83/fulllogs on /mnt/newlogs type cifs (rw,mand)

My kernel is :

Linux AY130716093217935dfdZ 2.6.18-308.el5 #1 SMP Tue Feb 21 20:06:06 EST 2012 x86_64 x86_64 x86_64 GNU/Linux

Os:

CentOS release 5.8 (Final)

machine was created by XEN. as a XEN guest.

I want to know, what situation will produce such large slab memory usage of proc_inode_cache ?

what maybe happens to my server ?

Thank you !

UPDATE:

by view the source code of 2.6.18 kernel.
in exit.c:

do_exit -> release_task ->proc_flush_task(p);

the function proc_flush_task:

/**                                                                                                                                                                                                         
 * proc_flush_task -  Remove dcache entries for @task from the /proc dcache.

which means when task quit, the proc will be reaped, so, this can explain why slab shrink when I stop the java process.

after now, there is question because the thread of java is 600, which equals to other time, why so much the proc_inode it holds ?

linux system slab cache proc_node_cache usage too large

Answer

Leave a Comment Cancel reply