One of our Hadoop datanode with CentOS7 64 bit is using all of it’s RAM, I tried to figure out which process is using so much RAM but couldn’t. Please help me to check this please:
System: CentOS 7 64bit, with 64 GiB RAM
HTOP on server:
Htop show that all memory is used:PS to find which process is use much RAM – see only 3 process use more than 10%, but total less than 50%:
~]# ps aux | awk '{print $4"\t"$11}' | sort | uniq -c | awk '{print $2" "$1" "$3}' | sort -nr|more 14.0 1 /zserver/java/jdk1.8.0_11-x64/bin/java 13.8 1 /zserver/java/jdk1.8.0_11-x64/bin/java 12.3 1 /zserver/java/jdk1.8.0_11-x64/bin/java 1.7 1 /zserver/java/jdk1.8.0_11-x64/bin/java 0.2 1 /opt/dell/srvadmin/sbin/dsm_om_connsvcd 0.1 1 /zserver/java/jdk1.8.0_11-x64/bin/java %MEM 1 COMMAND 0.0 2 sshd: 0.0 2 sort 0.0 2 /opt/dell/srvadmin/sbin/dsm_sa_datamgrd 0.0 2 [ext4-unrsv-conv] 0.0 2 [ext4-rsv-conver] 0.0 2 /bin/bashMemory Info:
~]# cat /proc/meminfo MemTotal: 65696732 kB MemFree: 300744 kB MemAvailable: 603864 kB Buffers: 724 kB Cached: 401512 kB SwapCached: 661620 kB Active: 25435684 kB Inactive: 3525360 kB Active(anon): 25367048 kB Inactive(anon): 3362788 kB Active(file): 68636 kB Inactive(file): 162572 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 14680056 kB SwapFree: 6062316 kB Dirty: 2464 kB Writeback: 0 kB AnonPages: 27894800 kB Mapped: 25504 kB Shmem: 173228 kB Slab: 442388 kB SReclaimable: 317624 kB SUnreclaim: 124764 kB KernelStack: 20272 kB PageTables: 75916 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 14680056 kB Committed_AS: 34058880 kB VmallocTotal: 34359738367 kB VmallocUsed: 410120 kB VmallocChunk: 34325729008 kB HardwareCorrupted: 0 kB AnonHugePages: 1187840 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 110288 kB DirectMap2M: 3985408 kB DirectMap1G: 65011712 kBvmstat -s
~]# vmstat -s 65696732 K total memory 64522660 K used memory 25454076 K active memory 3453168 K inactive memory 334916 K free memory 1276 K buffer memory 837880 K swap cache 14680056 K total swap 8610992 K used swap 6069064 K free swap 6298373517 non-nice user cpu ticks 4033 nice user cpu ticks 554662032 system cpu ticks 40820968522 idle cpu ticks 245079249 IO-wait cpu ticks 34653 IRQ cpu ticks 29684952 softirq cpu ticks 0 stolen cpu ticks 411523475646 pages paged in 247562770289 pages paged out 21705066 pages swapped in 15439699 pages swapped out 1995305583 interrupts 3450539885 CPU context switches 1440495600 boot time 394260663 forkscat /proc/slabinfo
slabinfo - version: 2.1 # name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail> xfs_icr 0 0 144 56 2 : tunables 0 0 0 : slabdata 0 0 0 xfs_ili 11686 11686 152 53 2 : tunables 0 0 0 : slabdata 221 221 0 xfs_inode 119224 119868 1024 32 8 : tunables 0 0 0 : slabdata 3920 3920 0 xfs_efd_item 4960 4960 400 40 4 : tunables 0 0 0 : slabdata 124 124 0 xfs_da_state 1608 1608 488 67 8 : tunables 0 0 0 : slabdata 24 24 0 xfs_btree_cur 936 936 208 39 2 : tunables 0 0 0 : slabdata 24 24 0 xfs_log_ticket 15180 15400 184 44 2 : tunables 0 0 0 : slabdata 350 350 0 fat_inode_cache 180 180 720 45 8 : tunables 0 0 0 : slabdata 4 4 0 fat_cache 102 102 40 102 1 : tunables 0 0 0 : slabdata 1 1 0 kvm_async_pf 0 0 136 60 2 : tunables 0 0 0 : slabdata 0 0 0 kvm_vcpu 0 0 16256 2 8 : tunables 0 0 0 : slabdata 0 0 0 kvm_mmu_page_header 0 0 168 48 2 : tunables 0 0 0 : slabdata 0 0 0 kcopyd_job 0 0 3312 9 8 : tunables 0 0 0 : slabdata 0 0 0 dm_uevent 0 0 2608 12 8 : tunables 0 0 0 : slabdata 0 0 0 dm_rq_target_io 0 0 424 38 4 : tunables 0 0 0 : slabdata 0 0 0 ext4_groupinfo_4k 3720 3720 136 60 2 : tunables 0 0 0 : slabdata 62 62 0 ext4_inode_cache 3894 6047 1048 31 8 : tunables 0 0 0 : slabdata 205 205 0 ext4_xattr 46 46 88 46 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_free_data 9920 10240 64 64 1 : tunables 0 0 0 : slabdata 160 160 0 ext4_allocation_context 1536 1536 128 64 2 : tunables 0 0 0 : slabdata 24 24 0 ext4_io_end 12699 12699 80 51 1 : tunables 0 0 0 : slabdata 249 249 0 ext4_extent_status 25808 26520 40 102 1 : tunables 0 0 0 : slabdata 260 260 0 jbd2_journal_handle 2040 2040 48 85 1 : tunables 0 0 0 : slabdata 24 24 0 jbd2_journal_head 6552 6552 112 36 1 : tunables 0 0 0 : slabdata 182 182 0 jbd2_revoke_table_s 512 512 16 256 1 : tunables 0 0 0 : slabdata 2 2 0 jbd2_revoke_record_s 3072 3072 32 128 1 : tunables 0 0 0 : slabdata 24 24 0 scsi_cmd_cache 5976 6156 448 36 4 : tunables 0 0 0 : slabdata 171 171 0 UDPLITEv6 0 0 1152 28 8 : tunables 0 0 0 : slabdata 0 0 0 UDPv6 672 672 1152 28 8 : tunables 0 0 0 : slabdata 24 24 0 tw_sock_TCPv6 2816 3264 256 64 4 : tunables 0 0 0 : slabdata 51 51 0 TCPv6 1148 1280 2048 16 8 : tunables 0 0 0 : slabdata 80 80 0 cfq_queue 20230 20580 232 70 4 : tunables 0 0 0 : slabdata 294 294 0 bsg_cmd 0 0 312 52 4 : tunables 0 0 0 : slabdata 0 0 0 mqueue_inode_cache 36 36 896 36 8 : tunables 0 0 0 : slabdata 1 1 0 hugetlbfs_inode_cache 53 53 608 53 8 : tunables 0 0 0 : slabdata 1 1 0 configfs_dir_cache 0 0 88 46 1 : tunables 0 0 0 : slabdata 0 0 0 dquot 1536 1536 256 64 4 : tunables 0 0 0 : slabdata 24 24 0 kioctx 0 0 576 56 8 : tunables 0 0 0 : slabdata 0 0 0 pid_namespace 0 0 2176 15 8 : tunables 0 0 0 : slabdata 0 0 0 posix_timers_cache 0 0 248 66 4 : tunables 0 0 0 : slabdata 0 0 0 UDP-Lite 0 0 1024 32 8 : tunables 0 0 0 : slabdata 0 0 0 ip_fib_trie 146 146 56 73 1 : tunables 0 0 0 : slabdata 2 2 0 RAW 952 952 960 34 8 : tunables 0 0 0 : slabdata 28 28 0 UDP 768 768 1024 32 8 : tunables 0 0 0 : slabdata 24 24 0 tw_sock_TCP 11904 12224 256 64 4 : tunables 0 0 0 : slabdata 191 191 0 TCP 1385 1553 1920 17 8 : tunables 0 0 0 : slabdata 94 94 0 blkdev_queue 305 352 2016 16 8 : tunables 0 0 0 : slabdata 22 22 0 blkdev_requests 43216 43216 384 42 4 : tunables 0 0 0 : slabdata 1032 1032 0 blkdev_ioc 1248 1248 104 39 1 : tunables 0 0 0 : slabdata 32 32 0 fsnotify_event_holder 4080 4080 24 170 1 : tunables 0 0 0 : slabdata 24 24 0 fsnotify_event 18089 18428 120 68 2 : tunables 0 0 0 : slabdata 271 271 0 sock_inode_cache 7157 7548 640 51 8 : tunables 0 0 0 : slabdata 148 148 0 net_namespace 0 0 4224 7 8 : tunables 0 0 0 : slabdata 0 0 0 shmem_inode_cache 3600 3600 680 48 8 : tunables 0 0 0 : slabdata 75 75 0 Acpi-ParseExt 126896 126896 72 56 1 : tunables 0 0 0 : slabdata 2266 2266 0 Acpi-Namespace 12036 12036 40 102 1 : tunables 0 0 0 : slabdata 118 118 0 taskstats 1176 1176 328 49 4 : tunables 0 0 0 : slabdata 24 24 0 proc_inode_cache 10722 10961 656 49 8 : tunables 0 0 0 : slabdata 415 415 0 sigqueue 1275 1275 160 51 2 : tunables 0 0 0 : slabdata 25 25 0 bdev_cache 429 429 832 39 8 : tunables 0 0 0 : slabdata 11 11 0 sysfs_dir_cache 46944 46944 112 36 1 : tunables 0 0 0 : slabdata 1304 1304 0 inode_cache 16541 16732 592 55 8 : tunables 0 0 0 : slabdata 306 306 0 dentry 69396 69489 192 42 2 : tunables 0 0 0 : slabdata 1655 1655 0 iint_cache 0 0 80 51 1 : tunables 0 0 0 : slabdata 0 0 0 selinux_inode_security 15819 16524 80 51 1 : tunables 0 0 0 : slabdata 324 324 0 buffer_head 17377 21060 104 39 1 : tunables 0 0 0 : slabdata 540 540 0 vm_area_struct 11403 12654 216 37 2 : tunables 0 0 0 : slabdata 342 342 0 mm_struct 800 800 1600 20 8 : tunables 0 0 0 : slabdata 40 40 0 files_cache 1428 1428 640 51 8 : tunables 0 0 0 : slabdata 28 28 0 signal_cache 3180 3360 1152 28 8 : tunables 0 0 0 : slabdata 120 120 0 sighand_cache 1770 1803 2112 15 8 : tunables 0 0 0 : slabdata 123 123 0 task_xstate 6575 7106 832 39 8 : tunables 0 0 0 : slabdata 184 184 0 task_struct 1770 1805 2912 11 8 : tunables 0 0 0 : slabdata 165 165 0 anon_vma 11769 12288 64 64 1 : tunables 0 0 0 : slabdata 192 192 0 shared_policy_node 11730 11730 48 85 1 : tunables 0 0 0 : slabdata 138 138 0 numa_policy 186 186 264 62 4 : tunables 0 0 0 : slabdata 3 3 0 radix_tree_node 158606 260645 584 56 8 : tunables 0 0 0 : slabdata 4664 4664 0 idr_layer_cache 743 780 2112 15 8 : tunables 0 0 0 : slabdata 52 52 0 dma-kmalloc-8192 0 0 8192 4 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-4096 0 0 4096 8 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-2048 0 0 2048 16 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-1024 0 0 1024 32 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-512 0 0 512 64 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-256 0 0 256 64 4 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-128 0 0 128 64 2 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-64 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-32 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-16 0 0 16 256 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-8 0 0 8 512 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-192 0 0 192 42 2 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-96 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-8192 273 310 8192 4 8 : tunables 0 0 0 : slabdata 79 79 0 kmalloc-4096 862 897 4096 8 8 : tunables 0 0 0 : slabdata 113 113 0 kmalloc-2048 1696 1830 2048 16 8 : tunables 0 0 0 : slabdata 124 124 0 kmalloc-1024 5193 5440 1024 32 8 : tunables 0 0 0 : slabdata 170 170 0 kmalloc-512 9437 10232 512 64 8 : tunables 0 0 0 : slabdata 166 166 0 kmalloc-256 9572 13392 256 64 4 : tunables 0 0 0 : slabdata 210 210 0 kmalloc-192 13970 15078 192 42 2 : tunables 0 0 0 : slabdata 359 359 0 kmalloc-128 22042 24576 128 64 2 : tunables 0 0 0 : slabdata 384 384 0 kmalloc-96 4454 4620 96 42 1 : tunables 0 0 0 : slabdata 110 110 0 kmalloc-64 43106 51072 64 64 1 : tunables 0 0 0 : slabdata 798 798 0 kmalloc-32 19072 19072 32 128 1 : tunables 0 0 0 : slabdata 149 149 0 kmalloc-16 13312 13568 16 256 1 : tunables 0 0 0 : slabdata 53 53 0 kmalloc-8 16896 16896 8 512 1 : tunables 0 0 0 : slabdata 33 33 0 kmem_cache_node 682 704 64 64 1 : tunables 0 0 0 : slabdata 11 11 0 kmem_cache 576 576 256 64 4 : tunables 0 0 0 : slabdata 9 9 0Have tried to drop the system cache, but the memory doesn’t free up:
# sync; echo 1 > /proc/sys/vm/drop_caches # sync; echo 2 > /proc/sys/vm/drop_caches # sync; echo 3 > /proc/sys/vm/drop_cachesslabtop -o
Active / Total Objects (% used) : 1244950 / 1358427 (91.6%) Active / Total Slabs (% used) : 31188 / 31188 (100.0%) Active / Total Caches (% used) : 77 / 106 (72.6%) Active / Total Size (% used) : 473366.03K / 528796.00K (89.5%) Minimum / Average / Maximum Object : 0.01K / 0.39K / 15.88K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 231616 169413 73% 0.57K 4143 56 132576K radix_tree_node 203516 195136 95% 1.00K 8291 32 265312K xfs_inode 126896 126896 100% 0.07K 2266 56 9064K Acpi-ParseExt 101166 92953 91% 0.10K 2594 39 10376K buffer_head 96453 86834 90% 0.19K 2297 42 18376K dentry 66044 62392 94% 0.38K 1574 42 25184K blkdev_requests 51072 45181 88% 0.06K 798 64 3192K kmalloc-64 46944 46944 100% 0.11K 1304 36 5216K sysfs_dir_cache 26520 25808 97% 0.04K 260 102 1040K ext4_extent_status 24704 23935 96% 0.12K 386 64 3088K kmalloc-128 20160 19810 98% 0.23K 288 70 4608K cfq_queue 19072 19072 100% 0.03K 149 128 596K kmalloc-32 18428 18089 98% 0.12K 271 68 2168K fsnotify_event 16896 16896 100% 0.01K 33 512 132K kmalloc-8 16732 16541 98% 0.58K 306 55 9792K inode_cache 16524 15819 95% 0.08K 324 51 1296K selinux_inode_security 15444 15224 98% 0.18K 351 44 2808K xfs_log_ticket 15036 14311 95% 0.19K 358 42 2864K kmalloc-192 14813 14813 100% 0.15K 280 53 2240K xfs_ili 14016 12421 88% 0.06K 219 64 876K anon_vma 13584 10797 79% 0.25K 213 64 3408K kmalloc-256 13568 13312 98% 0.02K 53 256 212K kmalloc-16 12950 12528 96% 0.21K 350 37 2800K vm_area_struct 12699 12699 100% 0.08K 249 51 996K ext4_io_end 12224 11904 97% 0.25K 191 64 3056K tw_sock_TCP 12036 12036 100% 0.04K 118 102 472K Acpi-Namespace 11730 11730 100% 0.05K 138 85 552K shared_policy_node 10736 10680 99% 0.50K 180 64 5760K kmalloc-512 10240 9920 96% 0.06K 160 64 640K ext4_free_data 8573 8531 99% 0.64K 189 49 6048K proc_inode_cache 7548 7157 94% 0.62K 148 51 4736K sock_inode_cache 7067 6574 93% 0.81K 183 39 5856K task_xstate 6552 6552 100% 0.11K 182 36 728K jbd2_journal_head 6372 6012 94% 0.44K 177 36 2832K scsi_cmd_cache 6047 3894 64% 1.02K 205 31 6560K ext4_inode_cache 5568 5165 92% 1.00K 174 32 5568K kmalloc-1024 4452 4171 93% 0.09K 106 42 424K kmalloc-96 4080 4080 100% 0.02K 24 170 96K fsnotify_event_holder 4010 3730 93% 0.39K 101 40 1616K xfs_efd_item 3720 3720 100% 0.13K 62 60 496K ext4_groupinfo_4k 3600 3600 100% 0.66K 75 48 2400K shmem_inode_cache 3360 3180 94% 1.12K 120 28 3840K signal_cache 3264 2816 86% 0.25K 51 64 816K tw_sock_TCPv6 3072 3072 100% 0.03K 24 128 96K jbd2_revoke_record_s 2040 2040 100% 0.05K 24 85 96K jbd2_journal_handle 1838 1702 92% 2.84K 168 11 5376K task_struct 1803 1770 98% 2.06K 123 15 3936K sighand_cache 1630 1540 94% 2.00K 108 16 3456K kmalloc-2048Any idea please?
Thanks all.
Answer
Java is using 40% memory, those processes run inside the JVM and have different process scheduler
jps | grep JAVA_NAME | awk ‘{print $1}’
Also how much memory are you giving to Java, it could be claiming that memory on start up, I think it uses its own memory manager
Attribution
Source : Link , Question Author : ZTE.A , Answer Author : Sum1sAdmin
