My system is a software network intercepter, which runs on a Ubuntu 14.04.4 LTS (GNU/Linux 3.16.0-71-generic x86_64).
I have it deployed in several sites, yet a specific one gives me the following weird problem.
During the system’s operation it consumes a lot of memory, until it fills the entire swap up and becomes very low. Mind you, that is not one of our most input intense installations.
My problem is that when I open up htop, the amount of memory used by processes doesn’t add up.
So, I have turned off all of my system’s processes. Furthermore, we use MongoDB, which uses up a lot of memory, so I have turned that off as well.
After turning off everything, and the CPU goes down to virtually zero, there are still 20GB used, seemingly leaked due to some kernel issue.
So I turn to /proc/meminfo, yet the results simply do not tell me anything regarding where the missing memory is:MemTotal: 32948744 kB MemFree: 10695472 kB MemAvailable: 10925256 kB Buffers: 34780 kB Cached: 294564 kB SwapCached: 564 kB Active: 262064 kB Inactive: 158464 kB Active(anon): 82344 kB Inactive(anon): 8896 kB Active(file): 179720 kB Inactive(file): 149568 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 16777180 kB SwapFree: 16753824 kB Dirty: 11816 kB Writeback: 0 kB AnonPages: 90708 kB Mapped: 32500 kB Shmem: 56 kB Slab: 168804 kB SReclaimable: 138864 kB SUnreclaim: 29940 kB KernelStack: 3040 kB PageTables: 4600 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 33251552 kB Committed_AS: 538280 kB VmallocTotal: 34359738367 kB VmallocUsed: 203084 kB VmallocChunk: 34359505748 kB HardwareCorrupted: 0 kB AnonHugePages: 18432 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 65472 kB DirectMap2M: 33488896 kBAccording to what I’ve read online regarding the output of meminfo, the numbers simply don’t add up – there is no big consumer of memory.
Additionally, here are the top memory users according to htop:
840 _statsd 20 0 1181M 22672 4024 S 0.0 0.1 5h54:36 statsd /etc/statsd/localConfig.js 1302 _graphite 20 0 145M 16044 2164 S 0.0 0.0 4h55:38 /usr/bin/python /usr/bin/carbon-cache --c 837 _statsd 20 0 1178M 13892 3100 S 0.0 0.0 9:15.03 /usr/bin/nodejs proxy.js /etc/statsd/prox 20639 root 20 0 101M 6216 5244 S 0.0 0.0 0:00.03 sshd: paf [priv] 20820 root 20 0 101M 6196 5228 S 0.0 0.0 0:00.02 sshd: paf [priv] 20870 paf 20 0 22504 5208 3268 S 0.0 0.0 0:00.20 -bash 20732 paf 20 0 22408 4860 3020 S 0.0 0.0 0:00.25 -bash 20869 paf 20 0 101M 3692 2732 S 0.0 0.0 0:00.01 sshd: paf@pts/3 20731 paf 20 0 101M 3636 2672 S 0.0 0.0 0:00.12 sshd: paf@pts/1 21101 paf 20 0 26060 3300 2404 R 1.4 0.0 0:00.41 htop 1 root 20 0 33976 2744 1292 S 0.0 0.0 0:34.74 /sbin/initEdit (extra diagnostics):
~$ free -m total used free shared buffers cached Mem: 32176 23619 8556 0 130 838 -/+ buffers/cache: 22650 9526 Swap: 16383 37 16346slabtop:
Active / Total Objects (% used) : 631869 / 707961 (89.3%) Active / Total Slabs (% used) : 23836 / 23836 (100.0%) Active / Total Caches (% used) : 70 / 102 (68.6%) Active / Total Size (% used) : 174386.89K / 194368.76K (89.7%) Minimum / Average / Maximum Object : 0.01K / 0.27K / 8.00K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 221169 198528 89% 0.10K 5671 39 22684K buffer_head 123690 121130 97% 0.19K 5890 21 23560K dentry 82768 81847 98% 0.98K 5173 16 82768K ext4_inode_cache 50708 31368 61% 0.57K 1811 28 28976K radix_tree_node 39372 30686 77% 0.04K 386 102 1544K ext4_extent_status 31790 31790 100% 0.12K 935 34 3740K kernfs_node_cache 14091 6729 47% 0.19K 671 21 2684K kmalloc-192 13376 8603 64% 0.06K 209 64 836K kmalloc-64 13312 11934 89% 0.03K 104 128 416K kmalloc-32 11776 11776 100% 0.03K 92 128 368K jbd2_revoke_record_s 8500 7085 83% 0.05K 100 85 400K shared_policy_node 8064 8064 100% 0.07K 144 56 576K Acpi-ParseExt 7700 7700 100% 0.57K 275 28 4400K inode_cache 7680 7680 100% 0.01K 15 512 60K kmalloc-8 7424 7424 100% 0.02K 29 256 116K kmalloc-16 6928 3739 53% 0.25K 433 16 1732K kmalloc-256 5712 5265 92% 0.08K 112 51 448K anon_vma 5088 4025 79% 0.12K 159 32 636K kmalloc-128 4992 4992 100% 0.06K 78 64 312K ext4_free_data 4312 4312 100% 0.07K 77 56 308K ext4_io_end 3948 3948 100% 0.09K 94 42 376K kmalloc-96 3876 3876 100% 0.04K 38 102 152K Acpi-Namespace 3575 2475 69% 0.63K 143 25 2288K proc_inode_cache 2232 2124 95% 0.11K 62 36 248K jbd2_journal_head 2016 2016 100% 0.14K 72 28 288K ext4_groupinfo_4k 1888 1459 77% 0.50K 118 16 944K kmalloc-512 1804 1804 100% 0.36K 82 22 656K blkdev_requests 1575 1479 93% 0.31K 63 25 504K nf_conntrack_ffffffff81cd3800 1512 1512 100% 0.11K 42 36 168K task_delay_info 1400 1400 100% 0.62K 56 25 896K sock_inode_cache 1232 1158 93% 1.00K 77 16 1232K kmalloc-1024 1104 1104 100% 0.09K 24 46 96K ftrace_event_file 1104 1048 94% 0.66K 46 24 736K shmem_inode_cache 1020 1020 100% 0.02K 6 170 24K numa_policy 903 903 100% 0.38K 43 21 344K scsi_cmd_cache 896 850 94% 2.00K 56 16 1792K kmalloc-2048 819 819 100% 0.10K 21 39 84K blkdev_ioc 810 810 100% 1.06K 27 30 864K UDPv6 732 403 55% 2.55K 61 12 1952K task_struct 560 544 97% 0.25K 35 16 140K dquot 510 510 100% 0.08K 10 51 40K Acpi-State 510 510 100% 0.05K 6 85 24K jbd2_journal_handle 504 504 100% 1.75K 28 18 896K TCP 476 476 100% 1.12K 17 28 544K signal_cache 368 368 100% 0.25K 23 16 92K tw_sock_TCP 320 320 100% 1.94K 20 16 640K TCPv6 315 315 100% 2.05K 21 15 672K idr_layer_cache 315 276 87% 2.06K 21 15 672K sighand_cache 288 288 100% 0.25K 18 16 72K tw_sock_TCPv6 256 256 100% 0.06K 4 64 16K kmem_cache_node 256 256 100% 0.02K 1 256 4K jbd2_revoke_table_s 234 234 100% 0.88K 13 18 208K mm_struct 225 225 100% 0.62K 9 25 144K files_cache 219 219 100% 0.05K 3 73 12K ip_fib_trie 204 204 100% 0.94K 12 17 192K UNIX 192 168 87% 4.00K 24 8 768K kmalloc-4096 192 192 100% 0.12K 6 32 24K ext4_allocation_context 150 150 100% 0.16K 6 25 24K sigqueueCould anyone please give me a lead regarding how to investigate this issue further?
Answer
Attribution
Source : Link , Question Author : borisdan , Answer Author : Community
