df command not showing correct values

by

This is a RHEL server, I’m running a MySQL server on it, the database and log files (however, logging is disabled) are located on the /srv directory where plenty of space is available.

Recently I had a crashed table, so I tried fixing it but on the next day I found that MySQL can’t respond to many of queries with an error indicating that there is no disk space:

ERROR 1030 (HY000): Got error 28 from storage engine

So I ran the following command to see what’s taking space

[root@tms /]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/server-slash
                      9.9G  9.5G     0 100% /
tmpfs                 7.8G     0  7.8G   0% /dev/shm
/dev/sda1             485M   58M  402M  13% /boot
/dev/mapper/server-var
                      739G  252G  450G  36% /srv

Surprisingly it’s the / directory. But more surprisingly is that the directories under / do not indicate used space more than 2 GB, while df shows total space 9.9 GB for /.

[root@tms /]# du -sh /*
7.5M    /bin
48M     /boot
200K    /dev
24M     /etc
4.0K    /home
223M    /lib
21M     /lib64
16K     /lost+found
4.0K    /media
4.0K    /mnt
183M    /opt
...deleted some file-not-found errors for files under /proc
0       /proc
144K    /root
14M     /sbin
4.0K    /selinux
252G    /srv
0       /sys
44K     /tmp
917M    /usr
259M    /var

So why does df show wrong values ? And how can I findout what’s actually taking space ?

Answer

A common programming technique is to create a temporary file and immediately unlink() it. This leaves the file (and its space) available for the duration of the program but automatically causes its removal when the program using it terminates. One advantage is that no epilog (cleanup) code is necessary to write.

To determine if you have a process holding an unlinked file open, do:

lsof -a +L1 /dev/server_slash

(or)

lsof +D /dev/server_slash +L1

Look for any files with an NLINK value of zero (0). These would be files with a zero link count that will vanish when the last process terminates. The SIZE/OFFSET column will offer the character size of the file in question.

Attribution
Source : Link , Question Author : Muhammad Gelbana , Answer Author : JRFerguson

Leave a Comment