Centos Server Recently slowed down significantly

I have Dedicated Server on 1and1 with

RAM - 16 GB
HD 1000GB
CPU: Intel(R) Xeon(R) CPU E3-1270 v3 @ 3.50GHz, 4 cores
Centos 6
Apache/2.2.15
PHP 5.3.3
Mysql 

I have a video sharing website (on joomla) with about 5000 media items
3500 media items on Amazon and rest is on server (about 489GB) and recently website slowed down significantly. THe pages started to load for 5 – 10 seconds, backend about 10 – 20 seconds. 2 – 3 days before it worked much better, not perfect but much better.

I need to identify the problem, myabe you can suggest some commands for tracking performance wich would help me to find the problem?? Because something really went wrong with server.

———-‘top’ result————–

top - 10:35:45 up 17:57,  1 user,  load average: 41.21, 46.46, 55.28
Tasks: 327 total,   7 running, 320 sleeping,   0 stopped,   0 zombie
Cpu(s): 62.6%us,  0.0%sy,  0.0%ni, 28.8%id,  8.6%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16294080k total, 15210716k used,  1083364k free,    55924k buffers
Swap:  4194288k total,    30268k used,  4164020k free, 11651236k cached


1457 apache    20   0 1167m 157m 4632 S 34.2  1.0  16:17.50 ffmpeg
614 root      20   0  265m  56m 4260 R 33.2  0.4  20:00.14 ffmpeg
1059 root      20   0  265m  55m 4260 R 33.2  0.3  16:52.33 ffmpeg
2733 apache    20   0  266m  55m 4256 R 33.2  0.4   5:25.97 ffmpeg
2786 apache    20   0  266m  56m 4264 R 33.2  0.4   4:50.05 ffmpeg
2854 root      20   0  147m  25m 3580 R 33.2  0.2   3:25.02 ffmpeg
29788 root      20   0  267m  56m 4268 R 33.2  0.4  34:31.32 ffmpeg
31612 root      20   0  266m  55m 4264 R 33.2  0.4  27:22.21 ffmpeg
32589 root      20   0  266m  57m 4264 R 33.2  0.4  24:25.37 ffmpeg
2625 apache    20   0  265m  54m 4256 R 32.9  0.3   6:18.87 ffmpeg
2895 apache    20   0  266m  57m 4260 R 32.9  0.4   2:22.71 ffmpeg
28797 root      20   0  267m  58m 4268 R 32.9  0.4  46:03.94 ffmpeg
32297 root      20   0  266m  57m 4264 R 32.9  0.4  25:24.76 ffmpeg
28343 root      20   0  268m  57m 4268 R 32.6  0.4  54:09.69 ffmpeg
1436 root      20   0  265m  55m 4260 R 32.2  0.4  15:02.93 ffmpeg
2138 root      20   0  265m  55m 4260 R 32.2  0.3  10:41.26 ffmpeg
30435 root      20   0  266m  56m 4264 R 32.2  0.4  30:41.72 ffmpeg
2647 apache    20   0  266m  57m 4272 R 31.9  0.4   6:15.76 ffmpeg
2978 root      20   0  144m  22m 3572 R 31.9  0.1   1:15.28 ffmpeg

[root@u17669867 ~]# iostat
Linux 2.6.32-431.20.5.el6.x86_64 (u17669867)    08/15/2014      _x86_64_        (8 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          78.37   15.62    0.38    2.05    0.00    3.58

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sdc               8.57       495.09      2368.59   33674022  161101560
sda               8.41        30.21       171.06    2054642   11634608
sdd               6.85       390.15      2366.70   26536668  160972704
sdb               4.84         2.84       135.32     193088    9203664
md3               4.35        20.51        27.92    1395238    1898808
md1               5.24         2.25        41.40     153236    2816112
md11            303.70       885.20      2368.26   60207426  161079040
dm-0            303.66       885.19      2368.26   60206658  161079048
dm-1              0.12         4.33         0.01     294802        720
dm-2              3.70        16.15        27.91    1098146    1898064
dm-3              0.00         0.02         0.00       1602         24

Answer

Start by tracing what is taking too long with Chrome/IE developer tools (network tab).

Confirming that the slowness is happening for all requests irrespective of type, trace the path from end user to your server (traceroute) looking for any excessive delays. Test from multiple places.

On the server, check the number of open connections (netstat -antp | grep httpd), you may be hitting a limit of concurrent users and the connections/requests get queued up waiting for a slot.

Also check CPU (top, ps aux), memory (free -m) and disk usage (iostat).

EDIT: Your top results show CPU starvation and the iostat results aren’t any better. Consider adding another server or move the CPU-intensive work away from the web server.

Attribution
Source : Link , Question Author : user1584043 , Answer Author : Giovanni Tirloni

Leave a Comment