JVM heap is being swapped causing GC go haywire – Tomcat

by

Info:

# of VMs= 4, each with an instance of Tomcat 8.5.* in cluster
Apps = 5 war applications- 2 UI applications and 3 Webservices.
Java Version = java 1.8.*
Configuratin = 2 LTMS and 2 Apache Webservers – 1 LTM on top of tomcat clusters which processes Webservice requests (200K+ a day during weekdays). Another LTM on top of Webserver which is also on top on Tomcat instances which handles the UI application requests (10K+ requests a day).
JVM parameters: all default with -Xms3072m -Xmx3072

Tomcat configuration:

 Connector port="xxxx"                 
       protocol="HTTP/1.1"  
           connectionTimeout="3000"  
           enableLookups="false"  
           redirectPort="yyyy"  
           maxThreads="80"   
Connector port="yyyy"  
          protocol="org.apache.coyote.http11.Http11NioProtocol"  
          redirectPort="yyyy"  
          secure="true"    
          scheme="https"  
          clientAuth="false"  
          sslProtocol="TLS"   
          sslEnabledProtocols="SSLv2Hello,TLSv1,TLSv1.1,TLSv1.2"  
          SSLEnabled="true"  
          maxThreads="70"  
          maxKeepAliveRequests="100"  
          keepAliveTimeout="5000"  
          connectionTimeout="10000"  
          keystoreFile="....."  
          keyPass="..."  
          keystorePass="..."  
          keyAlias="....."  
          truststoreFile="..."  
          truststorePass="..."  
          ciphers="......."        
   Connector port="zzzz"
     scheme="https"     
     protocol="AJP/1.3"     
     redirectPort="yyyy"

Issue: We are having to recycle tomcat every once a week, and we do it during the weekend. IF not recycled, on the 7th or 8th day of uptime, minor GC time goes up to anywhere from 5 sec to 30 sec, if still not restarted then whenever major GC happens it takes a minute atleast causing several failed transactions. Up on checking the VM status on all 4 nodes, during this time, we see a lot of swapping happening. Memory utilization is under 55% the whole time, also cpu utilization is below 25%. Surprisingly, this happens during the weekend when there is very little to no load. We have not ever seen any OOM errors, so far it appears that heap tuning is a non-issue (I may be wrong). We also have same configuration in a production simulation environment where the load is not as much as production servers, and there is no such swapping/GC problems in that environment.
Any insight or any advice on this would be great help. Please let me know if any other info is needed.

Answer

Swapping is due to inactive memory pages, objects that not used for a while but not subject to GC, try decrease system swapiness to 10%.

Also check code cache utilization before recycling tomcat it may cause slowness with time.

Try to change GC to g1gc and check, it should have less STW pauses.

Attribution
Source : Link , Question Author : prashma , Answer Author : hoshoh

Leave a Comment