Tomcat threads number inexplicably peaking immediately after end of load test

I am testing an app on Tomcat with one hour of crazy traffic from 200 users.

Everything is fine during the test.
But immediately AFTER the test (no more traffic), the number of threads rises inexplicably.
After a short but dangerous peak, it starts to decrease.

I computed an histogram to show the problem:

The yellow line (number of socketRead threads in Tomcat’s thread pool) looks stable at first, but then jumps at the end of traffic (black vertical line).

QUESTION: Have you already seen this behavior, any idea what could be causing it?

Sometimes in production this phenomenon makes Tomcat reach maxThreads, causing all sorts of other problems.

Cluster of 5 nodes, Tomcat 6.0.26 / JK / Alfresco

Details about the other kinds of threads, if needed:

• Red: socket read threads outside thread pool
• Blue: threads waiting in thread pool
• Green: java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill
• Purple: Application-side DAO threads (the app is Alfresco, used as a WebDAV server)

PS: If you have any interest in generating such graphs, I can make my tool open source, let me know.

Answer

Attribution
Source : Link , Question Author : Nicolas Raoul , Answer Author : The Unix Janitor