tomcat8: ‘free(): corrupted unsorted chunks’

by

We upgraded from Ubuntu 16LTS to Ubuntu 18LTS a couple of days ago and since then have had Apache Tomcat crash completely twice (once per day, about 25 hrs apart). The java process stops running.

One line is added to the log file at this point:

free(): corrupted unsorted chunks

(that’s it – no timestamp)

It doesn’t seem to correlate to any particular action happening on the server at those times, at least it’s hard to tell with only two instances, but my suspicion is it may have something to do with garbage collection.
That’s because the server monitoring shows memory usage for the java process drop from 7.90GB to 0.93GB within a minute at that point (in actual fact the java process quits so it’s possible that smaller amount is after I manually restarted tomcat). The server’s max memory setting -Xmmx is set to 8GB and gradually before the crash it creeped up to just below that through the course of the day.

Furthermore, the very first lines of the tomcat log are

NOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

In /etc/default/tomcat8 I have the line 

JAVA_OPTS="-Djava.awt.headless=true -Xmx8g -XX:+UseConcMarkSweepGC"

That UseConcMarkSweepGC option was added back on Ubuntu 16 where tomcat7 was used and I seem to remember it was recommended. In fact the config. file still says 

'Use "-XX:+UseConcMarkSweepGC" to enable the CMS garbage collector (improved response time)

Anyway I’ve now removed it in the hope that that was the issue. We will see if there’s a crash tomorrow but in the meantime does that seem reasonable? Can anyone suggest any steps to validate this hypothesis or debug further, or any other ideas for what might be happening?

Answer

In case anyone else has this issue, thanks to the helpful java-core-libs mailing list, I found the cause was the use of the APR/native libraries in Tomcat. Uninstalling these (apt-get remove libapr1) and removing configuration references from server.xml resolved the issue

Attribution
Source : Link , Question Author : Oliver Kohll , Answer Author : Oliver Kohll

Leave a Comment