Jboss 4.2.3 Application with huge number of anonymous Threads

Question

We maintain a legacy (jboss 4.2.3 , ejb 3 ,jsp, jdk 1.6.45 64b ) application.

This application make heavy usage of database, and respond to

• User Http request
• Web service request
• Some CORBA connections
• Some external JSM call
• Some MDB internal process

Http connection pool is standard (tomcat 5.5 with 250 connections) , database pool connection is raised to 300 (instead of 60 standard).

The jboss service.xml (where default thread pool isdefined) is

<mbean code="org.jboss.util.threadpool.BasicThreadPool"name="jboss.system:service=ThreadPool">
  <attribute name="Name">JBoss System Threads</attribute>
  <attribute name="ThreadGroupName">System Threads</attribute>
  <attribute name="KeepAliveTime">60000</attribute>
  <attribute name="MaximumPoolSize">10</attribute>
  <attribute name="MaximumQueueSize">1000</attribute>
  <attribute name="BlockingMode">run</attribute>

One of our customer have some bad performance during part of the day.
There is a lot of threads generated without any relation with the users activities.

Normal activities threads count is 300/400 threads

• Sometimes it grows up to 4 k in seconds and decrease slowly in minutes
• Sometimes it can grows up to 25 k threads and takes hours to decrease and application become unresponsive (need a restart)

Thread dump show that all this threads are RUNNABLE , but there is no stack trace (other "normal" thread show usual stack trace).

All those threads are nammed "Thread-xxxxxx" and belong to the "JBoss Pooled Threads"

We are sure that those threads are not created by the application, but we cannot find what could be responsible for this threads generation.

Any ideas ?

Answer 1

Resolved ! The problem was related to the SQL SERVER Jdbc Driver, our code use a setQueryTimeout instruction on a Statement , there is a bug in the driver that cause it to create a thread for each query to manage the timeout.

see : https://connect.microsoft.com/SQLServer/feedbackdetail/view/785983/jdbc-additional-thread-for-every-query-when-using-setquerytimeout
https://connect.microsoft.com/SQLServer/feedback/details/669427/property-to-create-timer-threads-per-statement-or-connection-in-jdbc-driver

Removing the setQueryTimeout instruction solve the problem.

HTH !