Liferay tomcat(atomikos) threads are blocked

160 views Asked by At

I have an application (Liferay + Tomcat + PostgreSQL). Tomcat also has configuration for Atomikos as transaction manager:

<Resource 
    name="jdbc/liferay"
    auth="Container"
    type="com.atomikos.jdbc.AtomikosDataSourceBean"
    factory="com.atomikos.tomcat.EnhancedTomcatAtomikosBeanFactory"
    uniqueResourceName="jdbc/liferay"
    maxPoolSize="500"
    minPoolSize="10"
    borrowConnectionTimeout="30"
    maxIdleTime="60"
    reapTimeout="0"
    maintenanceInterval="60"
    testQuery="SELECT 1"
    xaDataSourceClassName="org.postgresql.xa.PGXADataSource"
    xaProperties.serverName="${host}"           
    xaProperties.portNumber="${port}"           
    xaProperties.user="${user}"           
    xaProperties.databaseName="${database}"
    xaProperties.password="${password}"  
    />
    
<Resource 
    name="jdbc/service"
    auth="Container"
    type="com.atomikos.jdbc.AtomikosDataSourceBean"
    factory="com.atomikos.tomcat.EnhancedTomcatAtomikosBeanFactory"
    uniqueResourceName="jdbc/service"
    maxPoolSize="100"
    minPoolSize="10"
    borrowConnectionTimeout="30"
    maxIdleTime="60"
    reapTimeout="0"
    maintenanceInterval="60"
    testQuery="SELECT 1"
    xaDataSourceClassName="org.postgresql.xa.PGXADataSource"
    xaProperties.serverName="${host}" 
    xaProperties.portNumber="${port}"           
    xaProperties.databaseName="service"
    xaProperties.user="${user}"           
    xaProperties.password="${password}" 
  />

At random moments (it can be few times in hour) application gets stuck and doesn't process any responses.

I got threads dump and every time when application got stuck, there is a spike of threads. You can check it on screenshots here and here. Most of this threads are in BLOCKED state e.g.:

"http-apr-443-exec-163" - Thread t@17821
   java.lang.Thread.State: BLOCKED
    at com.atomikos.datasource.pool.ConnectionPool.borrowConnection(ConnectionPool.java:127)
    - waiting to lock <8e7fbf9> (a com.atomikos.datasource.pool.ConnectionPool) owned by "http-apr-443-exec-10" t@223

But this thread (http-apr-443-exec-10) is in RUNNABLE state:

"http-apr-443-exec-10" - Thread t@223
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:171)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:161)
    at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:128)
    at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:113)
    at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
    at org.postgresql.core.PGStream.receiveChar(PGStream.java:443)
    at org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:505)
    at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:149)
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213)
    at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)
    at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:223)
    at org.postgresql.Driver.makeConnection(Driver.java:465)
    at org.postgresql.Driver.connect(Driver.java:264)

Any ideas what can it be? Deadlock or miss configuration of Atomikos?

0

There are 0 answers