Sorting Projects list leads to "Transaction was rolled back in a different thread!" and many other errors

Description

On translate.zanata.org (which has a large number of projects), sorting the Projects table (by clicking on a column header) eventually leads to a transaction timeout and many varied errors on subsequent requests, all of which seem to have the same cause.

javax.persistence.TransactionRequiredException: No active JTA transaction on joinTransaction call
javax.resource.ResourceException: IJ000459: Transaction is not active: tx=TransactionImple < ac, BasicAction: 0:ffff0a196a24:731414b0:57a9f41b:54ed8 status: ActionStatus.ABORTED >
org.hibernate.HibernateException: Logical connection is closed
org.hibernate.HibernateException: Transaction was rolled back in a different thread!
javax.resource.ResourceException: IJ000460: Error checking for a transaction

I think this may be the most important stack trace, because it shows that a richfaces component triggered an in-memory merge sort, during which a database operation was invoked (very bad):

org.hibernate.HibernateException: Transaction was rolled back in a different thread!
	at org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorTrackingImpl.processAnyDelayedAfterCompletion(SynchronizationCallbackCoordinatorTrackingImpl.java:105)
	at org.hibernate.internal.SessionImpl.delayedAfterCompletion(SessionImpl.java:636)
	at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1236)
	at org.hibernate.internal.QueryImpl.list(QueryImpl.java:101)
	at org.zanata.dao.ProjectDAO.getOffsetListOrderByName(ProjectDAO.java:92)
	at org.zanata.dao.ProjectDAO$Proxy$_$$_WeldClientProxy.getOffsetListOrderByName(ProjectDAO$Proxy$_$$_WeldClientProxy.java)
	at org.zanata.action.ProjectPagedListDataModel.fetchPage(ProjectPagedListDataModel.java:50)
	at org.zanata.action.PagedListDataModel.getRowData(PagedListDataModel.java:114)
	at org.ajax4jsf.model.SequenceDataModel.getRowData(SequenceDataModel.java:124)
	at org.richfaces.model.ArrangeableModel.compare(ArrangeableModel.java:286)
	at org.richfaces.model.ArrangeableModel.access$200(ArrangeableModel.java:51)
	at org.richfaces.model.ArrangeableModel$2.compare(ArrangeableModel.java:255)
	at java.util.TimSort.mergeLo(TimSort.java:717)
	at java.util.TimSort.mergeAt(TimSort.java:514)
	at java.util.TimSort.mergeCollapse(TimSort.java:439)
	at java.util.TimSort.sort(TimSort.java:245)
	at java.util.Arrays.sort(Arrays.java:1512)
	at java.util.ArrayList.sort(ArrayList.java:1454)
	at java.util.Collections.sort(Collections.java:175)
	at org.richfaces.model.ArrangeableModel.sort(ArrangeableModel.java:253)
	at org.richfaces.model.ArrangeableModel.arrange(ArrangeableModel.java:207)
	at org.richfaces.component.UIDataTableBase.createExtendedDataModel(UIDataTableBase.java:207)
	at org.richfaces.component.UIDataAdaptor.getExtendedDataModel(UIDataAdaptor.java:467)
	at org.richfaces.component.UIDataAdaptor.setRowKey(UIDataAdaptor.java:280)
	at org.richfaces.renderkit.AbstractTableRenderer.encodeTableFacets(AbstractTableRenderer.java:118)
	at org.richfaces.renderkit.DataTableRenderer.doEncodeBegin(DataTableRenderer.java:306)
	at org.richfaces.renderkit.RendererBase.encodeBegin(RendererBase.java:116)
	at javax.faces.component.UIComponentBase.encodeBegin(UIComponentBase.java:822)
	at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1776)
	at com.sun.faces.context.PartialViewContextImpl$PhaseAwareVisitCallback.visit(PartialViewContextImpl.java:543)
	at org.richfaces.context.MetaComponentEncodingVisitCallback.visit(MetaComponentEncodingVisitCallback.java:83)
	at org.richfaces.context.BaseExtendedVisitContext.invokeVisitCallback(BaseExtendedVisitContext.java:103)
	at org.richfaces.context.ExtendedRenderVisitContext.invokeVisitCallback(ExtendedRenderVisitContext.java:65)
	at org.richfaces.component.UIDataAdaptor.visitTree(UIDataAdaptor.java:1330)
	at javax.faces.component.UIComponent.visitTree(UIComponent.java:1622)
	at javax.faces.component.UIForm.visitTree(UIForm.java:371)
	at javax.faces.component.UIComponent.visitTree(UIComponent.java:1622)
	at javax.faces.component.UIComponent.visitTree(UIComponent.java:1622)
	at javax.faces.component.UIComponent.visitTree(UIComponent.java:1622)
	at javax.faces.component.UIComponent.visitTree(UIComponent.java:1622)
	at com.sun.faces.context.PartialViewContextImpl.processComponents(PartialViewContextImpl.java:383)
	at com.sun.faces.context.PartialViewContextImpl.processPartial(PartialViewContextImpl.java:302)
	at org.richfaces.context.ExtendedPartialViewContext.processPartial(ExtendedPartialViewContext.java:264)
	at javax.faces.component.UIViewRoot.encodeChildren(UIViewRoot.java:973)
	at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1778)
	at com.sun.faces.application.view.FaceletViewHandlingStrategy.renderView(FaceletViewHandlingStrategy.java:426)
	at com.sun.faces.application.view.MultiViewHandler.renderView(MultiViewHandler.java:125)
	at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:286)
	at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:286)
	at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:286)
	at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:120)
	at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101)
	at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:139)
	at org.apache.deltaspike.jsf.impl.listener.request.DeltaSpikeLifecycleWrapper.render(DeltaSpikeLifecycleWrapper.java:111)
	at javax.faces.webapp.FacesServlet.service(FacesServlet.java:594)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
	at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:226)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
	at org.zanata.servlet.DuplicateParamFilter.doFilter(DuplicateParamFilter.java:69)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
	at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:832)
	at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:620)
	at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:553)
	at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:482)
	at org.ocpsoft.rewrite.servlet.impl.HttpRewriteResultHandler.handleResult(HttpRewriteResultHandler.java:42)
	at org.ocpsoft.rewrite.servlet.RewriteFilter.rewrite(RewriteFilter.java:297)
	at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:198)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
	at org.zanata.servlet.DuplicateParamFilter.doFilter(DuplicateParamFilter.java:69)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
	at org.zanata.servlet.JavaMelodyFilter.doFilter(JavaMelodyFilter.java:65)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
	at org.zanata.servlet.MDCInsertingServletFilter.doFilter(MDCInsertingServletFilter.java:76)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
	at org.zanata.servlet.RequestLoggingFilter.doFilter(RequestLoggingFilter.java:88)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149)
	at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)
	at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)
	at org.jboss.security.negotiation.NegotiationAuthenticator$WrapperValve.invoke(NegotiationAuthenticator.java:492)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:420)
	at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:150)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:854)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:654)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926)
	at java.lang.Thread.run(Thread.java:745)

In the short term, I think we need to disable sorting on the projects list.

Environment

None

Status

Assignee

Alex Eng

Reporter

Sean Flanigan

Labels

None

Tested Version/s

None

Time tracking

Sprint

None

Fix versions

server-3.9.4

Affects versions

server-3.9.3

Priority

High