Need better exception handling for transaction errors
When we get a transaction timeout ("rolled back in a different thread" triggered by Transaction Reaper), it produces an incredible number of follow-on errors, and can make the server unusable.
We need to find a way of reducing the severity of such problems. There must be something wrong with our exception handling. We should check our transaction handling code (eg from DeltaSpike, and TransactionPhaseListener) to ensure that transaction timeouts/rollbacks are handled by the book: https://docs.jboss.org/hibernate/orm/5.0/manual/en-US/html/ch13.html
From what I can tell, DeltaSpike's BeanManagedUserTransactionStrategy (which our NoNestingTransactionStrategy extends) does the right thing, that is BeanManagedUserTransactionStrategy.UserTransactionAdapter.rollback() does call UserTransaction.rollback() when status is STATUS_ROLLEDBACK (as it should).
It could well be that our threads have JTA stuck with STATUS_ROLLEDBACK because the original thread never calls commit or rollback.
Our TransactionPhaseListener is probably doing the wrong thing. To be safe, we should also check DeltaSpike's TransactionStrategy (and our NoNestingTransactionStrategy).