Progressively degrading performance saving MT to database

Description

MT performance seems to worsen the longer it continues, for larger documents.
The time it takes to process the translations in a (Hibernate?) request at org.zanata.service.impl.TranslationServiceImpl#translate(...) seems to increase from 500~1000 every use, suggesting there's a cache or leak in this area.

The debugging shows:

Explanation: (^^ means return to)

org.zanata.service.impl.MachineTranslationServiceImpl#addMachineTranslationsToDoc
Sending batch 0-1000 sending a batch of 1000 textflows to MT provider
Received response [ batch size] (time) time taken to receive the result

org.zanata.service.impl.MachineTranslationServiceImpl#saveTranslationsInBatches
Saving in a batch immediately before the batch loop
Saving batch x-y immediately before sublist
Transaction... immediately before try { transaction.run(() ->

org.zanata.service.impl.MachineTranslationServiceImpl#makeUpdateRequestsForBatch
Creating save requests... before process loop
Done creating (time) time taken to create the list of TransUnitUpdateRequest

org.zanata.service.impl.MachineTranslationServiceImpl#saveTranslationsInBatches ^^
Committing translations... immediately before translationService.translate(targetLocale.getLocaleId(), updateRequests);

org.zanata.service.impl.TranslationServiceImpl#translate(org.zanata.common.LocaleId, java.util.List<org.zanata.webtrans.shared.model.TransUnitUpdateRequest>, boolean)
Processing translations... before for (TransUnitUpdateRequest request : translationRequests) {
Done processing (time) time taken to process the requests in loop

org.zanata.service.impl.MachineTranslationServiceImpl#saveTranslationsInBatches ^^
Done committing translations. immediately after translationService.translate
Done transaction 100-100(time) loop iteration finished
Done batches all results processed

Environment

None

Status

Assignee

Unassigned

Reporter

Damian Jansen

Labels

Tested Version/s

None

Components

Sprint

None

Fix versions

Affects versions

Priority

High
Configure