[MGNLACTIVATION-121] Prevent memory leak of temp files references at activation transporter Created: 08/Dec/15  Updated: 09/Jan/17  Resolved: 04/Feb/16

Status: Closed
Project: Activation
Component/s: None
Affects Version/s: 5.4.2
Fix Version/s: 5.4.3

Type: Bug Priority: Major
Reporter: Nicolas Barithel Assignee: Evzen Fochr
Resolution: Fixed Votes: 1
Labels: support
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Cloners
Template:
Acceptance criteria:
Empty
Task DoD:
[ ]* Doc/release notes changes? Comment present?
[ ]* Downstream builds green?
[ ]* Solution information and context easily available?
[ ]* Tests
[ ]* FixVersion filled and not yet released
[ ]  Architecture Decision Record (ADR)
Bug DoR:
[ ]* Steps to reproduce, expected, and actual results filled
[ ]* Affected version filled
Date of First Response:
Epic Link: deleteOnExit
Sprint: Kromeriz 29
Story Points: 3

 Description   

We have found a memory leak problem with the temporary activation files on Magnolia 4.4.4.

Two conditions are required to trigger the problem :

  • The Magnolia server is not restarted during a long time (a few weeks)
  • A lot of content is activated continuously

After some time, the Magnolia Author get really slow until it does not respond anymore and client restart it.
We have investigated this problem for some time, first we have lower down the Xmx so outOfMemory error could be throw, and we have configure the JVM to perform heapDump on outOfMemory error.

We got our hand on a heapDump last week and we found that most of the memory where occupy by a large map hold by "java.io.DeleteOnExitHook" that is referencing temporary files created by Magnolia (about 200 000 references)

We found the origin of this leak, in the magnolia-exchange-activation-module, version 4.4.4, there is a class named "Transporter" with a method "prepareTempFile". The temporary activation files are created in the method and are referenced to be deleted when the JVM terminates (Method deleteOnExit of File class) and these refereces are not deleted when the file is deleted by Magnolia.
This problem is referenced in sereral threads (http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6664633, http://bugs.java.com/bugdatabase/view_bug.do?bug_id=4872014, http://bugs.java.com/bugdatabase/view_bug.do?bug_id=4809375)

We are going to correct this problem by removing the deleteOnExit call on the temp file.

But in the activation module version 5.4.2, we can see that the method prepareTempFile in the ActivationContent class still call the method "tempFile.deleteOnExit();" (line 82), and as far as we know, this problem with the method "deleteOnExit" is still present in more recent versions of the JVM (7 or 8).



 Comments   
Comment by Hans Ardon [ 13/Jan/16 ]

Hi,

Unfortunately we have experienced this matter (author instance getting slower and slower) for several times now, but last night we did not even get to restarting the server. Apparently one of our customers had started the activation process with a lot of files. We really think that the problem described by Soisik caused a shutdown of Tomcat on production. Do you have any idea when this problem can be fixed? And in the meantime, what can we do to prevent this from happening again? Should we restart the author instance every night until the problem has been fixed?

Please advice, regards,

Hans

Comment by Soisik Froger [ 13/Jan/16 ]

Quick precisions on this issue :

  • this memory leak issue is not causing any crashes in production : it turned out this issue had nothing to do with the crashes of our customer server, since it was a different memory problem, related to permGemSpace.
  • nevertheless, we would like to see this issue fixed in next versions, because when Magnoloa register a file for deletion on exit, it is eating a (tiny) bit of heap memory every time activation are made, so theorically it could lead to heap memory saturation if server is not restarted for a long time.

@Hans : I think you may want to investigate further your memory problem : if it is a memory problem, i would advise you use a heapDump taken during an OutOfMemory incident, so you can check whether heap memory is fully occupied with the content of java.io.DeleteOnExitHook map or if you have an other issues.

Comment by Hans Ardon [ 13/Jan/16 ]

Thanks for your comment Soisik, we will check the heapDump.

Generated at Sun Feb 11 22:59:53 CET 2024 using Jira 9.4.2#940002-sha1:46d1a51de284217efdcb32434eab47a99af2938b.