[MAGNOLIA-7326] Improve import & export functions to avoid OOM error Created: 31/May/18  Updated: 20/Aug/21  Resolved: 16/Oct/19

Status: Closed
Project: Magnolia
Component/s: None
Affects Version/s: 5.5.4, 5.6.5
Fix Version/s: 5.5.16, 5.6.12, 5.7.6, 6.1.3, 6.2

Type: Bug Priority: Neutral
Reporter: Viet Nguyen Assignee: Michael Duerig
Resolution: Fixed Votes: 2
Labels: export
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File jstat.log    
Issue Links:
Relates
relates to MAGNOLIA-7191 Out of heapspace when publish 100MB b... Closed
relates to MAGNOLIA-7890 Exporting content larger approx. 5mb ... Closed
relates to MGNLIMG-213 Asset upload in ZIP file to DAM may c... Closed
causality
duplicate
duplicates MGNLDAM-717 Cannot export a file with large size,... Closed
relation
is related to MAGNOLIA-7655 OOME when exporting to YAML Closed
is related to MAGNOLIA-7652 OOME when exporting to YAML Closed
is related to MAGNOLIA-7702 Export of large files fails on Java 1... Closed
Template:
Acceptance criteria:
Empty
Task DoD:
[ ]* Doc/release notes changes? Comment present?
[ ]* Downstream builds green?
[ ]* Solution information and context easily available?
[ ]* Tests
[ ]* FixVersion filled and not yet released
[ ]  Architecture Decision Record (ADR)
Bug DoR:
[ ]* Steps to reproduce, expected, and actual results filled
[ ]* Affected version filled
Release notes required:
Yes
Date of First Response:
Epic Link: Support
Sprint: UI Framework 8
Story Points: 5

 Description   

There have been more and more customers reporting Out of memory when exporting their data while using our JCR export function. Let's find a way to remedy this situation.
Expectation:
System just use its allowed heap space to export data and not run out of memory.



 Comments   
Comment by Michael Duerig [ 27/Sep/19 ]

The JcrExportCommand relies on JDOM, which builds an in memory representation of the exported workspace. This is the most likely causing the OOME for big workpaces.

AFAICS, fixing this would require re-implementing the export functionality without relying on JDOM.

Comment by Roman Kovařík [ 27/Sep/19 ]

While you're already on it, could you prove it (by commenting out that part of the code)?

Comment by Michael Duerig [ 30/Sep/19 ]

See this commit for a POC reproducing the problem:

Run info.magnolia.importexport.command.JcrExportCommandTest.exportOOME and observe jstat -gcutil (see jstat.log ). Use -DJDOM.off=true to disable accumulating the XML in JDOMResult and observer the difference. In the former case the system gets stuck in full gc cycles an does not advance beyond handling approximately 60k nodes. In the latter case all 100k nodes are handled in just about 10 seconds.

Comment by Roman Kovařík [ 30/Sep/19 ]

I don't have access to your repo but feel free to create PR where anybody can join the discussion.

Comment by Michael Duerig [ 30/Sep/19 ]

PR here.

Comment by Michael Duerig [ 03/Oct/19 ]

asiska, rkovarik, I just submitted PR #746 with a proposed fix for this issue. Please have a look once you get around to it.

Comment by Michael Duerig [ 16/Oct/19 ]

Merged https://git.magnolia-cms.com/projects/PLATFORM/repos/main/pull-requests/750/overview

Generated at Mon Feb 12 04:22:47 CET 2024 using Jira 9.4.2#940002-sha1:46d1a51de284217efdcb32434eab47a99af2938b.