[PUBLISHING-202] Not able to publish huge amount of items in timely fashion Created: 23/Mar/23  Updated: 03/Nov/23  Resolved: 13/Apr/23

Status: Closed
Project: Publishing
Component/s: None
Affects Version/s: 1.3.7
Fix Version/s: 1.4.0, 1.3.9

Type: Bug Priority: Neutral
Reporter: Roman Kovařík Assignee: Roman Kovařík
Resolution: Fixed Votes: 0
Labels: None
Σ Remaining Estimate: Not Specified Remaining Estimate: Not Specified
Σ Time Spent: 6.5h Time Spent: 6.5h
Σ Original Estimate: Not Specified Original Estimate: Not Specified

Attachments: Zip Archive contacts_xml_export.zip     File publishRequestPerFolder.groovy    
Issue Links:
Relates
relates to MAGNOLIA-8760 Usage of PostgreSQL to publish heavy ... Closed
relates to MAGNOLIA-9053 Improve scalabity of publishing content Open
Sub-Tasks:
Key
Summary
Type
Status
Assignee
PUBLISHING-204 Preint QA + PM Sub-task Completed Daniel Alonso  
PUBLISHING-205 Implementation Sub-task Completed Roman Kovařík  
PUBLISHING-216 Rw Sub-task Completed Daniel Alonso  
PUBLISHING-217 QA Sub-task Completed Quach Hao Thien  
PUBLISHING-218 Docu Sub-task Completed Roman Kovařík  
PUBLISHING-221 Docu rv Documentation Task Completed Martin Drápela  
PUBLISHING-229 QA on 6.3 content app Sub-task Completed Daniel Alonso  
Template:
Acceptance criteria:
Empty
Task DoD:
[X]* Doc/release notes changes? Comment present?
[X]* Downstream builds green?
[X]* Solution information and context easily available?
[X]* Tests
[X]* FixVersion filled and not yet released
[ ]  Architecture Decision Record (ADR)
Bug DoR:
[ ]* Steps to reproduce, expected, and actual results filled
[ ]* Affected version filled
Release notes required:
Yes
Documentation update required:
Yes
Epic Link: Performance problems with too many child nodes
Sprint: Nucleus 33, Nucleus 34
Story Points: 5
Team: Nucleus
Work Started:
Approved:
Yes

 Description   

Steps to reproduce

  1. Open a content app
  2. Import attached bootstrap file into a folder
  3. Click Publish recursive

Expected results

Content is published in acceptable time (minutes, depending)

Actual results

Publication takes hours, PostgreSQL might suffer from MAGNOLIA-8760 (to validate with other database).

Workaround

Decorate the app with:

subApps: 
  browser: 
    actions: 
      activateRecursive: 
        params: 
          itemTypes: nt:base

This might be not only a workaround but a proper way to publish huge amount of items.
This is applicable for content with total size in MB (the size of exported items).
This might be not applicable for binary content (e.g. assets, the size of exported items in GB).
This might not applicable for workflow (although we haven't received such use case), see the next section.

Development notes

PublicationCommand might do this with a configurable threshold parameter if needed e.g.

Recursive Publishing 1000 items which is above the threshold of 100, packing all nodes to reduce the amount of HTTP transactions.

If there is a use case including workflow

  • it's not possible to use this approach as workflow creates separate versions for each node so it's not possible to export a parent node including children.
  • we would need to create a wrapper which would collect these different versions so by exporting parent we would be exporting also children but redirecting to their proper versions.

 



 Comments   
Comment by Roman Kovařík [ 28/Mar/23 ]

Discovery:

To check if the suggested approach might work

Comment by Roman Kovařík [ 19/Apr/23 ]

Introduced itemsPerRequest property:

Generated at Mon Feb 12 10:36:20 CET 2024 using Jira 9.4.2#940002-sha1:46d1a51de284217efdcb32434eab47a99af2938b.