[MAGNOLIA-9053] Improve scalabity of publishing content Created: 10/Aug/23  Updated: 06/Feb/24

Status: Open
Project: Magnolia
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Epic Priority: Neutral
Reporter: Michael Duerig Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: dx-core-6.3
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File publishing 10 deletions is slow (50 seconds).MP4    
Issue Links:
Relates
relates to MAGNOLIA-8880 Performance problems with too many ch... Open
relates to MGNLUI-7087 Browser freezes after publication of ... Open
relates to PUBLISHING-246 extend definition configuration with ... Open
relates to PUBLISHING-202 Not able to publish huge amount of it... Closed
relates to PUBLISHING-134 orderSiblings causes the nodes orderi... Closed
relates to PUBLISHING-254 Reduce the publishing load by optimiz... Closed
dependency
is depended upon by PUBLISHING-295 reduce data load during publishing by... Closed
Template:
Epic Name: publishing scalability
Acceptance criteria:
Empty

 Description   

Context

See the notes from UHZ for pain points and initial findings

Removing old tasks created by publishing will also help in keeping the repository lean resulting in e.g. better search performance. See SUPPORT-6301.

Customer feedback

From the product board (1, 2):

  • Belimo: "Sub-nodes are never-ending due to the amount of content. 40k pagesĀ 
    Mgnl can handle the content but lagging (GWS-1310, this is Arvato's ref.no.) becomes a problem when there are embargoed publication needs:
    IR section - Semi & Annual report - Push all Master changes simultaneously. We tried to break them up into groups and break down publications, but it still takes some time: the status bar of progress gets stuck"
  • Argos/Habitat/Sainsburys: "The author's instance publishing slows down when a too number of tasks are happening at the same time"
  • Argos/Habitat/Sainsburys: "Sync module (with some customisation) to populate new publics with content. Takes up to 1 hours to sync content (currently runs sequential but looking to move to parallel to speed up process)
  • Argos/Habitat/Sainsburys: "Publishing workflow speed"
  • Belimo, Pooja Bhavsar: "Sub-nodes are never-ending due to the amount of content. 40k pagesĀ 
    Mgnl can handle the content but lagging (GWS-1310, this is Arvato's ref.no.) becomes a problem when there are embargoed publication needs:
    IR section - Semi & Annual report - Push all Master changes simultaneously. We tried to break them up into groups and break down publications, but it still takes some time: the status bar of progress gets stuck"

Questions for discovery

Publishing content in Magnolia is currently transactional: on publication all public instances will update to the new content, only then will the author instance show the content as published.

  • How could we implement non transactional publishing?
  • What consistency guarantees could we provide? What use cases would those cover? Which not?
  • Could we use a git based solution (see also MAGNOLIA-9052)?
  • Can we offload publishing to a queuing solution?
  • How would external solutions (git, queue) affect overall complexity? See also this customer comment.
  • How should we reflect the state of content in the author UI for these cases?

Customer feedback


Generated at Mon Feb 12 04:38:10 CET 2024 using Jira 9.4.2#940002-sha1:46d1a51de284217efdcb32434eab47a99af2938b.