[MAGNOLIA-8059] LightModule resources distribution: Define a convenient best practice Created: 25/Jan/16  Updated: 15/Apr/21

Status: Accepted
Project: Magnolia
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Major
Reporter: Christian Ringele Assignee: Christopher Zimmermann
Resolution: Unresolved Votes: 3
Labels: collector-0cf1239d, discuss-in-pm, light-development
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Issue Links:
relation
Template:
Acceptance criteria:
Empty
Task DoD:
[ ]* Doc/release notes changes? Comment present?
[ ]* Downstream builds green?
[ ]* Solution information and context easily available?
[ ]* Tests
[ ]* FixVersion filled and not yet released
[ ]  Architecture Decision Record (ADR)
Date of First Response:

 Description   

Using light module development approach for front end leaves a couple of questions open in regards of: How to use it and distribute changes on life systems.

A main problems I see is:

  • how to guarantie that the newest changes in light modules are distributed 100% equally trough all instances?
  • how can a customer know which node has wich state of resources loaded? (How can I be sure public2 has the same resources as public1 or public3).
  • When setting magnolia.develop=false changes in the light modules (file system) are detected, BUT the JS & CSS is still served cached. And we have no mechanism to flush the caceh on public instances (except activating something).

My suggestion:
We could define a simple best practice approach:
They are observed & loaded & managed on the author, and distributed to publics as (JCR) CMS content via the publication mechanism.
Meaning that on the life public instances the resources are not loaded form a light module in the file system, but from a distribution over JCR.

  • the author loads the light modules from the file system
  • It provides a possibility to create a "live" set of it -> copy them into JCR repo.
    This is in oposite to a "hot fix", its a recursive creation of the whole light module. Of all in it, not just one file.
    Either all or none, not split-able (except by hot-fixing one singly).
  • This "live" set would be published.
  • A version of the whole tree would be created.

What it solves / advantages:

  • All the main mechanisms we already have: Hot fixing for JCR copies. The loading cascading (classpath/filesyste/jcr), Publishing with transactional activation.
    So not that much would have to be developed.
  • On public its all within the CMS -> clone one public, you have it.
    No need to think about the file system & resources in scope of public instances.
  • Transactional activation ensures that its in sync on all publics.
  • JCR versioning would provide:
    • Clear knowledge of what was when active
    • Possibility to roll back
  • As its activation of content, Cache will automatically flush -> JS & CSS are re cached.
  • Also simplifies for OnDemand, plugging in more Public nodes on the fly.
  • It is not contradicting of using on life a file system approach., just remove the activation possibility on the author.

I see the chance of defining a convenient approach which can be easily be understood and sold.
A proper & clear "best practice", rather than that every customer needs to find a way to use it, even all they want is using it a simple/standard way..
And it is not restrictive to anybody, you can still use any file system based approach in combination with Git etc if needed.



 Comments   
Comment by Christopher Zimmermann [ 04/Feb/16 ]

I like this suggestion. We've been very focussed on the developer experience, and have made massive improvements. But currently teams are rather on their own when it comes to pushing their changes to the public instances. I'd like them to have the freedom to use a git deployment approach - but I think this suggestion is a good default for people who "want magnolia to do the work for them". Also the set that gets pushed could actually be a combination of the active mix of resources on the author - including classpath/file/jcr hotfix.

This problem was also brought up by a client (runger):
"To be quite clear: the way magnolia stores templates and resources in JCR, and permits publishing them from author to web nodes was one of the big reasons we chose magnolia in the first place, and we don't want this stuff on the file-system or (shudder!!) on the classpath." (https://wiki.magnolia-cms.com/display/~runger/Feedback+for+Version+5.4.3?focusedCommentId=114066342#comment-114066342)

Comment by Brian Warrick [ 04/Feb/16 ]

I believe it was nbarbe who helped Atlassian build a git connector to pull in resources (primarily css for them, I think). The question of deploying updates for templates and css comes up with many US customers. Ultimately, they like the flexibility to hotfix items, but they recognize that allowing production updates that do not go through some form of testing and source code management is risky.

Comment by Magnolia International [ 04/Feb/16 ]

The approach described here has the advantage to be fully independent of the infrastructure and architecture since it is managed 100% by Magnolia. The location of the public and author instance, the way it is deployed and operated does not matter. It clearly reduces the operation cost.

I personally prefer letting the infrastructure managing the distribution of the lightmodule resources. It is true that it will require additional work for the customer to setup this distribution. But I see more advantage in keeping things simple: Git + ssh already solved this problem quite easily and in a secured and standard way. It also enables a lot of possible scenarios like having an instance on a specific development branch only. A simple "git checkout" is enough, which is fully integrated in all devops tools, and this is the beauty!

So I'm not saying that my idea is better, actually both are totally valid and fulfill needs of different users.

Comment by Christian Ringele [ 10/Feb/16 ]

Regarding the company's using the "pure" FS & git way:

If I think about it in production mode, there comes one big question most directly ask first place:
If using git pulls etc. how cna be guaranteed that on all public instances the absolute same resources are used?

Meaning:
The git and FS approach is nice, but its completely decoupled from Magnolia.
Lets assume, on one of the three public instances the git pull doesn't work. The script doing the pull detects it, and now what?
How to react to it? How telling Magnolia that it should not use it, or even go offline until a successful pull?
Clear it is all manageable by informing a LB and tell him not to use node 2.
But its hard to teach Magnolia that one public node is actually not in a clear and usefull state.

Where the activation has one big advantage:
Either it works on all or on none -> you just know what the state of the resources is on all public nodes.

I'm not saying, that the FS approach should not be allowed.
As Topher says, its not contradicting. But companies using the FS & git approach will really know what they do and customize quite a lot to be able to react to it.

Comment by Christian Ringele [ 14/Dec/16 ]

But I see more advantage in keeping things simple: Git + ssh already solved this problem quite easily and in a secured and standard way. It also enables a lot of possible scenarios like having an instance on a specific development branch only. A simple "git checkout" is enough, which is fully integrated in all devops tools, and this is the beauty!

A last coment here form my side, cause I think my suggestion is not fully understood:
The discussion goes into direction of "Git & pull vs. Distribution over JCR" which is WRONG!

I talk about HOW the resources get from the author to the public.
This does NOT contradict the "git & pull" !
On the author, all changes in FS would be fetched by the "git & pull", because its just a neat and nice way to do it, no opposition here.
But after the pull/after the update of the FS files, the distribution of them to the publics via JCR Activation.
Its NOT contradicting, its is collaborating and using the best of both worlds.

What is the gain that all instances have to pull at the same time? I don't see any.
Why not just on the author? And the distributing them either manually, or automatically? whatever fits the customer best. Both could be provided OTB.
And for example when sending them to the publics via Activation, the git commit ID could be used to refer which exact version is not sent to the live system, and more could be thought of and added...

Comment by Richard Unger [ 09/Nov/17 ]

+1 from me (us at LFRZ)

We can't easily deploy via GIT due to security policies, and we don't like the fact that we can't activate the resources.

We currently solve this by using webdav to copy our light modules into JCR resources storage on the author systems, and activate them from there. We would love to see a more unified approach as described above.

Comment by Viet Nguyen [ 10/Nov/17 ]

Ok also from SUPPORT-8104:

Currently, info.magnolia.module.model.reader.LightModuleDefinitionReader goes directly against the filesystem. This means it does not "find" light modules which are stored in the JCR resources workspace. In contrast, the other areas of light development (template registry, dialogs, themes, etc...) have no problem finding the definitions regardless of storage location because they operate on the Resources abstraction.

Please fix LightModuleDefinitionReader to use the Resources Abstraction to locate modules rather than going directly to the file system. This will enable JCR-stored Light Modules, and also enable "JCR patching" for the module descriptors, even when they are present on the filesystem.

Comment by Vivian Steller [ 22/Mar/19 ]

Approach 1: JCR Publishing: +1 but with solution for * (see below)

Approach 2: for SSH/Git: -1 reasons
a) versioned packages/bundles (NPM, JAR, ..) should be deployed, not "sources" (IMO)
b) we avoid external tool dependencies where we can; it complicates the infrastructure (even though it's "just one more package to install") * 

Other approaches:

Approach 3: Resources dir in NFS *: +/-0
author & public share the same resources dir which is stored in an NFS;
 no staging: install on author = install on public
 very simple & works

Approach 4: Command to download bundle/package from somewhere (Nexus, NPM) *: +2
  author and public can be updated individually, also from external (via REST), e.g. CI-System
maybe even the author server could be the host of the repository!

 

Problem *

All the approaches suffer from the following problem: regardless the fact that publishing is transactional, reloading definitions on public is not "transactional"; at least if you have many files/folders in your resources folder, scanning / reloading light modules takes some seconds. If in that particular moment user requests to pages/components using these definitions arrive, (Freemarker) exceptions will be thrown.

I see two ways to solve this - very important - issue:
1) Implement definition reloading transactionally: Magnolia dismisses old definitions with new ones only after the new ones have been instantiated fully; +1
2) Load balancer control to handle this case; this breaks, however, approaches 1 & 2 as there's no way to hook LB-switches between publication or NFS distribution

Generated at Mon Feb 12 04:29:16 CET 2024 using Jira 9.4.2#940002-sha1:46d1a51de284217efdcb32434eab47a99af2938b.