[MAGNOLIA-7358] Performance: Resource origin should include fewer resources by default Created: 07/Aug/18  Updated: 14/Dec/23  Resolved: 09/Nov/23

Status: Closed
Project: Magnolia
Component/s: None
Affects Version/s: None
Fix Version/s: 6.3.0, 6.2.41

Type: Improvement Priority: Neutral
Reporter: Jan Haderka Assignee: Oanh Thai Hoang
Resolution: Fixed Votes: 0
Labels: dx-core-6.3, performance
Remaining Estimate: Not Specified
Time Spent: 5h
Original Estimate: Not Specified

Attachments: PNG File after-fix-definition-app.png     PNG File after-fix-resource.png     PNG File before-fix-definition-app.png     PNG File before-fix-resource-app.png     PNG File debug.png    
Issue Links:
Relates
relates to MAGNOLIA-9174 Introduce magnolia.resources.filesyst... Selected
Template:
Acceptance criteria:
Empty
Task DoD:
[ ]* Doc/release notes changes? Comment present?
[ ]* Downstream builds green?
[ ]* Solution information and context easily available?
[ ]* Tests
[ ]* FixVersion filled and not yet released
[ ]  Architecture Decision Record (ADR)
Release notes required:
Yes
Documentation update required:
Yes
Date of First Response:
Epic Link: Throughput improvements
Work Started:
Approved:
Yes

 Description   

If you install vanilla magnolia (no light modules) and create and execute ResourceVisitor to go through all resources observed by the system, you will get number in vicinity of 10K. While it doesn't seem to slow down the system in any way, and might be possibly useful in dev mode (I presume number comes from observation of classpath) where files on classpath can indeed change, i think in production we could reduce the number and thus increase performance in all places that traverse resources (e.g. resfn, definitions app, etc).

Issue was discovered when investigating rendering performance issue in instance with node_modules directory in light module (brought in by use of webpack) and it containing additional 30K files (facepalm). This fact, together with multiple calls to resfn in single template lead to performance issue (traversing of 40K files while searching for resources took approx 2 seconds).



 Comments   
Comment by Aleksandr Pchelintcev [ 07/Aug/18 ]

If you install vanilla magnolia (no light modules) and create and execute ResourceVisitor to go through all resources observed by the system, you will get number in vicinity of 10K. While it doesn't seem to slow down the system in any way, and might be possibly useful in dev mode (I presume number comes from observation of classpath) where files on classpath can indeed change, i think in production we could reduce the number and thus increase performance in all places that traverse resources (e.g. resfn, definitions app, etc).

this seems to be a problem with ClasspathResourceOrigin - there's API that allows to reduce the amount of classpath resources discoverable by the origin (see e.g. DefaultClasspathServiceConfigurations#excludedPackages), but there's no config property that would allow to modify the exclusion patterns without code changes, we could introduce such property effortlessly.

Issue was discovered when investigating rendering performance issue in instance with node_modules directory in light module (brought in by use of webpack) and it containing additional 30K files (facepalm). This fact, together with multiple calls to resfn in single template lead to performance issue (traversing of 40K files while searching for resources took approx 2 seconds).

this is a different issue. However, it is also an understandable one and can be possibly addressed by altering the property magnolia.resources.filesystem.observation.excludedDirectories, which should contain comma-separated list of directories to be not included into FS resource origin (I am not sure if that property supports *-templates tho, if not - we should add such support).

Generated at Mon Feb 12 04:23:04 CET 2024 using Jira 9.4.2#940002-sha1:46d1a51de284217efdcb32434eab47a99af2938b.