[EXTDAM-46] S3: performance improvements Created: 22/Jan/20  Updated: 12/Mar/20  Resolved: 06/Mar/20

Status: Closed
Project: External DAMs
Component/s: None
Affects Version/s: None
Fix Version/s: 1.0

Type: Story Priority: Neutral
Reporter: Ilgun Ilgun Assignee: Jesus Alonso
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: 0d
Time Spent: 1.25h
Original Estimate: Not Specified

Issue Links:
dependency
is depended upon by EXTDAM-82 (Test only) S3: Error after creating ... Closed
Template:
Acceptance criteria:
Empty
Task DoD:
[ ]* Doc/release notes changes? Comment present?
[ ]* Downstream builds green?
[ ]* Solution information and context easily available?
[ ]* Tests
[ ]* FixVersion filled and not yet released
[ ]  Architecture Decision Record (ADR)
Date of First Response:
Epic Link: DAMs backend
Sprint: Add-Ons 31, Add-Ons 32, Add-Ons 33
Story Points: 5

 Description   

Currently, S3 provider does 1 API call per folder, ignoring the keys not belonging on that folder. So for each bucket there is N calls to display the tree. However, it is possible to load all the assets and folders in one single API call. We will do so adding a children collection to S3Folder to store all the descendants. It is also needed to update the caching strategy.

One additional improvement consists on adding a map of bucket-region in AmazonS3Gateway. Regions are inmutable and so we avoid doing extra calls for obtaining the region of a bucket. Moreover, as S3 is eventually consistent, when a bucket is created, there is a race condition that could raise an error if the new bucket has not yet been created in the default region. 



 Comments   
Comment by Evzen Fochr [ 18/Feb/20 ]

I am afraid this will more kill performance than help it. S3 is usually used for thousands and milions of assets and if we try to cache them all at once we overload magnolia instance memory. Lazy loading with more calls is only way even if it is more costly because of more calls to s3.

Generated at Mon Feb 12 01:51:21 CET 2024 using Jira 9.4.2#940002-sha1:46d1a51de284217efdcb32434eab47a99af2938b.