[EXTDAM-46] S3: performance improvements Created: 22/Jan/20 Updated: 12/Mar/20 Resolved: 06/Mar/20 |
|
| Status: | Closed |
| Project: | External DAMs |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 1.0 |
| Type: | Story | Priority: | Neutral |
| Reporter: | Ilgun Ilgun | Assignee: | Jesus Alonso |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | 0d | ||
| Time Spent: | 1.25h | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Template: |
|
||||||||
| Acceptance criteria: |
Empty
|
||||||||
| Task DoD: |
[ ]*
Doc/release notes changes? Comment present?
[ ]*
Downstream builds green?
[ ]*
Solution information and context easily available?
[ ]*
Tests
[ ]*
FixVersion filled and not yet released
[ ] 
Architecture Decision Record (ADR)
|
||||||||
| Date of First Response: | |||||||||
| Epic Link: | DAMs backend | ||||||||
| Sprint: | Add-Ons 31, Add-Ons 32, Add-Ons 33 | ||||||||
| Story Points: | 5 | ||||||||
| Description |
|
Currently, S3 provider does 1 API call per folder, ignoring the keys not belonging on that folder. So for each bucket there is N calls to display the tree. However, it is possible to load all the assets and folders in one single API call. We will do so adding a children collection to S3Folder to store all the descendants. It is also needed to update the caching strategy. One additional improvement consists on adding a map of bucket-region in AmazonS3Gateway. Regions are inmutable and so we avoid doing extra calls for obtaining the region of a bucket. Moreover, as S3 is eventually consistent, when a bucket is created, there is a race condition that could raise an error if the new bucket has not yet been created in the default region. |
| Comments |
| Comment by Evzen Fochr [ 18/Feb/20 ] |
|
I am afraid this will more kill performance than help it. S3 is usually used for thousands and milions of assets and if we try to cache them all at once we overload magnolia instance memory. Lazy loading with more calls is only way even if it is more costly because of more calls to s3. |