[MAGNOLIA-7981] Optimise Registry definition access for high throughput Created: 20/Jan/21  Updated: 29/Jan/24  Resolved: 12/Feb/21

Status: Closed
Project: Magnolia
Component/s: None
Affects Version/s: 6.2.5
Fix Version/s: 6.2.7

Type: Task Priority: Neutral
Reporter: Aleksandr Pchelintcev Assignee: Aleksandr Pchelintcev
Resolution: Done Votes: 0
Labels: cs-bk, maintenance, performance
Remaining Estimate: Not Specified
Time Spent: 1d 6.5h
Original Estimate: Not Specified

Attachments: Text File 0001-MAGNOLIA-7981-Avoid-calling-AbstractRegistry-getProv.patch     PNG File 6.2.7-snapshot-performance.png     PNG File hotfix-performance.png     Text File new-0001-MAGNOLIA-7981-Avoid-calling-AbstractRegistry-getProv.patch    
Issue Links:
Git Code Review
git code review opened MAGNOLIA-7987 RegistryMap: remove API synchronizati... Closed
Problem/Incident
Template:
Acceptance criteria:
Empty
Task DoR:
Empty
Release notes required:
Yes
Documentation update required:
Yes
Date of First Response:
Epic Link: Throughput improvements
Sprint: Maintenance 41, Maintenance 42, Maintenance 43, Maintenance 44
Story Points: 5

 Description   

Problem

Performance of our configuration registries may decrease in terms of throughput under heavy stress in a highly concurrent workload. The following aspects may lead to problems:

  • <major> eventual bottleneck - synchronisation locks in the RegistryMap class. If an enormous amount of concurrent threads actively try to access the RegistryMap, lot's of the requests will get blocked and delayed.
  • <minor> besides of concurrent locking each definition provider access will attempt to apply the registered decorators and validators. These operations at least cause a scan over the decorators and validators collections, usually rather small and negligible, which still could cause a constant overhead. (* *see a side note for more details)
  • <major> the easiest way to amplify the effect of the problem described in the first point is to invoke the AbstractRegistry#getAllProviders API concurrently in all requests mentioned. Not only this would linearly increase the side effects of decoration and validation, it would largely increase the chance of a thread getting blocked: instead of using the RegistryMap#values API once, we fetch the definition providers one by one (each call is a synchronised registry access).

side note: decorators do cache the resulting state for a short period of time and never try to reapply themselves during it, exactly for throughput purposes, otherwise the overhead would include also the access and timestamp checks of the underlying resources.

Counter-measures:

  • The most tangible action is to optimise the #getAllProviders performance - we should minimise the chance of blocking there, which would already reduce the risk of the thread blocking in the described conditions. Optimisation would involve copying the un-decorated and un-validate def providers in a separate collections with RegistryMap#values and applying everything while iterating over it without additional blocking.
  • Consider possibility of making sure that the recently decorated/validated definitions are cached/memoized for some time (kind of pushing info.magnolia.config.registry.decoration.CachingDefinitionDecorator one level up). This would reduce the complexity of a single definition provider access, although that actually has never been considered to be a big issue.
  • Consider a possibility of caching a set of resolved definitions on "application level", much like we do in rest or content-types' generated app do, reacting on registry events and definition dependencies.


 Comments   
Comment by Bradley Andersen [ 20/Jan/21 ]

/cc tduffey

Comment by Federico Grilli [ 12/Feb/21 ]

mdrapela akhamis For the release notes: the main measures to counteract this were

  • RegistryMap no longer has synchronised methods and rather relies on ConcurrentHashMap
  • Optimised implementation of AbstractRegistry#getAllProviders()
Generated at Mon Feb 12 04:28:33 CET 2024 using Jira 9.4.2#940002-sha1:46d1a51de284217efdcb32434eab47a99af2938b.