[MGNLIMG-211] Implement memory safeguard Created: 09/Nov/18  Updated: 17/Mar/21  Resolved: 17/Mar/21

Status: Closed
Project: Imaging
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Neutral
Reporter: Jan Haderka Assignee: Unassigned
Resolution: Won't Do Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Template:
Acceptance criteria:
Empty
Task DoD:
[ ]* Doc/release notes changes? Comment present?
[ ]* Downstream builds green?
[ ]* Solution information and context easily available?
[ ]* Tests
[ ]* FixVersion filled and not yet released
[ ]  Architecture Decision Record (ADR)
Epic Link: Imaging performance

 Description   

Imaging operation can end up being resource intensive. In particular memory intensive (see MGNLIMG-210 ). For such we need to implement safeguard that would monitor memory and keep track of amount of imaging operations being executed at any given moment. If needed (in case of low resources) such safeguard would delay executing of more operations until resources are recovered, when plenty resources are available, more ops can be executed. It is preferable to let request time out in case of low resources on the server than risking crashing the server.

Keep in mind that memory is consumed gradually during operation and just starting the op would not allocate all memory for it ... some estimates/assumptions on how much memory would get used will have to be made, perhaps we could even monitor how much was really used and store that for next time.

If possible, it would be also of advantage to be able to distinguish between low-memory and high-memory kind of operations/requests and when queueing requests, prioritize those with low-memory requirements so that absolute length of queue is shortest possible one.

As for solution itself, I see 2 possible entry points:

  • imaging operations start from ImagingServlet. We could put safe guard around it.
  • we could add safeguard inside of image chain before or after loading images and having more info about them to make decisions

Solution could be also combination of both entry points or inserted as some other place.

Acceptance criteria:

  • server doesn't crash under the load even if there's 1k hi-res images requested for resizing with various output resolutions
  • all the images are eventually rendered (except for those where client stopped waiting for them (BrokenPipe))
  • it is possible to look up amount of images waiting to be rendered at any given time
  • it is possible to "flush" all requests waiting to be rendered

Possible future improvements:

  • replacing hi-mem ops with their low-mem equivalents saving the memory in time of need and sacrificing resolution to safe memory and keeping being responsive.


 Comments   
Comment by Jan Haderka [ 17/Mar/21 ]

Closing in favour of externalizing whole image processing

Generated at Mon Feb 12 02:13:22 CET 2024 using Jira 9.4.2#940002-sha1:46d1a51de284217efdcb32434eab47a99af2938b.