[MAGNOLIA-6775] Filesystem of public instances keeps growing exponentially Created: 15/Sep/16  Updated: 15/Sep/16  Resolved: 15/Sep/16

Status: Closed
Project: Magnolia
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Critical
Reporter: Dirk Rossmann GmbH Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File magnolia-info.PNG     File pbwvl-ascms01.catalina.out     PNG File pbwvl-ascms01.filesystem.png     Text File pbwvl-ascms01.magnolia-error.log    
Template:
Acceptance criteria:
Empty
Task DoD:
[ ]* Doc/release notes changes? Comment present?
[ ]* Downstream builds green?
[ ]* Solution information and context easily available?
[ ]* Tests
[ ]* FixVersion filled and not yet released
[ ]  Architecture Decision Record (ADR)
Bug DoR:
[ ]* Steps to reproduce, expected, and actual results filled
[ ]* Affected version filled

 Description   

Situation:
As of Thursday, ~08:00 AM our three public instances started growing rapidly.
As you can see from pbwvl-ascms01.filesystem.png, the filesystem grew several GB until wednesday, where the instances acutally stopped working.
We extended the filesystem by another 30GB and rebooted all three instances.
After the reboot, the use of the filesystem dropped back to a reasonable level.
However, the filesystem started to grow again and keeps growing exponentially.
We rebooted all three instances again and the use of the filesystem dropped again.
This problem only occurs on the public instances.
We are running Magnolia 5.3.10 Enterprise Edition - Professional on a Tomcat 8.0.28 ("About Magnolia" is attached to this ticket)

Additional information from our system admins:
The datastore inside the repository folder seems to be the one growing unnaturally, but this is yet to be confirmed.

27G /opt/tomcat/instances/ascms03/magnolia/repositories/public/magnolia/repository

27G /opt/tomcat/instances/ascms03/magnolia/repositories/public/magnolia/repository/datastore
12K /opt/tomcat/instances/ascms03/magnolia/repositories/public/magnolia/repository/namespaces
4.0K /opt/tomcat/instances/ascms03/magnolia/repositories/public/magnolia/repository/privileges
36K /opt/tomcat/instances/ascms03/magnolia/repositories/public/magnolia/repository/nodetypes
8.0K /opt/tomcat/instances/ascms03/magnolia/repositories/public/magnolia/repository/meta
27G /opt/tomcat/instances/ascms03/magnolia/repositories/public/magnolia/repository

We also attached our current catalina.out and magnolia-error.log from one of our public instances.

Expectations:

  • We need immediate help
  • We need to find out, what is growing so fast
  • We need to understand, why it is growing so fast
  • We need to know, what we can do about it, because as the situation is right now, we won't make it through the weekend and we cannot keep rebooting our public instances frequently
  • Feel free to contact Steffen Brand (+49 5139 898 - 1725) or Andreas Christmann (+49 5139 898 - 5134) via phone
  • Use entwicklungcms@rossmann.de for email communication

Generated at Mon Feb 12 04:17:43 CET 2024 using Jira 9.4.2#940002-sha1:46d1a51de284217efdcb32434eab47a99af2938b.