Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
None
-
None
-
None
-
None
Description
Situation:
As of Thursday, ~08:00 AM our three public instances started growing rapidly.
As you can see from pbwvl-ascms01.filesystem.png, the filesystem grew several GB until wednesday, where the instances acutally stopped working.
We extended the filesystem by another 30GB and rebooted all three instances.
After the reboot, the use of the filesystem dropped back to a reasonable level.
However, the filesystem started to grow again and keeps growing exponentially.
We rebooted all three instances again and the use of the filesystem dropped again.
This problem only occurs on the public instances.
We are running Magnolia 5.3.10 Enterprise Edition - Professional on a Tomcat 8.0.28 ("About Magnolia" is attached to this ticket)
Additional information from our system admins:
The datastore inside the repository folder seems
to be the one growing unnaturally, but this is yet to be confirmed.
27G /opt/tomcat/instances/ascms03/magnolia/repositories/public/magnolia/repository
27G /opt/tomcat/instances/ascms03/magnolia/repositories/public/magnolia/repository/datastore
12K /opt/tomcat/instances/ascms03/magnolia/repositories/public/magnolia/repository/namespaces
4.0K /opt/tomcat/instances/ascms03/magnolia/repositories/public/magnolia/repository/privileges
36K /opt/tomcat/instances/ascms03/magnolia/repositories/public/magnolia/repository/nodetypes
8.0K /opt/tomcat/instances/ascms03/magnolia/repositories/public/magnolia/repository/meta
27G /opt/tomcat/instances/ascms03/magnolia/repositories/public/magnolia/repository
We also attached our current catalina.out and magnolia-error.log from one of our public instances.
Expectations:
- We need immediate help
- We need to find out, what is growing so fast
- We need to understand, why it is growing so fast
- We need to know, what we can do about it, because as the situation is right now, we won't make it through the weekend and we cannot keep rebooting our public instances frequently
- Feel free to contact Steffen Brand (+49 5139 898 - 1725) or Andreas Christmann (+49 5139 898 - 5134) via phone
- Use entwicklungcms@rossmann.de for email communication