Details
-
Task
-
Resolution: Done
-
Neutral
-
None
-
None
-
None
-
-
Empty show more show less
-
Empty show more show less
Description
Every day, usage metrics reports from Magnolia instances are sent to our AWS-managed ES server. This has been the case for months, and the problem is that at no point does the data come out. It is only a matter of time before this introduces larger problems (index health, Kibana responsiveness, disk space, billing).
Kibana comes with a builtin tool to rotate the data. However, that feature is not available on the open-source version that AWS runs for us. This ticket is about researching and implementing:
- having the data automatically split into monthly indexes such as: magnolia-2020-01, magnolia-2020-06, magnolia-2021-12, etc.
backing up older quarters, where?this might actually not bring any performance benefits once indexes are split- not losing any data
Resources:
- https://medium.com/datadriveninvestor/making-elasticsearch-indices-manageable-d49ac113a6be configure ES to work with monthly indexes
- https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/curator.html use a Lambda that triggers the rollover every month
- let us use the Java API, though: https://github.com/dadoonet/elasticsearch-java-client-demo
Checklists
Acceptance criteria