[ESRCH-45] Index consistency on JVM crash Created: 01/Dec/20  Updated: 02/Dec/20  Resolved: 02/Dec/20

Status: Closed
Project: External Enterprise Search
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Neutral
Reporter: Michael Duerig Assignee: Michael Duerig
Resolution: Obsolete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Template:
Acceptance criteria:
Empty
Task DoR:
Empty
Date of First Response:
Epic Link: Elastic Search

 Description   

Goal

Clarify expectations, goals and options wrt. keeping the ES index consistent.

Problem

The SearchIndex interface does not give us consistency guarantees beyond those that we have for the integrated Lucene index in Jackrabbit. After a server / JVM crash the index can still go out of sync, requiring a full re-index.

Solution sketch

To improve on index resilience we would need to:

  • Put changes on a persisted indexing queue before they are stored in JCR
  • Process the changes from the indexing queue after they have been stored in JCR:
    • if storing was successful, send the changes to ES and remove them from the indexing queue
    • if storing was not successful, remove the changes from the indexing queue
    • A non empty indexing queue on a server restart indicates there was an abnormal termination and index recovery is needed: determine whether the changes in the indexing queue are present in JCR and only send them to ES if not.

Implementation considerations


Generated at Mon Feb 12 01:55:43 CET 2024 using Jira 9.4.2#940002-sha1:46d1a51de284217efdcb32434eab47a99af2938b.