We rely heavily on observed configurations injected through Guice in our application. We experienced a growing heap usage during our load tests and the heap dump shows that ~1.5 GB are occupied by the class org.apache.jackrabbit.core.observation.ObservationDispatcher.

Further investigations show that the activeConsumers HashSet of the ObservationDispatcher has grown to ~320000 entries, these EventConsumers all have the info.magnolia.cms.util.ObservationUtil$DeferringEventListener associated as event listener, which is used to compute the hashCode of the EventConsumer along with the hashCode of the associated Session.

So now as the info.magnolia.cms.util.ObservationUtil$DeferringEventListener does not implement hashCode() nor equals(), an added EventConsumer does never get removed, thus each request adds the EventConsumers again and again and heap usage grows.

This is how EventConsumers are added in Jackrabbit(irrelevant parts removed), where it can be seen clearly that removal is based on hashCode() implementation.

    /**
     * Adds or replaces an event consumer.
     * @param consumer the <code>EventConsumer</code> to add or replace.
     */
    void addConsumer(EventConsumer consumer) {
        synchronized (consumerChange) {
            if (consumer.getEventListener() instanceof SynchronousEventListener) {
               /****   8< ------   SNIP ----- */
            } else {
                // remove existing if any
                activeConsumers.remove(consumer);
                // re-add it
                activeConsumers.add(consumer);
                // reset read only consumer set
                readOnlyConsumers = null;
            }
        }
    }

I currently cannot test if Magnolia 5.4 is also affected but from a quick code review it should be the same problem, as well as Magnolia 5.2.

Let me know if you need more information