Uploaded image for project: 'Queued Asynchronous Activation over RabbitMQ'
  1. Queued Asynchronous Activation over RabbitMQ
  2. QAARQ-17

publish large files using rabbitmq result in out of memory exception

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Neutral
    • None
    • None
    • None
    • None
    • SUSE Linux Enterprise Server 11 (x86_64)
      Memory: 16 GB

    Description

      We are just doing our final test before using rabbitmq in our productive environment but we found a problem if we try to publish a large asset (well not really large ~> 200MB).
      Every time we publish such a file we get an OutOfMemory exception on the live server (see StackTrace below). But that is not the only problem: if we continue publishing webpages or smaller assets they are all queued up in the respective rabbitmq queue and we did not found a way to send them to the live server. Even a restart of the Magnolia live server bound to that queue does not solve the problem.
      The only way to fix it is to delete the queue and with it all queued messages. That's annoying because on the authoring server the files are marked as published but they are not and if only on live server has the OufOfMemory problem the live might be out of sync.

      Note, that increasing Memory (we actually have 50GB on our live server) allow to publish larger files but that will no solve the problem but only postpone it.

      Any idea? It would even help a lot if it is possible to delete such a message so that the live serve can consume the remaining queued message.

      --- StackTrace:
      017-03-07 11:42:50,355 WARN  itmq.activation.jobs.AbstractActivationConsumerJob: Could not connect to fan1 , no connection from
       uzh-client...
      Exception in thread "Thread-20" java.lang.OutOfMemoryError: Java heap space
              at java.util.Arrays.copyOfRange(Unknown Source)
              at java.lang.String.<init>(Unknown Source)
              at java.lang.StringBuilder.toString(Unknown Source)
              at org.codehaus.jackson.util.TextBuffer.contentsAsString(TextBuffer.java:362)
              at org.codehaus.jackson.impl.Utf8StreamParser.getText(Utf8StreamParser.java:278)
              at org.codehaus.jackson.map.deser.std.UntypedObjectDeserializer.deserialize(UntypedObjectDeserializer.java:59)
              at org.codehaus.jackson.map.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:217)
              at org.codehaus.jackson.map.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:194)
              at org.codehaus.jackson.map.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:30)
              at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2732)
              at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1935)
              at info.magnolia.rabbitmq.activation.io.NodeServiceImpl.toNodesFromStackAck(NodeServiceImpl.java:117)
              at info.magnolia.rabbitmq.activation.jobs.AbstractActivationConsumerJob.activate(AbstractActivationConsumerJob.java:116)
              at info.magnolia.rabbitmq.activation.jobs.ActivationConsumerJob.processMessage(ActivationConsumerJob.java:74)
              at info.magnolia.rabbitmq.activation.jobs.AbstractActivationConsumerJob.run(AbstractActivationConsumerJob.java:76)
              at java.lang.Thread.run(Unknown Source)
      2017-03-07 14:07:02,311 WARN  org.apache.jackrabbit.core.SessionImpl            : Unclosed session detected. The session was ope
      ned here: 
      java.lang.Exception: Stack Trace
              at org.apache.jackrabbit.core.SessionImpl.<init>(SessionImpl.java:222)
              at org.apache.jackrabbit.core.SessionImpl.<init>(SessionImpl.java:239)
              at org.apache.jackrabbit.core.XASessionImpl.<init>(XASessionImpl.java:101)
              at org.apache.jackrabbit.core.RepositoryImpl.createSessionInstance(RepositoryImpl.java:1613)
              at org.apache.jackrabbit.core.RepositoryImpl.createSession(RepositoryImpl.java:956)
              at org.apache.jackrabbit.core.RepositoryImpl.login(RepositoryImpl.java:1501)
              at org.apache.jackrabbit.core.jndi.BindableRepository.login(BindableRepository.java:162)
              at info.magnolia.jackrabbit.ProviderImpl.getSystemSession(ProviderImpl.java:527)
              at info.magnolia.repository.DefaultRepositoryManager.getSystemSession(DefaultRepositoryManager.java:277)
              at info.magnolia.context.SystemRepositoryStrategy.internalGetSession(SystemRepositoryStrategy.java:54)
              at info.magnolia.context.AbstractRepositoryStrategy.getSession(AbstractRepositoryStrategy.java:74)
              at info.magnolia.context.AbstractContext.getJCRSession(AbstractContext.java:132)
              at info.magnolia.context.AbstractContext.getHierarchyManager(AbstractContext.java:205)
              at info.magnolia.context.MgnlContext.getHierarchyManager(MgnlContext.java:128)
              at info.magnolia.cms.core.version.MgnlVersioningNodeWrapper$1.exec(MgnlVersioningNodeWrapper.java:120)
              at info.magnolia.cms.core.version.MgnlVersioningNodeWrapper$1.exec(MgnlVersioningNodeWrapper.java:115)
              at info.magnolia.context.MgnlContext.doInSystemContext(MgnlContext.java:385)
              at info.magnolia.context.MgnlContext.doInSystemContext(MgnlContext.java:371)
              at info.magnolia.cms.core.version.MgnlVersioningNodeWrapper.remove(MgnlVersioningNodeWrapper.java:115)
              at info.magnolia.jcr.wrapper.DelegateNodeWrapper.remove(DelegateNodeWrapper.java:536)
              at info.magnolia.jcr.wrapper.MgnlPropertySettingNodeWrapper.remove(MgnlPropertySettingNodeWrapper.java:238)
              at info.magnolia.jcr.wrapper.DelegateNodeWrapper.remove(DelegateNodeWrapper.java:536)
              at info.magnolia.audit.MgnlAuditLoggingContentDecoratorNodeWrapper.remove(MgnlAuditLoggingContentDecoratorNodeWrapper.ja
      va:94)
              at info.magnolia.rabbitmq.activation.jobs.AbstractActivationConsumerJob.deactivate(AbstractActivationConsumerJob.java:13
      8)
              at info.magnolia.rabbitmq.activation.jobs.ActivationConsumerJob.processMessage(ActivationConsumerJob.java:72)
              at info.magnolia.rabbitmq.activation.jobs.AbstractActivationConsumerJob.run(AbstractActivationConsumerJob.java:76)
              at java.lang.Thread.run(Unknown Source)
      20
       

      Checklists

        Acceptance criteria

        Attachments

          Activity

            People

              Unassigned Unassigned
              jforrer Jann Forrer
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Checklists

                  Bug DoR
                  Task DoD