Uploaded image for project: 'Magnolia'
  1. Magnolia
  2. MAGNOLIA-6695

Instances gradually die when in multi-clustered environment

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Do
    • Critical
    • None
    • 5.4.3
    • None
    • None

    Description

      Instance dies after intermittent amount of time, from 20minutes to 6hours, but never lasts for more than a day.

      No strange logs before death, nonetheless see logs snippet below for preview, full trace attached:

      Jun 14, 2016 2:53:21 PM org.apache.catalina.core.StandardServer await
      INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
      Jun 14, 2016 2:53:21 PM org.apache.coyote.AbstractProtocol pause
      INFO: Pausing ProtocolHandler ["http-bio-8080"]
      Jun 14, 2016 2:53:22 PM org.apache.coyote.AbstractProtocol pause
      INFO: Pausing ProtocolHandler ["ajp-bio-8009"]
      Jun 14, 2016 2:53:22 PM org.apache.catalina.core.StandardService stopInternal
      INFO: Stopping service Catalina
      Jun 14, 2016 2:53:22 PM org.apache.catalina.core.ApplicationContext log
      INFO: Destroying Spring FrameworkServlet 'rest'
      Jun 14, 2016 2:53:22 PM com.vaadin.event.ListenerMethod writeObject
      WARNING: Error in serialization of the application: Class info.magnolia.ui.admincentral.AdmincentralVaadinServlet$1$1 must implement serialization.
      Jun 14, 2016 2:53:22 PM org.apache.catalina.session.StandardSession writeObject
      WARNING: Cannot serialize session attribute com.vaadin.server.VaadinSession.AdminCentral for session 1AC4EB995557B3DA7D0A57C0045369B0
      java.io.NotSerializableException: org.apache.catalina.loader.WebappClassLoader
      	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
      	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
      	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
      	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
      	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
      	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
      	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
      	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
      	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
      	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
      	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
      	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
      	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
      	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
      	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
      	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
      	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
      	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
      

      If clustering is removed all instances (Author and public 1) is stable for days .... If multiple publics are added instance, server life time becomes very short.

      Setup is as follow:

      Author has two sets of repositories:

      1. magnolia
        1. holds default workspaces list
        2. points to mysql database named "author"
        3. uses local file system
      2. magnoliacluster-shared
        1. holds users, dam, dms, imaging and custom developed workspaces
        2. points to mysql database named "author"
        3. uses local file system

      Public 1 has two sets of repositories:

      1. magnolia
        1. holds default workspaces list
        2. points to mysql database named "public"
        3. uses local file system
      2. magnoliacluster-shared
        1. holds users, dam, dms, imaging and custom developed workspaces
        2. points to mysql database named "author"
        3. NFS mount to authors file system

      Public 2 has two sets of repositories:

      1. magnolia
        1. holds default workspaces list
        2. points to mysql database named "public"
        3. uses NFS mount to point to public 1 file system
      2. magnoliacluster-shared
        1. holds users, dam, dms, imaging and custom developed workspaces
        2. points to mysql database named "author"
        3. NFS mount to authors file system

      Please also have a look at repositories.xml, to a get better picture

      Checklists

        Acceptance criteria

        Attachments

          1. error.log
            501 kB
          2. repositories.xml
            6 kB

          Activity

            People

              Unassigned Unassigned
              g.zalys@hotmail.com Gediminas Zalys
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Checklists

                  Bug DoR
                  Task DoD