Uploaded image for project: 'Machine Learning'
  1. Machine Learning
  2. MLEARN-6

Neural network storage debouncer is global, breaking multiple-ranker setups

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Neutral
    • 1.1
    • None
    • None

    Description

      The issue was uncovered while doing load and performance tests on periscope with a RankingNetworkStorageStrategy per user (the default one, at the moment). 
      In a concurrent scenario with 50 different users, we should end up with 50 different nodes in the rankings workspace, one for each user and their neural network.
      At the end of the tests, the actual nodes stored were less than 50, meaning that for some users their NN was not saved after the training step, as it was supposed to be.

      Currently, we only use one global debouncer to limit how often ranking neural networks are stored (see RankingNetworkStorage).
      As a result, multiple storage requests from different rankers may end up in all of them but the last one being dropped.

      Instead, each separate ranker should be handled by a dedicated debouncer, meaning that we limit how often ranking information is persisted per individual ranker.

      Implementation ideas

      • RankingNetworkStorage could be non-singleton, i.e. one instance for each ranker.
      • RankingNetworkStorage could keep track of multiple debouncers (might be complicated though).
      • Debouncing could be somehow handled by the strategy.

      Checklists

        Acceptance criteria

        Attachments

          Issue Links

            Activity

              People

                fgrilli Federico Grilli
                creichenbach Cedric Reichenbach
                Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                  Created:
                  Updated:
                  Resolved:

                  Checklists

                    Bug DoR
                    Task DoD

                    Time Tracking

                      Estimated:
                      Original Estimate - Not Specified
                      Not Specified
                      Remaining:
                      Remaining Estimate - 0d
                      0d
                      Logged:
                      Time Spent - 5h
                      5h