Uploaded image for project: 'Machine Learning'
  1. Machine Learning
  2. MLEARN-6

Neural network storage debouncer is global, breaking multiple-ranker setups

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Neutral Neutral
    • 1.1
    • None
    • None

      The issue was uncovered while doing load and performance tests on periscope with a RankingNetworkStorageStrategy per user (the default one, at the moment). 
      In a concurrent scenario with 50 different users, we should end up with 50 different nodes in the rankings workspace, one for each user and their neural network.
      At the end of the tests, the actual nodes stored were less than 50, meaning that for some users their NN was not saved after the training step, as it was supposed to be.

      Currently, we only use one global debouncer to limit how often ranking neural networks are stored (see RankingNetworkStorage).
      As a result, multiple storage requests from different rankers may end up in all of them but the last one being dropped.

      Instead, each separate ranker should be handled by a dedicated debouncer, meaning that we limit how often ranking information is persisted per individual ranker.

      Implementation ideas

      • RankingNetworkStorage could be non-singleton, i.e. one instance for each ranker.
      • RankingNetworkStorage could keep track of multiple debouncers (might be complicated though).
      • Debouncing could be somehow handled by the strategy.

        Acceptance criteria

              fgrilli Federico Grilli
              creichenbach Cedric Reichenbach
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved:

                  Bug DoR
                  Task DoD

                    Estimated:
                    Original Estimate - Not Specified
                    Not Specified
                    Remaining:
                    Remaining Estimate - 0d
                    0d
                    Logged:
                    Time Spent - 5h
                    5h