[MGNLRSSAGG-136] As a user I am able to import 500 well performing feed sources within 5 minutes. Created: 04/Dec/13  Updated: 04/Jul/14  Resolved: 04/Jul/14

Status: Closed
Project: Magnolia RSS Aggregator Module
Component/s: rss_importer
Affects Version/s: 2.2
Fix Version/s: 2.3.x

Type: Story Priority: Major
Reporter: Lars Fischer Assignee: Lars Fischer
Resolution: Fixed Votes: 0
Labels: rssaggregator
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Template:
Acceptance criteria:
Empty
Task DoD:
[ ]* Doc/release notes changes? Comment present?
[ ]* Downstream builds green?
[ ]* Solution information and context easily available?
[ ]* Tests
[ ]* FixVersion filled and not yet released
[ ]  Architecture Decision Record (ADR)
Date of First Response:

 Description   

At the moment, all feeds defined in RSS Aggregators are imported sequentially.

If there are many feeds defined, it takes too long for some clients until the feed importing is finished (even for well performing feed sources).

Therefore we need a better strategy for the feed import (e.g. starting several jobs at the same time).

One idea would be to have a flag on every RSS Aggregator determining the priority of the feed.

What we should guarantee:

Be able to import 500 (well performing) feed sources within 5 minutes.



 Comments   
Comment by Jan Haderka [ 11/Jun/14 ]

Did you try this with FastRSSFeedFetcher configured instead of simple one that we run by default? Fast one executes feeds in parallel so it should be no problem to achieve this task with it.

Comment by Lars Fischer [ 04/Jul/14 ]

With version 5.3 every feed can have it's own scheduler instead of one scheduler for all feeds. The main bottleneck will remain the speed of the external feed sources and that cannot be solved within Magnolia.

Generated at Mon Feb 12 07:05:54 CET 2024 using Jira 9.4.2#940002-sha1:46d1a51de284217efdcb32434eab47a99af2938b.