[MGNLRSSAGG-136] As a user I am able to import 500 well performing feed sources within 5 minutes. Created: 04/Dec/13 Updated: 04/Jul/14 Resolved: 04/Jul/14 |
|
| Status: | Closed |
| Project: | Magnolia RSS Aggregator Module |
| Component/s: | rss_importer |
| Affects Version/s: | 2.2 |
| Fix Version/s: | 2.3.x |
| Type: | Story | Priority: | Major |
| Reporter: | Lars Fischer | Assignee: | Lars Fischer |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | rssaggregator | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Template: |
|
| Acceptance criteria: |
Empty
|
| Task DoD: |
[ ]*
Doc/release notes changes? Comment present?
[ ]*
Downstream builds green?
[ ]*
Solution information and context easily available?
[ ]*
Tests
[ ]*
FixVersion filled and not yet released
[ ] 
Architecture Decision Record (ADR)
|
| Date of First Response: |
| Description |
|
At the moment, all feeds defined in RSS Aggregators are imported sequentially. If there are many feeds defined, it takes too long for some clients until the feed importing is finished (even for well performing feed sources). Therefore we need a better strategy for the feed import (e.g. starting several jobs at the same time). One idea would be to have a flag on every RSS Aggregator determining the priority of the feed. What we should guarantee: Be able to import 500 (well performing) feed sources within 5 minutes. |
| Comments |
| Comment by Jan Haderka [ 11/Jun/14 ] |
|
Did you try this with FastRSSFeedFetcher configured instead of simple one that we run by default? Fast one executes feeds in parallel so it should be no problem to achieve this task with it. |
| Comment by Lars Fischer [ 04/Jul/14 ] |
|
With version 5.3 every feed can have it's own scheduler instead of one scheduler for all feeds. The main bottleneck will remain the speed of the external feed sources and that cannot be solved within Magnolia. |