[MGNLMIGRATION-234] Optimize performance for large projects Created: 21/Mar/13 Updated: 29/Mar/13 Resolved: 29/Mar/13 |
|
| Status: | Closed |
| Project: | Migration 4.4 to 4.5 (closed) |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 1.2.2 |
| Type: | Improvement | Priority: | Critical |
| Reporter: | Zdenek Skodik | Assignee: | Roman Kovařík |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Template: |
|
||||||||||||||||
| Acceptance criteria: |
Empty
|
||||||||||||||||
| Date of First Response: | |||||||||||||||||
| Description |
|
The module works with session based saving. The more nodes to migrate, the slower it is. This is an exponential function. It makes migrating large sets of nodes a neverending job. We should think about workspace based saving/export/import/... F.e. having about 10 thousands pages and focusing on "update node type" task, the migration is able to perform about 20 thousands of node types changes in first hour, 15 thousands in second hour, 10 thousands in third hour, ..., 3 thousands in 18th hour, etc. Such job would need about 3 days, which can't be acceptable. The linked migration.log is a good reference (covers less then a day), it was running on 8GB RAM and 2 CPUs. |
| Comments |
| Comment by Roman Kovařík [ 29/Mar/13 ] |
|
UpdateWebsitePrimaryTypeMigrationTaskTest is already part of |
| Comment by Jan Haderka [ 29/Mar/13 ] |
|
I would prefer there was an explicit comment around all those session.save() calls saying this is to prevent OOME for gib migrations. Also Node type of " + countOfNodes + " nodes already changed. message should pbly read "Changed node type of xxx nodes already." |