[MGNLREST-197] Lacking of total size of items when doing REST requests Created: 13/Dec/18 Updated: 09/Mar/23 Resolved: 08/Apr/22 |
|
| Status: | Closed |
| Project: | Magnolia REST Framework |
| Component/s: | delivery |
| Affects Version/s: | 2.1.1 |
| Fix Version/s: | 2.2.12 |
| Type: | Improvement | Priority: | Neutral |
| Reporter: | Viet Nguyen | Assignee: | Chuong Doan Huy |
| Resolution: | Done | Votes: | 1 |
| Labels: | headless-team-support | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | 3d 6.5h | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||
| Template: |
|
||||||||||||||||||||||||||||||||
| Acceptance criteria: |
Empty
|
||||||||||||||||||||||||||||||||
| Task DoD: |
[X]*
Doc/release notes changes? Comment present?
[X]*
Downstream builds green?
[X]*
Solution information and context easily available?
[X]*
Tests
[X]*
FixVersion filled and not yet released
[ ] 
Architecture Decision Record (ADR)
|
||||||||||||||||||||||||||||||||
| Date of First Response: | |||||||||||||||||||||||||||||||||
| Epic Link: | Support | ||||||||||||||||||||||||||||||||
| Sprint: | DevX 6, DevX 7 | ||||||||||||||||||||||||||||||||
| Story Points: | 3 | ||||||||||||||||||||||||||||||||
| Team: | |||||||||||||||||||||||||||||||||
| Description |
|
Timebox: 3SP User Story: As a developer I want to know the total number of results, so I can inform the user, and in some cases so that my frontend can display an accurate scrollbar or other control type. Note: This is not the number of returned results, we need the total number of results in the system that match the query in order to implement comprehensive paging behaviour on the client/frontend. As described in SUPPORT-9407 that in info.magnolia.rest.delivery.jcr.NodesResultWriter.writeTo(...)
This prevented customer to have total size of items when doing REST requests. Suggestion (from bstaryga ) It makes it hard to paginate/lazy load with help of offset and limit, as we do not know how many pages should there be. It would be nice to extend the response to have the following properties:
e.g.:
{
"limit": 5,
"size": 10,
"start": 0,
"results": [...]
}
Input from customer: "We are wanting in the Delivery API to have a way to see the total number of results for a given query so we can better manage pagination. When completing a query with an offset and limit, we are able to retrieve a given query in chunks which is helpful, but we also need the number total results so that we can know how many pages of data there are and produce a UI which includes Page numbers, not just next and previous. This seems to be common in API's which offer pagination so we would love to see that here." |
| Comments |
| Comment by Christopher Zimmermann [ 01/Apr/20 ] |
|
If there are some cases that would be hard or performance intensive then endpoint could return a -1 to indicate "not available"/"too many". We should implement the feature for when an answer is easily available, which seems like it would be much of the time. Just as an example, if the number of results is over 200 we could return a -1. |
| Comment by Christopher Zimmermann [ 04/Jan/22 ] |
|
Comment from rkovarik from linked ticket: I don't think this is possible without iterating all results on server. Related forum post: https://groups.google.com/a/magnolia-cms.com/g/user-list/c/UbPAaE7JKEY/m/EEk_YwRHCQAJ?utm_medium=email&utm_source=footer. |
| Comment by Christopher Zimmermann [ 04/Jan/22 ] |
|
bstaryga what do you think about my proposal above on April 1st 2000? As a way to address the performance issues mentioned. |
| Comment by Bartosz Staryga [ 05/Jan/22 ] |
|
czimmermann not a fan You also say: if the number of results is over 200 we could return a -1 Regardless of how hard it is to implement, I believe we should invest more time and do it the right way. |
| Comment by Christopher Zimmermann [ 17/Mar/22 ] |
|
bstarygafor technical reasons in JCR it could be the case that this information is too costly of system resources, from Roman above "I don't think this is possible without iterating all results on server.". If that is the case, can you think of any other improvements that we could make that would help customers? That's what I was trying to get at with the idea that it could return a number up to some value - and above that return "sorry its more then 100 items, but I don't know how many." |
| Comment by Bartosz Staryga [ 17/Mar/22 ] |
|
czimmermann fair enough. How about by default we always return what we did till now
{
"results": [...]
}
If I want, I can add the paginate info explicitly, knowing that it will affect the speed (info in docs with exmplanation)
{
"limit": 5,
"size": 10,
"start": 0,
"results": [...]
}
So if I am willing to pay that trade-off I can |
| Comment by Christopher Zimmermann [ 17/Mar/22 ] |
|
Good idea, thanks. Still what to do if there's like 10000 or 1000000 or so, aand the server would take 30 seconds to iterate all of them on the server? (Meanwhile eating CPU) I could also see including a timeout, for example 1 second. But then still what should the endpoint return if after counting for 1 second it had iterated 458 records, but there were definitely more? |
| Comment by Bartosz Staryga [ 17/Mar/22 ] |
|
czimmermann let's implement and test for 10000 and 1000000 and see, maybe it is not as bad as we think. |