[MGNLREST-197] Lacking of total size of items when doing REST requests Created: 13/Dec/18  Updated: 09/Mar/23  Resolved: 08/Apr/22

Status: Closed
Project: Magnolia REST Framework
Component/s: delivery
Affects Version/s: 2.1.1
Fix Version/s: 2.2.12

Type: Improvement Priority: Neutral
Reporter: Viet Nguyen Assignee: Chuong Doan Huy
Resolution: Done Votes: 1
Labels: headless-team-support
Remaining Estimate: Not Specified
Time Spent: 3d 6.5h
Original Estimate: Not Specified

Issue Links:
Relates
dependency
documentation
to be documented by MGNLREST-360 DOC: New metas written into responses... Closed
duplicate
is duplicated by MGNLREST-234 Delivery API Pagination - Way to get ... Closed
is duplicated by MGNLREST-340 Delivery API to return limit, size an... Closed
relation
Template:
Acceptance criteria:
Empty
Task DoD:
[X]* Doc/release notes changes? Comment present?
[X]* Downstream builds green?
[X]* Solution information and context easily available?
[X]* Tests
[X]* FixVersion filled and not yet released
[ ]  Architecture Decision Record (ADR)
Date of First Response:
Epic Link: Support
Sprint: DevX 6, DevX 7
Story Points: 3
Team: DeveloperX

 Description   

Timebox: 3SP

User Story: As a developer I want to know the total number of results, so I can inform the user, and in some cases so that my frontend can display an accurate scrollbar or other control type.

Note: This is not the number of returned results, we need the total number of results in the system that match the query in order to implement comprehensive paging behaviour on the client/frontend.

As described in SUPPORT-9407 that in info.magnolia.rest.delivery.jcr.NodesResultWriter.writeTo(...)
We had a TODO item:

temporary exclude total query metadata as api is not ready to provide estimated query size.

This prevented customer to have total size of items when doing REST requests.
Please consider to have a fix for this.

Suggestion

(from bstaryga )

It makes it hard to paginate/lazy load with help of offset and limit, as we do not know how many pages should there be.

It would be nice to extend the response to have the following properties:

  • limit
  • size
  • start

e.g.:

{
  "limit": 5,
  "size": 10,
  "start": 0,
  "results": [...]
}

 

 

Input from customer:

"We are wanting in the Delivery API to have a way to see the total number of results for a given query so we can better manage pagination. When completing a query with an offset and limit, we are able to retrieve a given query in chunks which is helpful, but we also need the number total results so that we can know how many pages of data there are and produce a UI which includes Page numbers, not just next and previous. This seems to be common in API's which offer pagination so we would love to see that here."



 Comments   
Comment by Christopher Zimmermann [ 01/Apr/20 ]

If there are some cases that would be hard or performance intensive then endpoint could return a -1 to indicate "not available"/"too many". We should implement the feature for when an answer is easily available, which seems like it would be much of the time.

Just as an example, if the number of results is over 200 we could return a -1.

Comment by Christopher Zimmermann [ 04/Jan/22 ]

Comment from rkovarik  from linked ticket:

I don't think this is possible without iterating all results on server.

Related forum post: https://groups.google.com/a/magnolia-cms.com/g/user-list/c/UbPAaE7JKEY/m/EEk_YwRHCQAJ?utm_medium=email&utm_source=footer.

Comment by Christopher Zimmermann [ 04/Jan/22 ]

bstaryga what do you think about my proposal above on April 1st 2000? As a way to address the performance issues mentioned.

Comment by Bartosz Staryga [ 05/Jan/22 ]

czimmermann not a fan -1 is for the majority is not found.

You also say: if the number of results is over 200 we could return a -1
To know this we need to know what is the number so we can just return the number

Regardless of how hard it is to implement, I believe we should invest more time and do it the right way.

Comment by Christopher Zimmermann [ 17/Mar/22 ]

bstarygafor technical reasons in JCR it could be the case that this information is too costly of system resources, from Roman above "I don't think this is possible without iterating all results on server.". If that is the case, can you think of any other improvements that we could make that would help customers? That's what I was trying to get at with the idea that it could return a number up to some value - and above that return "sorry its more then 100 items, but I don't know how many."
(But maybe even iterating on the server up to 100 or 200 would add a performance penalty so big that we would not want to introduce it. Not sure)

Comment by Bartosz Staryga [ 17/Mar/22 ]

czimmermann fair enough.

How about by default we always return what we did till now
fetch /.rest/my-endpoint

{
  "results": [...]
}

If I want, I can add the paginate info explicitly, knowing that it will affect the speed (info in docs with exmplanation)
fetch /.rest/my-endpoint?paginate=true

{
  "limit": 5,
  "size": 10,
  "start": 0,
  "results": [...]
}

So if I am willing to pay that trade-off I can

Comment by Christopher Zimmermann [ 17/Mar/22 ]

Good idea, thanks.

Still what to do if there's like 10000 or 1000000 or so, aand the server would take 30 seconds to iterate all of them on the server? (Meanwhile eating CPU)

I could also see including a timeout, for example 1 second. But then still what should the endpoint return if after counting for 1 second it had iterated 458 records, but there were definitely more?

Comment by Bartosz Staryga [ 17/Mar/22 ]

czimmermann let's implement and test for 10000 and 1000000 and see, maybe it is not as bad as we think.
Then if it's reasonably bad e.g. 10s+ we sit down and brainstorm the and which point we cut it off.

Generated at Mon Feb 12 06:57:35 CET 2024 using Jira 9.4.2#940002-sha1:46d1a51de284217efdcb32434eab47a99af2938b.