pagination of large file listings, shares or trash

[butonic]

Currently, listing large folders (10k+) becomes slow when expensive properties are requested. Not only because the server may internally have to make more requests, but also because parsing XML with 10k+ entries and handling them in browsers becomes challenging.

Add WebDAV pagination

We could use the Range header to let clients request a certain range of collections, eg. Range: rows=2-3 and in the response return Content-Range: rows 2-3; total=9. And we can announce range suppurt using Accept-Ranges: rows It would be compatible with the webdav RFCs, as long as we assume collections are ordered. The problem is that the web ui allows ordering by different properties and there is no example for how to specify the order of a collection in a PROPFIND request.

Note Web Distributed Authoring and Versioning (WebDAV) Ordered Collections Protocol
RFC 3648 treats order as a property of the collection that is set at creation or PATCHed later. It does not define any semantics that could be (mis)used to make the server list resources in a specific order.

So we would have to introduce OC-OrderBy: <property> to let clients tell the server how to order the list.

Use /graph/ driveItems

It already has $top & $skip for client driven pagination, server driven pagination and $orderby specified. (as well as $filtering and $expanding of resources)

common consequences

Regardless of the approach, sorting would have to happen on the server side, which would make the ocdav or graph services consume a lot more memory. However, we can then use ListContainer together with an If-Match etag and reduce number of requests to the storage.

related problems

• slow listing of shares: fetchShares in the sharesstorageprovider does not parallelize stat requests #9828
• listing trash becomes slow with lots of items