Using the "Range" HTTP header to enable paging of nested collections in REST resources

While working on a customer project, I recently imported a database that was actively used for over 20 years into Structr. Real-life data is often very different from your average testing data, both in size and complexity (unless you put a large amount of effort in the creation of your test data), so I quickly found out that the data editing components in the Structr UI were not able to handle thousands of relationships originating from a single node.

In order to fix this problem, we needed a way to implement paging of nested collections, because even if you load just a single user in the data view, this user might have a list of 40000 related nodes, which you don't want to display in a single table cell. Such a feature is also very useful for mobile applications where you want to limit (and control) the amount of data that is sent over the wire, and we had in mind something like that for a long time.

The solution

You can already use paging in Structr REST resources, but only in top-level resources, so we first thought about extending the page and pageSize request parameters to implement paging of sub-resources depending on the nesting level, i.e. using

/items?pageSize=10,10,10&page=1,1,2

to select the the first page on level 1, the first page on level 2, and the second page on level 3. This approach however was not sufficient to solve the problems we had in the Structr Ui, because we wanted to control the pageSize and page of each collection on a per-property basis.

This was the point where I discovered the "Range" HTTP header, which lets you define arbitrary keywords and assign ranges to them in your HTTP request. It is widely used together with the keyword bytes to enable for example download resuming etc. We decided to use it for paging control of nested collections, so you can for example control the contents of the items collection of a user by sending a "Range" header with the content items=0-10.

Conclusion

Together with other small changes, this feature has vastly improved the performance of the tabular data view in the Structr Ui. Most of the data is now loaded "on demand", and the total amount of data has been greatly reduced, resulting in a much better user experience.

In addition to that, you can now use the "Range" HTTP header in your REST requests to limit the amount of data that is returned to you. Just specify a range of item numbers (i.e. 0-10 for the first ten elements, 10-20 for the second "page" of ten elements) together with the name of the desired nested collection (e.g. items, or ownedNodes etc.).

Example

Return items 0-10 (exclusively) from the nested collection named "items" of the first user, do not return any children:

curl -H"Range:items=0-10,children=0-0" http://localhost:8080/structr/rest/users?pageSize=1

Please note that it is not possible to specify more than one range for the same collection right now.

Share this post

Comments