Skip to content
GitLab

pagination was added to the REST API

Add offset and page to all endpoints that query multiple entries.

Update March 2022:
Perhaps there has been a misunderstanding about this issue.
Pagination had already been implemented via limit-skip parameters.
It still had to be implemented for the new "search" functions (see: #104 (closed), #99 (closed)).

There is more discussion needed! (explanation: #87 (comment 128127))
Options:

  • leave as is
  • leave as is, but provide "page" and "size" parameters (which can easily be adapted to the limit-skip-schema)
  • leave as is, provide "page" and "size", and adapt result to pagination look-and-feel
  • switch to an elegant pagination module (That means longer loading times, which may be unacceptable for certain queries!)
  • switch to elegant pagination module and investigate cashing methods (functools.lru_cache did not work)
  • do a mix between the module's and our own pagination (using FastAPI's limit-skip-query under the hood for our own pagination)

Update 2022-03-28: Decision taken

  • "limit" and "offset" will be provided as is
  • "page" and "size" will be additionally provided (second option from above)
  • Pagination and databases (especially those with NRT data!) is always an issue
    (what happens, when data is queried by pagination while database updates take place?)
    This is a nice discussion (and solution!) on this topic:
    https://www.sitepoint.com/paginating-real-time-data-cursor-based-pagination/
    If, in the end, there is time left, one could think of implementing the above solution.

About real-time data pagination:

grafik

This technique relies on a pointer to a column with unique values that can be sorted by a time-related criterion.

In the TOAR database, we have such a column for stations and time series, namely the column "id", but we have explicitly dispensed with this in the table data in order not to unnecessarily inflate the table.

We now have to investigate whether we want to implement cursor-based pagination for stations and time series in the TOAR database REST API, while refusing pagination for the table “data” -- which is by the way to my eyes not useful anyway.

Main differences (advantages/disadvantages):

  • In offset pagination, we can sort by any column and paginate the results while cursor-based pagination depends on the sorting of the unique cursor column.
  • Offset pagination contains page numbers in addition to next and previous links. But due to the highly dynamic nature of the data, we can’t provide page numbers for cursor-based pagination.
  • Generally, offset pagination allows us to navigate in both directions while cursor-based pagination is mostly used for forward navigation.
Edited by Sabine Schröder