Leave more leeway for pagination methods

[baboulebou]

Currently the spec assumes a Pagination token for all Search results; i.e.,

"page": {
    "next_token": "alsehrq3495u8"
  }

But some implementations may support other means, such as "offset + limit" for example. Can we possibly make the pagination token explicitly non-normative, and mention that pagination details are implementation-specific ?

[davidjbrossard]

The pagination mechanism in the spec today is applied to:

• Subject search
• Action search
• Resource search

@identitymonk also points out that the context object in the access evaluation API and the response in the access evaluations (boxcar) API could benefit from a pagination mechanism.

We need to factor out the pagination mechanism from each section (currently mentioned 3x in each search API) and perhaps even delegate the specifics of pagination to the underlying implementation. As a result the next_token example becomes a MAY rather than a MUST.

Would that impact interoperability? If different implementations have different means of pagination, then how can interoperability be achieved? Perhaps the page field is part of the spec and the contents is either a next_token or the combination of offset and limit as suggested by @baboulebou .

Moving to the 6/24 meeting for debate.

[davidjbrossard]

@baboulebou recommends we return a JSON that would contain a more structured response than just the token page identifier.

@eazerad points out that the pagination mechanism needs to be interoperable enough to still promote interoperability between implementations

Recommendation:
We introduce a type field inside the page object.

Define pagination type values. There are 2 values at the moment:

• token
• offset

The type determines which other fields are present in the page object. For instance token will require a token field.

We also need to think about limits we have to apply to pagination to avoid DoS attacks or server overloads. The backend needs to have its own limits/validation. (either under security considerations or in the pagination section).

Recommendation #2: factor out pagination from the specific sections they are in into a single pagination section that applies to all parts pagination is relevant.

Note: we need to steer clear of transport-level breakup principles (chunked responses, multi-parts, etc...) that are specific to the transport mechanism chosen (HTTP REST vs. gRPC vs. other)

[davidjbrossard]

• Consistency vs. snapshot isolation

If we return metadata about the search, the metadata may be about the search at the original time of execution and that could be updated. We're not requiring implementations to achieve snapshot isolation to guarantee correctness or return the full set.

[davidjbrossard]

Note: use SCIM as inspiration