Nested collections and distinct results

[xtofs]

Motivation

We seem to currently lack a mechanism to extract and operate on elements of collection-valued properties in a collection of entities. For example, given:

[
  { "id": 1, "tags": ["a", "b"] },
  { "id": 2, "tags": ["b", "c"] }
]

There seems to be standard way to query for the unique set of tags ["a", "b", "c"] using$apply`. This capability is essential for analytics, faceted search, and tag-based filtering scenarios.

Overview

This proposal introduces a new transformation keyword, flatten and distinct, to the apply system query option in OData. It enables composable projection of scalar values from collection-valued properties, allowing operations such as deduplication (distinct()), aggregation, and filtering to be performed on the flattened stream.

3.1 Transformation Grammar Extension

flatten transformation :

Extend the $apply transformation grammar to include:

apply = ... / flatten
flatten = "flatten" "(" propertyPath [ "as" alias ] ")"

Where:

• propertyPath is a collection-valued property.
• alias is an optional identifier used to name the flattened scalar value.

which transforms the collection into collection of the values of the given property path with property name given by the alias

[
  { "tag": "a" },
  { "tag": "b" },
  { "tag": "c" }
]

note: flatten sounds a little bit colloquial but is on the other hand closer to English than flatMap or selectMany which are terms from functional languages and algebraic data processing

distinct transformation :

Extend the $apply transformation grammar to include:

apply = ... / distinct
distinct= "distinct()"

which transforms the collection into the collection of the same structure with duplicate values removed (TODO: precise meaning of duplicate, is this structural equality ?)

[mikepizzo]

Can we have a recursive version of flatten that would, for example, flatten all reports under a manager, or members of a group?

i.e., if an employee has directReports, it would be nice to be able to aggregate all reports of a particular employee, recursively:

/employees/1?$apply=recursiveFlatten(directReports as reports)

or, if directReports was on a derive manager type,

/employees/1?$apply=recursiveFlatten(ns.manager/directReports as reports)

similarly, if a group had members that could be users or other groups, you could expand the recursive set of members for each group:

/groups?$expand=members($apply=recursiveFlatten(ns.group/members as members))

or, I could get all members across all groups:

/groups?$apply=recursiveFlatten(ns.group/members as members)