Add QUANTILE function to PyDough #378

john-sanchez31 · 2025-06-20T21:50:37Z

Resolves issue #373

knassre-bodo

Great job @john-sanchez31, just a few comments and then it will be ready to merge

knassre-bodo · 2025-06-27T04:20:18Z

documentation/functions.md

+
+> [!NOTE]
+> `QUANTILE(X, 0.5)` is equivalent to `MEDIAN(X)`.  
+> The implementation uses the SQL standard `PERCENTILE_DISC` aggregate function where available.


Better phrasing: it is equivalent to the common PERCENTILE_DISC SQL aggregation function.

knassre-bodo · 2025-06-27T04:20:46Z

documentation/functions.md

+- If the quantile argument is not a valid number between 0 and 1, an error is raised.
+
+> [!NOTE]
+> `QUANTILE(X, 0.5)` is equivalent to `MEDIAN(X)`.  


This is incorrect, it isn't actually equivalent to MEDIAN it is just very similar to MEDIAN. The difference is that MEDIAN will, if there is an even number of rows, take the average of the two median rows, but QUANTILE (aka PERCENTILE_DISC) always selects a row instead of interpolating between rows.

I've made the corrections

knassre-bodo · 2025-06-27T04:21:21Z

documentation/functions.md

+# Returns the value at the 90th percentile of supply costs for each supplier
+Suppliers.CALCULATE(ninetieth_percentile_cost = QUANTILE(supply_records.supply_cost, 0.9))
+
+# Returns the median supply cost for each supplier (equivalent to MEDIAN)


Again don't say equivalent median, say the 50th percentile

Fixed as well

knassre-bodo · 2025-06-27T04:22:50Z

documentation/functions.md

+
+### QUANTILE
+
+The `QUANTILE` function returns the value at a specified quantile from the set of values it is called on. The quantile value `p` must be a numeric literal between 0 and 1 (inclusive), where `0` returns the minimum, `1` returns the maximum, and `0.5` returns the median.


You need to be very specific about what kind of quantile you are talking about here, since there are many different kinds of quantile computations in math/analytics. Specifically:

When doing QUANTILE(x, p), returns the smallest value of x such that p of the rows are less than or equal to it (this is what PERCENTILE_DISC is)

Ignores NULL records in computation

knassre-bodo · 2025-06-27T04:23:55Z

pydough/conversion/hybrid_translator.py

+        Rewrites a QUANTILE aggregation call into an equivalent expression using window functions.
+        This is typically used for dialects that do not natively support the PERCENTILE_DISC
+        aggregate function.
+
+        The rewritten expression selects the value at the specified quantile by:
+        - Ranking the rows within each partition.
+        - Calculating the number of rows (N) in each partition.
+        - Keeping only those rows where the rank is greater than INTEGER((1.0 - p) * N),
+            where p is the quantile argument.
+        - Taking the maximum value among the kept rows.
+
+        Args:
+            child_connection: The HybridConnection containing the aggregate call to QUANTILE.
+            expr: The HybridFunctionExpr representing the QUANTILE aggregation.
+            create_new_calc: If True, injects new expressions into a new CALCULATE operation.
+
+        Returns:
+            A HybridFunctionExpr representing the rewritten aggregation using window functions.


Make sure none of these lines are over 80 characters

knassre-bodo · 2025-06-27T04:28:55Z

pydough/conversion/hybrid_translator.py

+        )
+
+        # (1.0-args[1])
+        sub: HybridExpr = HybridFunctionExpr(pydop.SUB, [one, p], NumericType())


Since we know p is a numeric literal, we can forgo this expr entirely and just pass in the new literal:

Suggested change

sub: HybridExpr = HybridFunctionExpr(pydop.SUB, [one, p], NumericType())

sub: HybridExpr = HybridLiteralExpr(Literal(1.0 - p.literal.value, NumericType()))

Changed. It changed the sql, just make sure everything still looks fine

knassre-bodo · 2025-06-27T04:29:27Z

pydough/sqlglot/transform_bindings/base_transform_bindings.py

+        Converts a PyDough QUANTILE(X, p) function call to a SQLGlot expression
+        representing the SQL standard PERCENTILE_DISC aggregate function.
+
+        This produces an expression equivalent to:
+            PERCENTILE_DISC(p) WITHIN GROUP (ORDER BY X)
+
+        Args:
+            args: A list of two SQLGlot expressions, where args[0] is the column or
+                expression to order by (X), and args[1] is the quantile value (p) between 0 and 1.
+            types: The PyDough types of the arguments.
+
+        Returns:
+            A SQLGlotExpression representing the PERCENTILE_DISC(p) WITHIN GROUP (ORDER BY X)
+            aggregate function.


80 character lines

knassre-bodo · 2025-06-27T04:29:50Z

pydough/sqlglot/transform_bindings/base_transform_bindings.py

+        if (
+            not isinstance(args[1], sqlglot_expressions.Literal)
+            or args[1].is_string
+            # or not isinstance(args[1].this, (int, float))


Either fix or delete this, but don't leave the comment as dead code

knassre-bodo · 2025-06-27T04:30:20Z

pydough/sqlglot/transform_bindings/base_transform_bindings.py

+
+        assert len(args) == 2
+
+        # validation


This comment should be clearer: # Validate that the second argument is a numeric literal between 0 and 1

knassre-bodo

A few comments remaining while you address the testing

knassre-bodo · 2025-07-02T20:23:30Z

pydough/conversion/hybrid_translator.py

+        Rewrites a QUANTILE aggregation call into an equivalent expression using
+        window functions.
+        This is typically used for dialects that do not natively support the
+        PERCENTILE_DISC
+        aggregate function.


The newlines here are a bit off. Also, wrap functions like QUANTILE or PERCENTILE_DISC in backticks (`) for tooltip formatting.

Please address this before merging

knassre-bodo · 2025-07-03T02:05:59Z

tests/test_sql_refsols/quantile_test_4_ansi.sql

+WITH _s0 AS (
+  SELECT
+    n_name,
+    n_nationkey,
+    n_regionkey
+  FROM tpch.nation
+  ORDER BY


Don't forget to delete the test_3 and test_4 SQL files since they are no longer part of test_pydough_to_sql.py.

knassre-bodo · 2025-07-03T02:06:40Z

tests/test_pipeline_tpch_custom.py

+            PyDoughPandasTest(
+                quantile_function_test_1,
+                "TPCH",
+                # Answer


You don't need to write # Answer here every time

knassre-bodo · 2025-07-03T17:09:04Z

pydough/conversion/hybrid_translator.py

+        Rewrites a QUANTILE aggregation call into an equivalent expression using
+        window functions.
+        This is typically used for dialects that do not natively support the
+        PERCENTILE_DISC
+        aggregate function.


Please address this before merging

Initial commit

afd45d4

john-sanchez31 self-assigned this Jun 20, 2025

john-sanchez31 added documentation Improvements or additions to documentation extensibility Increasing situations in which PyDough works effort - medium mid-sized issue with average implementation time/difficulty labels Jun 20, 2025

john-sanchez31 linked an issue Jun 20, 2025 that may be closed by this pull request

Add QUANTILE function to PyDough #373

Closed

john-sanchez31 added 10 commits June 23, 2025 14:08

Initial implementation rewrite

6f60811

sqlglot quantile implementation

f6d8ad4

sqlglot quantile implementation fixed

3f66a30

quantile documentation

b7fef56

Merge branch 'main' into John/quantile

789b7c6

to sql test

08fe280

generated test files

757a7bd

e2e test added

454e24c

Merge branch 'main' into John/quantile

a8f3f7b

test finalized [RUN CI]

19451f6

john-sanchez31 marked this pull request as ready for review June 25, 2025 22:03

john-sanchez31 requested review from knassre-bodo and juankx-bodo June 25, 2025 22:03

knassre-bodo reviewed Jun 27, 2025

View reviewed changes

john-sanchez31 added 2 commits June 30, 2025 17:07

documentation and comments fixed [RUN CI]

6a9cda5

unnecessary var deleted [RUN CI]

c5b91a4

john-sanchez31 requested a review from knassre-bodo June 30, 2025 23:26

john-sanchez31 and others added 3 commits July 2, 2025 18:12

e2e test added (not working)

55dcf45

Fixing quantile behavior with aggregate window function input ejection

d8e5641

Fixing quantile behavior with aggregate window function input ejection

c169db2

knassre-bodo reviewed Jul 3, 2025

View reviewed changes

tests and comments fixed [RUN CI]

272492c

john-sanchez31 requested a review from knassre-bodo July 3, 2025 15:12

knassre-bodo approved these changes Jul 3, 2025

View reviewed changes

comment fixed [RUN CI]

22c722e

john-sanchez31 merged commit 85838f5 into main Jul 3, 2025
5 checks passed

john-sanchez31 deleted the John/quantile branch July 3, 2025 18:00


		### QUANTILE

		The `QUANTILE` function returns the value at a specified quantile from the set of values it is called on. The quantile value `p` must be a numeric literal between 0 and 1 (inclusive), where `0` returns the minimum, `1` returns the maximum, and `0.5` returns the median.

	sub: HybridExpr = HybridFunctionExpr(pydop.SUB, [one, p], NumericType())
	sub: HybridExpr = HybridLiteralExpr(Literal(1.0 - p.literal.value, NumericType()))

Add QUANTILE function to PyDough #378

Add QUANTILE function to PyDough #378

Uh oh!

Conversation

john-sanchez31 commented Jun 20, 2025

Uh oh!

knassre-bodo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

knassre-bodo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!