Skip to content

feat: support Ibis 6.x UDF annotations #718

@omriel1

Description

@omriel1

What happened?

Hi!
I've been trying to create a substrait plan with a udf using the new annotations (udf.scalar.python, starting of ibis 6.x). For some reason it fails. This is what I tried (this udf is taken from the ibis blog: https://ibis-project.org/blog/rendered/ibis-version-6.0.0-release/#udfs) :

import ibis
from ibis import _

from ibis import udf
from ibis_substrait.compiler.core import SubstraitCompiler

t = ibis.examples.penguins.fetch()

@udf.scalar.python
def num_vowels(s: str) -> int:
    return sum(map(s.lower().count, "aeiou"))

new_t = t.mutate(num_vowels=lambda t: num_vowels(t.species))
query = new_t.filter(_.num_vowels>5)

compiler = SubstraitCompiler(
    udf_uri="urn:arrow:substrait_simple_extension_function"
)
plan = compiler.compile(query)

I'd expect it to be compiled into a Substrait plan, which is similar to those created in your tests (https://github.com/ibis-project/ibis-substrait/blob/main/ibis_substrait/tests/integration/test_pyarrow.py)

I'll also mention that my end goal is just create an arbitrary udf using Ibis, create a query and compile it into a Substrait plan. Then register the udf in pyarrow/DuckDB and consume the plan. While it works fine for me with the legacy annotations, I wonder how should it be done now.
Thanks!

What version of ibis-substrait are you using?

ibis_substrait-2.29.1

What substrait consumer(s) are you using, if any?

Currently none, Ideally Acero and duckDB

Relevant log output

args = (<ibis.expr.operations.relations.DatabaseTable object at 0x14ff56b60>,)
kwargs = {'child_rel_field_offsets': {}, 'compiler': <ibis_substrait.compiler.core.SubstraitCompiler object at 0x13fb12e50>}

    @functools.singledispatch
    def translate(*args: Any, **kwargs: Any) -> Any:
>       raise NotImplementedError(*args)
E       NotImplementedError: <ibis.expr.operations.relations.DatabaseTable object at 0x14ff56b60>

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    Status

    backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions