-
Notifications
You must be signed in to change notification settings - Fork 23
Description
What happened?
Hi!
I've been trying to create a substrait plan with a udf using the new annotations (udf.scalar.python
, starting of ibis 6.x). For some reason it fails. This is what I tried (this udf is taken from the ibis blog: https://ibis-project.org/blog/rendered/ibis-version-6.0.0-release/#udfs) :
import ibis
from ibis import _
from ibis import udf
from ibis_substrait.compiler.core import SubstraitCompiler
t = ibis.examples.penguins.fetch()
@udf.scalar.python
def num_vowels(s: str) -> int:
return sum(map(s.lower().count, "aeiou"))
new_t = t.mutate(num_vowels=lambda t: num_vowels(t.species))
query = new_t.filter(_.num_vowels>5)
compiler = SubstraitCompiler(
udf_uri="urn:arrow:substrait_simple_extension_function"
)
plan = compiler.compile(query)
I'd expect it to be compiled into a Substrait plan, which is similar to those created in your tests (https://github.com/ibis-project/ibis-substrait/blob/main/ibis_substrait/tests/integration/test_pyarrow.py)
I'll also mention that my end goal is just create an arbitrary udf using Ibis, create a query and compile it into a Substrait plan. Then register the udf in pyarrow/DuckDB and consume the plan. While it works fine for me with the legacy annotations, I wonder how should it be done now.
Thanks!
What version of ibis-substrait are you using?
ibis_substrait-2.29.1
What substrait consumer(s) are you using, if any?
Currently none, Ideally Acero and duckDB
Relevant log output
args = (<ibis.expr.operations.relations.DatabaseTable object at 0x14ff56b60>,)
kwargs = {'child_rel_field_offsets': {}, 'compiler': <ibis_substrait.compiler.core.SubstraitCompiler object at 0x13fb12e50>}
@functools.singledispatch
def translate(*args: Any, **kwargs: Any) -> Any:
> raise NotImplementedError(*args)
E NotImplementedError: <ibis.expr.operations.relations.DatabaseTable object at 0x14ff56b60>
Metadata
Metadata
Assignees
Labels
Type
Projects
Status