If you're using %%cell_to_module my_module --execute it will try to push nodes back into the user namespace. This fails if the node includes a . in its name (internal nodes of @pipe_input, @pipe_output, @mutate, @load_from, etc.)
The following allows to only execute and push the specified nodes, and avoid this issue:
## cell 1
final_vars = ["foo", "bar"]
## cell 2
%%cell_to_module my_dataflow --execute final_vars
from hamilton.function_modifiers import pipe_input, step
def foo() -> dict:
return {...}
def _invert_mapping(d: dict) -> dict:
return {v: k for k, v in d.items()}
@pipe_input(step(_invert_mapping))
def bar(foo: dict) -> dict:
return foo
However, this will when calling %%cell_to_module --execute without an argument because it tries to execute and push all nodes (including the generated bar.with_invert_mapping).
Proposed solution
Before pushing back results to the user namespace, check if the name includes a . and replace by __. The double underscore is a good option because Hamilton already ignores __ suffix when parsing node names. Using single underscore is likely to cause naming conflicts with other user-created nodes
If you're using
%%cell_to_module my_module --executeit will try to push nodes back into the user namespace. This fails if the node includes a.in its name (internal nodes of@pipe_input,@pipe_output,@mutate,@load_from, etc.)The following allows to only execute and push the specified nodes, and avoid this issue:
However, this will when calling
%%cell_to_module --executewithout an argument because it tries to execute and push all nodes (including the generatedbar.with_invert_mapping).Proposed solution
Before pushing back results to the user namespace, check if the name includes a
.and replace by__. The double underscore is a good option because Hamilton already ignores__suffix when parsing node names. Using single underscore is likely to cause naming conflicts with other user-created nodes