Move to using minijinja for filter expressions?

[chrisjsewell]

There are a number of problems with the "Python filter strings" currently used, but mainly I want to focus on undefined and nulled values.

When evaluating a Python expression, if any variables are not in the context, then it will fail with a NameError,
or if you want to do a test on a "nullable" value, like "a" in None, then this will fail with a TypeError

This is a problem:

1. if you want to allow need types to have nullable fields, i.e. if they are not specified by the user,
2. if you want to allow need types to have different fields, for example, I have seen projects with 100s of extra_optionswhen really most of these fields are only applicable to one or a few need types
3. it would have very good compatibility with ubcode

With https://pypi.org/project/minijinja/ you get sane handling of undefined and nullable values, with still a lot of builtin filters and tests that should satisfy most use cases, for example:

import minijinja
minijinja.eval_expr('a|upper is startingwith "A"')   # -> False
minijinja.eval_expr('a|upper is startingwith "A"', a=None)   # -> False
minijinja.eval_expr('a|upper is startingwith "A"', a="a")   # -> True

See also https://mitsuhiko.github.io/minijinja-playground/

---

Obviously this could be quite back-breaking, but I feel it would be worth the effort

---

It is also important to note the context in which a filter string is used, which can have subtle differences:

• filtering in needimport; we are filtering on the data of an item within a needs.json, not necessarily the "fully resolved" need.
• filtering in need_global_options predicates; we are filtering on an "incompletely resolved" need
• filtering in needextend; we are filtering on an "incompletely resolved" need and need graph, e.g. it could still have unresolved dynamic functions and the backlinks have not been computed.
• filtering in needtable and other "analysis" directives; we are now filtering on the "fully resolved" need (and need graph).