How to formalise interactions between extra defaults, dynamic/variant functions, and needextends filters/operations?

[chrisjsewell]

TLDR; this is currently a blocker for correct ontology and type casting of field values, since some behaviours are based around values only ever being strings.

---

Take this example:

.. req:: ...
   :id: A

.. req:: ...
   :id: B
   :extra1: a
   :extra2: b

.. req:: ...
   :id: C
   :extra1: [[func_a()]]

.. needextend:: "a" in extra1
   :+extra1: [[func_b()]]
   :-extra2:

Currently, the needs selected would be B and C, due to the need "contexts" seen by the needextend filter being:

[{"id": "A", "extra1": "",  "extra2": ""},
 {"id": "B", "extra1": "a", "extra2": "b"},
 {"id": "C", "extra1": "[[func_a()]]", "extra2": ""}]

The "final" needs, before dynamic function "resolution", would then be:

[{"id": "A", "extra1": "[[func_b()]]",  "extra2": ""},
 {"id": "B", "extra1": "a [[func_b()]]", "extra2": ""},
 {"id": "C", "extra1": "[[func_a()]] [[func_b()]]", "extra2": ""}]

There are a number of "implicit" behaviours going on here:

1. extras that are not specifically given in a directive are set to an empty string ""
2. extras that contain dynamic functions are just treated as "regular" strings when generating the needextend filter context
3. when applying + (append) extend operations to extras, the strings are just concatenated
4. when applying - (delete) extend operations to extras, the fields are set to the empty string

---

Taking a corner case for these implicit behaviours, you could even have, for example:

.. req:: ...
   :id: D
   :extra1: [[func_b(

.. needextend:: id == "D"
   :+extra1: )]]

resulting in a need that ends up having a dynamic function that was not originally there:

{"id": "D", "extra1": "[[func_b( )]]"}

---

what if we now wanted to say both extra1 and extra2 should be integer fields, and had, for example:

.. req:: ...
   :id: A

.. req:: ...
   :id: B
   :extra1: 1
   :extra2: 2

.. req:: ...
   :id: C
   :extra1: [[func_a()]]

.. needextend:: extra1 > 0
   :+extra1: [[func_b()]]
   :-extra2:

For the simple B case, I believe we would want to initially read from the directive:

{"id": "B", "extra1": 1, "extra2": 2}

i.e. coercing the string values into integers

But how should the needs with dynamic functions, filter context and final needs look like?

1. should integer type extras, that are not specifically given in a directive, be set to 0?
2. how should dynamic functions be handled when generating the needextend filter context?
3. how do you treat + (append) extend operations, especially when the original and/or append value are dynamic functions?
4. should applying - (delete) extend operations result in a 0 value

---

In #1497, I try to bring the dynamic function "identification" into the initial processing of the need directives,
but it is really difficult to think how this should then interact with the needextend filter context and operations

---

The example above also does not take into account two other important questions:

• should we be trying to resolve dynamic functions from fields in needs read from needs.json, via needs_external_needs or needimport?
  The fields in these needs I would generally consider to be already "fully resolved"
• This doesn't even consider the mess that is variant functions: https://sphinx-needs.readthedocs.io/en/latest/directives/need.html#needs-variant-support, and their horrible implementation:

  sphinx-needs/sphinx_needs/utils.py

  Line 522 in 6f90e99

  def match_variants(