♻️ Store needs as NeedItem / NeedPartItem, rather than standard dict

[chrisjsewell]

Currently, need data items are stored as standard Python dicts, which is also exposed to the user in various APIs and as the needs.json.
This is problematic as it does not allow for

1. any internal checks to ensure data is stored/updated consistently (e.g. fields like id should never be changed),
2. any data to be "hid" from the user, or presented in a different way to its internal structure.
   • because the dict is a flat key/val store, it does not allow for distinction between internal / user set extra/link fields
   • we would like to improve how values that are dynamic/variant functions are stored (prior to evaluation)
   • we would also like to store source mapping data from distinct sources (directives, external, imports) in a more coherent way.
3. deprecations of any fields in a non-breaking manner

In this PR, we instead add a specific NeedItem class, which closely mimics adict, but also allows for additional constraints and functionality.

We also introduce a NeedPartItem which is distinctly different from a NeedItem, e.g. it should not be mutated.

To ensure that no existing user code is accidentally breaking this, e.g. by directly add dictionaries to the list of needs,
we perform a check, at the end of the post-processing stage, to make sure all need items are instances of NeedItem
(and point them towards the "correct" add_need API)

---

‼️ Breaking

This would be breaking for any users doing "non-API" modifications / additions to the needs data, i.e. directly adding dict items.
It should not change interactions with standard APIs like add_need or filter strings, etc

[codecov]

Codecov Report

❌ Patch coverage is 89.86175% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.74%. Comparing base (4e10030) to head (d30b46f).
⚠️ Report is 145 commits behind head on master.

Files with missing lines	Patch %	Lines
sphinx_needs/need_item.py	80.21%	18 Missing ⚠️
sphinx_needs/directives/need.py	87.50%	1 Missing ⚠️
sphinx_needs/functions/common.py	83.33%	1 Missing ⚠️
sphinx_needs/layout.py	87.50%	1 Missing ⚠️
sphinx_needs/views.py	92.30%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1485      +/-   ##
==========================================
+ Coverage   86.87%   88.74%   +1.86%     
==========================================
  Files          56       68      +12     
  Lines        6532     8473    +1941     
==========================================
+ Hits         5675     7519    +1844     
- Misses        857      954      +97     

Flag	Coverage Δ
pytests	88.74% <89.86%> (+1.86%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ubmarco]

This looks good to me and is a required step to get people to use the API or complain if something is missing. We should inform core users about this change.
I just have 2 comments, please decide whether those are issues right now or not.

[ubmarco]

id_parent should be optional, it's about need part parent - would it break the data model to change it to str | None ?

[chrisjsewell]

This particular PR is not intended to change any schema specifics; e.g. it shows that this change does not change any output needs.json
So it may well be a legitimate change, but should not be made here

[ubmarco]

same here, parent_need is optional, it's about need nesting. Wouldn't str | None be better?

[chrisjsewell]

Same as above