needextend doesn’t work correctly with incremental build

[twodrops]

If needextend is used in an rST file and then removed (with file modification), the extended behavior including modification flags are not removed. Only a clean build removes the extension.

Beginning:

After adding a 2nd needextend + build:

Removed previous 2nd needextend + build (expected: modifications = 1 and REQ_SYS_2 is removed):

[danwos]

Interesting bug, can't imagine right now, why this is not working.

Sure, Sphinx keeps all needs in its cached environment.
But a changed file should recreate the related need and overwrite the entry in the cache.
Need to debug this. Will try to fix this already with upcoming release.

Thanks for the good description 👍

[Chriner]

Perhaps this works only, if the need is in the same file as the needextend.
When needextend is alone in the file and only this file is changed, the file with the need is not read again and so the cache might not be overwritten?
In described issue, the need was imported with needsimport and modified in another rst file with needextend. Perhaps this also makes a difference.

[danwos]

@Chriner that's a good hint.
The need gets created, manipulated and stored during the first build.
On the second build, the need gets not recreated (as the related file was not changed), so it is taken with all manipulations from cache.

Puhh, that's a tough problem as I see no way to get the "original" need back from cache, so that it would not contain the manipulations. Only if it gets read in again (full build) the problem is solved.

solution idea

The only solution I see is a complete change in the architecture of Sphinx-Needs.
Currently we only store the final need-values, no matter how many manipulations and other voodoo has happened.
We only have the current set of data. No information about how it may have looked before.

Sphinx-Needs would need to store all manipulations / transformations in an extra storage.
And put this transformation together to get the final need and its values.
If a needextend gets removed, it most be removed from the transformation storage, so that the final "result" of a need would look different, even if the rst-file with the original need was no changed.

So Sphinx-Needs would recalculated all needs based on the stored transformations during each build (also incremental builds)

This change could break internally a lot and would need ~1-2 weeks full-time development.

Is this scenario of a removed needextend a common case in your project?
And is a full-build not acceptable as workaround?

[Chriner]

I did some tests and found out there is now an additional error since the incremental file writing works. I see now two problems:

1. Needs cache is wrong:
   This is described in this issue. When deleting needextend, the cache is not updated. But it works when needextend is modified. How can this be? All in the same file as the modified needextend.

2. Affected files are not rewritten
   No the incremental file writing works also with Sphinx-Needs. So changed needextend are not updated in other files. I think this is the same as in Incremental build support #354.

So I think it only helps if both problems are solved. Also there might be another solution:

Read in all files if needs are changed to build the need cache new. Then depending of the changes, it can be decided which files must be generated again. For me it looks like reading the files is very fast.

We have to decide if incremental build is always reliable or not.

[danwos]

Regarding reliability of an incremental build, I think it is not possible to get a 100% reliable, incremental build.
Mainly because need-values can be based on code and therefore on external values (e.g. a JIRA ticket field), which we can not check for update status.

As we already have figured out, we would need to run all need calculations and all needtables (and co.) filters again to be sure that their data is up to date and then decide, which files needs to be rewritten by Sphinx.

This will be fast compared to a full build, mostly because Sphinx does not need to write all files.
But I'm not sure if it is still fast enough for a developer to e.g. use it in IDE previews or as a fast check.

For me this would not be an "incremental build" anymore, because it will be perceptible slower.
It would be something between an incremental and a full build, maybe called "fastest reliable build" ;)

But I'm not sure if there are strong use cases, which need a "fastest reliable build" .
As developer I want a fast lookout of the result and I normally know if e.g. tables can not be up to date.
For a final, reliable build I throw the docs to a CI system per release, which maybe runs 1-2 hours (but that's okay).

In the moment I would like to not work on a "fastest reliable build" feature, as a lot of other stuff is open.
For sure bugs like the one in this issue needs to be resolved as good as possible.

But I'm open for any counter-arguments :)

[Chriner]

A "fastest reliable build" could be interesting, but doesn't have high prio. But there is a behaviour of needextend which I don't understand and might help in this problem:

In a clean build, the need is created and then modified by several needextend. The result is stored in cache and it is not possible to find out, how this result is calculated.

So If a needextend is deleted from rst file, the modification is not removed from the cache and output of that rst file is like the needextend is not modified.

In contrast to above behaviour following example: needextend has several links to other needs. If a single link is removed, I would expect the link is still in the cache and part of the rst output. But it is also removed in the output. How can this happen? It looks like part of the information is available (in cache or somewhere else)? Perhaps this could also be used for deleting a needextend?