Update braille when control+v, control+x, control+y, control+z, backspace or control+backspace is pressed and UIA not used in ms word #15491

burmancomp · 2023-09-21T07:49:50Z

Link to issue number:

Summary of the issue:

After paste, cut, redo, undo, backspace or control+backspace braille line was not necessarily updated when using object model.

Description of user facing changes

Braille line and review position should always be updated when using object model ("Use UI Automation to access Microsoft Word document controls" is not set to always in NVDA's advanced settings category), and paste (control+v), cut (control+x), redo (control+y) or undo (control+z / alt+backspace) is pressed. Braille should be also updated when using object model and backspace or control+backspace is pressed and held down.

Braille and review position are also updated when using UIA, and braille is tethered to review and review follows caret.

Description of development approach

To get control+v, control+x, control+y, control+z and alt+backspace to work, added helper script script_updateBrailleAndReviewPosition to NVDAObjects.IAccessible\winword.WordDocument, and assigned corresponding keyboard shortcuts to it. Minor modification of event_caret so that braille is always updated when caret event is executed from script_updateBrailleAndReviewPosition. To update braille always when backspace or control+backspace are pressed and held down, overridden _backspaceScriptHelper function.

In addition, to get braille updated when tethered to review and when review follows caret, modified event_textChange in UIA\wordDocument.py.

Testing strategy:

Tested with word 2019 (version 2308 Build 16.0.16731.20182) 32-bit using Albatross 80. More testing with various configurations would be needed.

Known issues with pull request:

none at this moment

Code Review Checklist:

Documentation:
- Change log entry
- User Documentation
- Developer / Technical Documentation
- Context sensitive help for GUI changes
Testing:
- Unit tests
- System (end to end) tests
- Manual testing
UX of all users considered:
- Speech
- Braille
- Low Vision
- Different web browsers
- Localization in other languages / culture than English
API is compatible with existing add-ons.
Security precautions taken.

burmancomp · 2023-09-21T07:50:41Z

cc: @LeonarddeR

LeonarddeR · 2023-09-21T08:05:49Z

source/NVDAObjects/IAccessible/winword.py

@@ -47,12 +47,6 @@ def _get_ignoreEditorRevisions(self):
 	ignoreFormatting=False

 	def event_caret(self):


I think we need to research why this code was ever added. It should be possible to do this with a git blame.

@michaelDCurran if super(WordDocument,self).event_caret() is not executed there seems to be problem that braille is not always updated when backspace is pressed (problem should occur at least if backspace is pressed and hold down).

I tent to agree with @LeonarddeR that checking with git blame in which commit this was introduced is safer. Would you be able to do this?

source/NVDAObjects/UIA/wordDocument.py

LeonarddeR · 2023-09-21T08:13:10Z

source/NVDAObjects/UIA/wordDocument.py


 	def event_UIA_notification(self, activityId=None, **kwargs):
 		# #10851: in recent Word 365 releases, UIA notification will cause NVDA to announce edit functions
 		# such as "delete back word" when Control+Backspace is pressed.
 		if activityId == "AccSN2":  # Delete activity ID
 			return
 		super(WordDocument, self).event_UIA_notification(**kwargs)
+		# Try to ensure that braille is updated when UIA is not used and


I don't think it is a good idea to do this here. There must be another event we're missing, and if not, it's simply a bug in Word that needs to be reported and fixed on their end.

It seems that sometimes event_caret (IAccessible\winword.py) is fired when control+z is pressed and sometimes not. UIA_notification event seems to be fired always.

I think I have tried to add textChange (and textInsert/textRemove) events to IAccessible\winword.py but I suppose they are not fired.

Do you intend this to work for Non UIA cases with this code? If so I think it's a false positive, because this code is not active when UIA is disabled.

When "Use UI Automation to access Microsoft Word document controls Only when necessary" event_UIA_notification is fired when pressing control+x and control+z.

Can reason be that object model is not available for some reason and thus UIA is used?

If a notification event is fired for undo but neither a caret nor a text change event is fired, this is something Microsoft needs to fix.

CyrilleB79 · 2023-09-21T09:22:54Z

Have you double checked other editing commands: del, shift+del, shift+backspace, redo (control+Y), etc.?

CyrilleB79 · 2023-09-21T11:52:31Z

@burmancomp, keep in mind that "Use UI Automation to access Microsoft Word document controls" only applies to situations when NVDA queries informations such as curent line, table, current text formatting, etc.

I does not apply to event triggered by an action and thus to UIA notifications. So does not apply to notifications received when performing control+Z or control+U.

Also to check if you are using UIA or object model to access Word document, you can press NVDA+F1 and look at dev info at the end of the log. If the dev info ends with UIA peroperties, you are accessing the document via UIA; else, if it ends with IAccessible properties, you are accessing it through object model.

At last, beware to #13704 when testing with Word: be sure to change the focus at least once before doing tests.

LeonarddeR · 2023-09-21T13:35:59Z

I does not apply to event triggered by an action and thus to UIA notifications. So does not apply to notifications received when performing control+Z or control+U.

I don't think that's true. Word sends UIA notifications to its UIA control, so when that control is used, NVDA will receive those. If the object model is active and uIA is disabled for Word, the notifications will still be send but NVDA silently ignores them anyway.

CyrilleB79 · 2023-09-21T17:02:59Z

I does not apply to event triggered by an action and thus to UIA notifications. So does not apply to notifications received when performing control+Z or control+U.

I don't think that's true. Word sends UIA notifications to its UIA control, so when that control is used, NVDA will receive those. If the object model is active and uIA is disabled for Word, the notifications will still be send but NVDA silently ignores them anyway.

@LeonarddeR, I have just tested with Word 2016, version '16.0.16731.20234', with "Use UIA" option to "Only when necessary", and I have double checked that UIA is actually not used with looking at dev info in the log viewer. I can confirm that even if NVDA uses object model to access Word document, the following actions trigger UIA notifications:

copy (control+C), cut (ctrl+X), paste (ctrl+V)
cancel (ctrl+Z), redo (ctrl+Y)
formatting bold (ctrl+G in French), italic (ctrl+I) and underline (ctrl+U): in this last case, both UIA notifications and NVDA's ui.message in formatting scripts are spoken.

LeonarddeR · 2023-09-21T19:47:28Z

@LeonarddeR, I have just tested with Word 2016, version '16.0.16731.20234', with "Use UIA" option to "Only when necessary", and I have double checked that UIA is actually not used with looking at dev info in the log viewer. I can confirm that even if NVDA uses object model to access Word document, the following actions trigger UIA notifications:

Wow, that's pretty spooky. In any case, that means we should not touch the notification code for now.

burmancomp · 2023-09-21T20:03:39Z

If event_UIA_notification code is not touched how braille is updated? I am suspicious that Microsoft would fix this. It is issue of older object model.

seanbudd · 2023-09-21T23:58:04Z

Please update the PR title to be more descriptive

burmancomp · 2023-09-26T10:46:35Z

@LeonarddeR what do you think about code now?

LeonarddeR · 2023-09-26T10:56:25Z

This is not going to work for older version of Word that don't provide UIA notifications.

burmancomp · 2023-09-26T12:16:55Z

How old office versions NVDA should support (is there official policy)? Support of office 2010 ended on october 2020, and support of office 2013 ended in april 2023.

@CyrilleB79 told in his comment about UIA notifications.

This pull request could be limited to try to solve issue from 2016.

source/NVDAObjects/UIA/wordDocument.py

AppVeyorBot · 2023-09-26T18:41:52Z

PASS: Translation comments check.
PASS: Unit tests.
PASS: Lint check.
FAIL: System tests (tags: installer NVDA). See test results for more information.
Build (for testing PR): https://ci.appveyor.com/api/buildjobs/3m2d62mehaug5s15/artifacts/output/nvda_snapshot_pr15491-29418,cc1375fb.exe
CI timing (mins):
INIT 0.0,
INSTALL_START 1.0,
INSTALL_END 0.9,
BUILD_START 0.0,
BUILD_END 24.9,
TESTSETUP_START 0.0,
TESTSETUP_END 0.3,
TEST_START 0.0,
TEST_END 13.9,
FINISH_END 0.1

See test results for failed build of commit cc1375fb40

AppVeyorBot · 2023-09-26T21:01:45Z

PASS: Translation comments check.
PASS: Unit tests.
PASS: Lint check.
FAIL: System tests (tags: installer NVDA). See test results for more information.
Build (for testing PR): https://ci.appveyor.com/api/buildjobs/l7poqgx51cyt7udg/artifacts/output/nvda_snapshot_pr15491-29419,51a93734.exe
CI timing (mins):
INIT 0.0,
INSTALL_START 0.8,
INSTALL_END 0.9,
BUILD_START 0.0,
BUILD_END 24.2,
TESTSETUP_START 0.0,
TESTSETUP_END 0.3,
TEST_START 0.0,
TEST_END 22.4,
FINISH_END 0.2

See test results for failed build of commit 51a93734a0

AppVeyorBot · 2023-10-06T13:02:31Z

PASS: Translation comments check.
PASS: Unit tests.
PASS: Lint check.
FAIL: System tests (tags: installer NVDA). See test results for more information.
Build (for testing PR): https://ci.appveyor.com/api/buildjobs/yhp7kdn50ub37dl2/artifacts/output/nvda_snapshot_pr15491-29540,c9815650.exe
CI timing (mins):
INIT 0.0,
INSTALL_START 1.0,
INSTALL_END 1.0,
BUILD_START 0.0,
BUILD_END 25.2,
TESTSETUP_START 0.0,
TESTSETUP_END 0.3,
TEST_START 0.0,
TEST_END 5.2,
FINISH_END 0.2

See test results for failed build of commit c9815650e8

burmancomp · 2023-10-07T06:39:16Z

As to letters y or z sometimes typed when control+y or control+z is pressed and held down, I suppose it is not issue of this pull request.

I have noticed that when backspace or control+backspace are pressed, held down and then released speech may be inaccurate (not reporting most recent deletion). This may happen also when using current main branch so I suppose this behavior is not merely issue of this pull request.

source/NVDAObjects/IAccessible/winword.py

seanbudd · 2023-10-09T00:13:11Z

Please put the word "fixes" or "closes" before the issue number in the PR description so GitHub automatically links the issue

source/NVDAObjects/IAccessible/winword.py

user_docs/en/changes.t2t

source/NVDAObjects/IAccessible/winword.py

lukaszgo1

LGTM!

seanbudd

Thanks @burmancomp

fix 3276

6c911ac

LeonarddeR suggested changes Sep 21, 2023

View reviewed changes

burmancomp changed the title ~~fix 3276~~ Update braille when control+z, control+x or backspace is pressed and UIA not used Sep 22, 2023

burmancomp changed the title ~~Update braille when control+z, control+x or backspace is pressed and UIA not used~~ Update braille when control+z, control+x or backspace is pressed and UIA not used in ms word Sep 22, 2023

Merge branch 'master' into fix_3276

3d08f47

seanbudd added this to the 2024.1 milestone Sep 26, 2023

burmancomp added 2 commits September 26, 2023 12:09

Merge branch 'master' into fix_3276

2dadf66

run added code when UIA should not be used, and minor modifications

823daf6

burmancomp changed the title ~~Update braille when control+z, control+x or backspace is pressed and UIA not used in ms word~~ Update braille when control+v, control+x, control+z or backspace is pressed and UIA not used in ms word Sep 26, 2023

lukaszgo1 suggested changes Sep 26, 2023

View reviewed changes

source/NVDAObjects/UIA/wordDocument.py Outdated Show resolved Hide resolved

source/NVDAObjects/UIA/wordDocument.py Outdated Show resolved Hide resolved

minor fix

02f94ed

using helper scripts for paste, cut and undo

3a96c9d

burmancomp added 2 commits September 27, 2023 07:03

some modifications

6aa015e

revert using api.getFocusObject

8bc8c01

burmancomp requested review from LeonarddeR and lukaszgo1 September 27, 2023 08:30

Merge branch 'master' into fix_3276

94be5c7

burmancomp added 4 commits October 5, 2023 09:45

control+y, control+backspace, alt+backspace and minor modifications

3f24e56

Merge branch 'master' into fix_3276

96e814f

Merge branch 'master' into fix_3276

542e1b1

minor modifications

40b6e8a

retry

ca9c4dd

burmancomp changed the title ~~Update braille when control+v, control+x, control+z or backspace is pressed and UIA not used in ms word~~ Update braille when control+v, control+x, control+y, control+z, backspace or control+backspace is pressed and UIA not used in ms word Oct 7, 2023

burmancomp marked this pull request as ready for review October 7, 2023 06:40

updated changes entry

357e88d

LeonarddeR reviewed Oct 7, 2023

View reviewed changes

source/NVDAObjects/IAccessible/winword.py Outdated Show resolved Hide resolved

burmancomp added 3 commits October 7, 2023 14:43

suggested change

1af7bfe

minor modification of changes entry

31afc49

code style/docstring/changes entry modifications

7cdefd1

seanbudd reviewed Oct 9, 2023

View reviewed changes

source/NVDAObjects/IAccessible/winword.py Outdated Show resolved Hide resolved

user_docs/en/changes.t2t Outdated Show resolved Hide resolved

burmancomp added 2 commits October 9, 2023 06:51

minor changes

31f2703

Merge branch 'master' into fix_3276

d3c7cf4

lukaszgo1 reviewed Oct 9, 2023

View reviewed changes

source/NVDAObjects/IAccessible/winword.py Outdated Show resolved Hide resolved

Merge branch 'master' into fix_3276

c1ba916

seanbudd marked this pull request as draft October 12, 2023 02:27

seanbudd added the conceptApproved Similar 'triaged' for issues, PR accepted in theory, implementation needs review. label Oct 16, 2023

burmancomp added 3 commits October 17, 2023 22:10

Merge branch 'master' into fix_3276

be7ca0f

suggested code refactoring

c56fed9

Merge branch 'master' into fix_3276

269120c

burmancomp marked this pull request as ready for review October 18, 2023 09:20

lukaszgo1 approved these changes Oct 18, 2023

View reviewed changes

seanbudd approved these changes Oct 18, 2023

View reviewed changes

seanbudd merged commit 7e9f9c7 into nvaccess:master Oct 18, 2023

burmancomp deleted the fix_3276 branch October 19, 2023 06:58

		@@ -47,12 +47,6 @@ def _get_ignoreEditorRevisions(self):
		ignoreFormatting=False

		def event_caret(self):

Uh oh!

Update braille when control+v, control+x, control+y, control+z, backspace or control+backspace is pressed and UIA not used in ms word #15491

Update braille when control+v, control+x, control+y, control+z, backspace or control+backspace is pressed and UIA not used in ms word #15491

Uh oh!

Conversation

burmancomp commented Sep 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Link to issue number:

Summary of the issue:

Description of user facing changes

Description of development approach

Testing strategy:

Known issues with pull request:

Code Review Checklist:

Uh oh!

burmancomp commented Sep 21, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CyrilleB79 commented Sep 21, 2023

Uh oh!

CyrilleB79 commented Sep 21, 2023

Uh oh!

LeonarddeR commented Sep 21, 2023

Uh oh!

CyrilleB79 commented Sep 21, 2023

Uh oh!

LeonarddeR commented Sep 21, 2023

Uh oh!

burmancomp commented Sep 21, 2023

Uh oh!

seanbudd commented Sep 21, 2023

Uh oh!

burmancomp commented Sep 26, 2023

Uh oh!

LeonarddeR commented Sep 26, 2023

Uh oh!

burmancomp commented Sep 26, 2023

Uh oh!

Uh oh!

Uh oh!

AppVeyorBot commented Sep 26, 2023

Uh oh!

AppVeyorBot commented Sep 26, 2023

Uh oh!

AppVeyorBot commented Oct 6, 2023

Uh oh!

burmancomp commented Oct 7, 2023

Uh oh!

Uh oh!

seanbudd commented Oct 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lukaszgo1 left a comment

Choose a reason for hiding this comment

Uh oh!

seanbudd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

burmancomp commented Sep 21, 2023 •

edited

Loading

seanbudd commented Oct 9, 2023 •

edited

Loading