Add visual highlighting for web math navigation#20372
Conversation
|
Very nice surprise to see this PR! Thanks @RyanMcCleary. I have not tested yet but have already somme questions. Given this PR takes into account a location for the subpart being read in the math equation, what about the new Magnifier? Does it follow? In a foreseable future, it would be nice that magnifier can follow the subpart being read, at least in math exploration/navigation mode.If it follows, to which type of tracking is it tied? System focus (which also includes browse mode cursor when applicable)? |
|
@CyrilleB79 The magnifier does not yet follow the highlighted subparts. My thinking was that we could handle that in a follow-up PR. As for the type of tracking, system focus seems like the logical choice. I do still have a couple open questions about this PR where I'd welcome your input. The math highlighter as it's implemented here is a an entirely new highlighter with its own highlighting color (orange). Do you think it should be included in the browse mode highlighter? I can see arguments both for and against this. On the one hand, including it in the browse mode highlighter is simpler and doesn't require adding a new highlighter. On the other hand, math interaction is separate from browse mode, and although it's usually activated from browse mode now, there might be other ways to activate it in the future. Thoughts? |
|
Yes, Magnifier tracking can be implemented in a follow-up PR. It's even preferable to keep PRs not too big to ease review. I have just mentioned it so that you have it in mind since both features may be related. My opinion is that browse mode cursor highlighter should be used in this case; the same color should be kept. I agree that this is discutable though.
|
|
Note, I've tested this PR and highlighting is working well. Thanks! |
|
Interesting point re: navigating OCR results. Given that there's already a precedent for this, it does seem like a good idea to use browse mode highlighting. I'll update the PR with that change. |
seanbudd
added
the
conceptApproved
There was a problem hiding this comment.
Pull request overview
This PR adds visual highlighting support for web MathML navigation by integrating MathCAT’s current navigation node with NVDA’s existing browse mode highlighter pipeline, improving low-vision usability while exploring math subexpressions.
Changes:
- Add a new vision extension point (
post_mathNavigation) and wire it intoNVDAHighlighterto drive browse mode highlight rectangles from math navigation. - Enhance MathCAT interaction to preprocess MathML with synthetic IDs and map MathCAT’s current nav node to IA2 object rectangles (web), falling back to the source object rectangle when needed.
- Update user documentation and changelog, and add unit tests for MathCAT nav node mapping helpers.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| user_docs/en/userGuide.md | Documents that Visual Highlight (browse mode cursor highlighting) now also highlights the current web math subpart during math navigation. |
| user_docs/en/changes.md | Adds changelog entry for math navigation highlighting on the web. |
| tests/unit/test_mathPres/test_mathCatNavNodeMapping.py | Adds unit tests for MathCAT nav node mapping helpers (namespace stripping, synthetic-id removal). |
| tests/unit/test_mathPres/init.py | Introduces a test package for math presentation unit tests. |
| source/visionEnhancementProviders/NVDAHighlighter.py | Registers and handles math navigation events, updating the browse mode highlight rect accordingly. |
| source/vision/visionHandlerExtensionPoints.py | Adds post_mathNavigation extension point to notify vision enhancement providers of math navigation rectangle changes. |
| source/vision/visionHandler.py | Adds handleMathNavigation to dispatch math navigation rectangle updates via extension points. |
| source/NVDAObjects/IAccessible/ia2Web.py | Adds traversal to build a path→(tag, rect) map for IA2 web MathML subtrees. |
| source/mathPres/mathMlNode.py | Adds shared types for identifying MathML nodes and their rectangles. |
| source/mathPres/MathCAT/navNodeMapping.py | New helper module for adding synthetic MathML IDs and mapping MathCAT nav node IDs to IA2 rectangles. |
| source/mathPres/MathCAT/MathCAT.py | Passes a source object into MathCAT interaction, updates highlight on navigation, and cleans synthetic IDs before clipboard export. |
| source/mathPres/init.py | Extends math interaction APIs to accept an optional sourceObj and clears math highlight on exiting interaction. |
| source/globalCommands.py | Passes a source object into math interaction when available (navigator object or object at review position). |
| source/browseMode.py | Passes the math object as sourceObj when entering math interaction from browse mode. |
| mathElementChildren = tuple(child for child in element if isinstance(child.tag, str)) | ||
| for index, child in enumerate(mathElementChildren): | ||
| yield from _iterMathMlElements(child, nodePath + (index,)) |
| def removeSyntheticIdsFromMathMl(mathml: str) -> str: | ||
| try: | ||
| root = ElementTree.fromstring(mathml) | ||
| except ElementTree.ParseError: | ||
| return mathml | ||
| for element, _nodePath in _iterMathMlElements(root, ()): | ||
| if element.get(_NAV_NODE_ID_ADDED_ATTR) != "true": | ||
| continue | ||
| originalId = element.attrib.pop(_NAV_NODE_ORIGINAL_ID_ATTR, None) | ||
| if originalId is not None: | ||
| element.set("id", originalId) | ||
| else: | ||
| element.attrib.pop("id", None) | ||
| element.attrib.pop(_NAV_NODE_ID_ADDED_ATTR, None) | ||
| return ElementTree.tostring(root, encoding="unicode") |
seanbudd
left a comment
seanbudd
left a comment
There was a problem hiding this comment.
Note: partial review.
Please review the changes and make sure to:
- move out imports if possible, if not leave a comment as to why they are required (E.g. to prevent circular imports)
- explain why some code seems to be arbitrarily wrapped in try/excepts with accompanying comments, or remove the try/excepts
- make sure to use sphinx style docs rather than epydoc on any changed/added code
| sourceObj = self.sourceObj | ||
| if not sourceObj: | ||
| return None | ||
| from NVDAObjects.IAccessible.ia2Web import Math as Ia2WebMath |
There was a problem hiding this comment.
can these imports be moved out?
There was a problem hiding this comment.
The other imports should definitely be moved out, but it seems to me like it would be a good idea to keep this import local. I've added a comment with my reasoning for that. I'm still happy to move it out if you think that's best though.
Co-authored-by: Sean Budd <seanbudd123@gmail.com>
Co-authored-by: Sean Budd <seanbudd123@gmail.com>
Co-authored-by: Sean Budd <seanbudd123@gmail.com>
Co-authored-by: Sean Budd <seanbudd123@gmail.com>
Link to issue number:
Part of #19191.
Summary of the issue:
When navigating math, there is currently no visual highlighting showing the focused subpart. Visual highlighting is very useful for low-vision users and can make it easier to use speech and magnification together while reading math.
Description of user facing changes:
When browse mode highlighting is enabled, the currently focused subpart is now highlighted while navigating math on the web. Similarly to how highlighting is handled for OCR results, math navigation highlighting is included in browse mode highlighting.
Description of developer facing changes:
The
MathCATInteractionclass now takes a source object as a constructor parameter (the source object is the root of the MathML being interacted with). A new filesource\mathPres\MathCAT\navNodeMapping.pywas also added. This file contains functions that are used to map from MathCAT's current navigation node to the corresponding IA2 object.Description of development approach:
The main tricky part here was mapping MathCAT's current navigation node to the position of that MathML element on the screen. This is done by mapping element IDs to the corresponding IA2 object, from which the position information can be retrieved. One wrinkle in this process was MathCAT creates its own internal IDs for elements that don't already have IDs, so it was necessary to preprocess the MathML to create IDs for elements that don't have them.
Testing strategy:
Unit testing and manual testing in Chrome and Firefox.
Known issues with pull request:
The only issue I know of is the possibility that there might be a more general solution. If it is possible to implement this so that it works for Microsoft Word, Outlook, Acrobat, etc. as well, then that solution might be preferred. However, it seems likely to me that those cases will need to be handled separately.
Code Review Checklist: