Refactor DiffMatchPatch diff handler

[Danstiv]

Link to issue number:

Follow up to #16027
Fixes #15850

Summary of the issue:

Diff Match Patch proxy crashes and the calling thread deadlocks

Description of user facing changes

Diff Match Patch proxy will become more stable

Description of development approach

Refactored DiffMatchPatch diff handler.

Now, when reading from stdout of a proxy process, if not enough bytes are read, the return code is checked.

If a return code was received, an exception is raised and a fallback to difflib occurs.

Testing strategy:

Changes were made to the dmp code that caused it to crash when receiving certain characters.
Next, a test sequence of characters was typed in the terminal, which caused dmp to crash.
Exception messages in dmp and fallback to difflib were logged in the NVDA log.
An attempt to restart NVDA was successful, nvda restarted without delay, and the exit sound was played. This did not happen with deadlocked dmp diff handler.

Known issues with pull request:

None yet

Code Review Checklist:

• Documentation:
  • Change log entry
  • User Documentation
  • Developer / Technical Documentation
  • Context sensitive help for GUI changes
• Testing:
  • Unit tests
  • System (end to end) tests
  • Manual testing
• UX of all users considered:
  • Speech
  • Braille
  • Low Vision
  • Different web browsers
  • Localization in other languages / culture than English
• API is compatible with existing add-ons.
• Security precautions taken.

[Danstiv]

besides this, some changes and fixes were made to the fast_diff_match_patch (thanks to @beqabeqa473).

The fast_diff_match_patch currently can process emoji, and also does not crash when processing large amounts of text.

Now all known issues have been fixed and I think nvda could switch back to the fast_diff_match_patch

I also refactored the nvda dmp application which could potentially improve performance, @codeofdusk could you review it please?

[seanbudd]

Do you know if this effects #15841 #15850 or #15786

[Danstiv]

Do you know if this effects #15841 #15850 or #15786

#15850 only

[seanbudd]

Could you please add a change log entry to changes.md, under bug fixes? Please reference the two issues addressed

[Danstiv]

@seanbudd, @gerald-hartig
What are we going to do with diff match patch proxy?
@codeofdusk hasn't answered me since february, and the changes, in my opinion, are quite important.
Firstly, @beqabeqa473 made changes to dmp-cpp-stl, which were accepted in the fast diff match patch repository.
These changes solve the problem of fast diff match patch crashing when calculating large diffs.
The problem with unicode characters has also been resolved, both on the fast diff match patch side and on the pr side in the diff match patch proxy (bytes used instead of strings).
Secondly, according to this comment, the performance of pure python diff match patch library is 4 to 10 times lower, which probably affects users who actively use terminal applications on Windows 10.

[codeofdusk]

Thanks for your work on this!

[codeofdusk]

LGTM!

[Danstiv]

@codeofdusk
Thanks for merge!

Can I update submodule and requirements.txt in this pr?

[codeofdusk]

See #16508.

[LeonarddeR]

On a linux machine, I'm getting several errors like this after a docker compose pull:

Exception in DMP, falling back to difflib
Traceback (most recent call last):
File "diffHandler.pyc", line 104, in diff
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb9 in position 2: invalid start byte

[Danstiv]

I was able to reproduce.
It seems that this is another problem with unicode in the fast diff match patch.
Now I'll try to investigate.

[LeonarddeR]

I'd consider usimg utf-32 for data transferrather than utf8.

[Danstiv]

This can probably be fixed by a partial revert of dmp changes.
When strings instead of byte sequences used, the problem with docker compose pull is not reproduced.
@codeofdusk, what do you think?

[Danstiv]

The problem described here does not occur with the new fast diff match patch, even when passing strings instead of bytes.

[Danstiv]

I created a pr just in case, I will make additional changes if necessary.