Introduce cppjieba as a submodule for Chinese word segmentation

[CrazySteve0605]

Introduce cppjieba, an NLP-based Chinese tokenizer, for implementing Chinese word navigation and braille output.

cppjieba is the C++ version of the popular Python library jieba, offering near-identical runtime performance.

A detialed integration analysis

1. Current State of Chinese Segmentation Tools

Chinese segmentation techniques trade accuracy for runtime and memory in predictable ways:

1. Dictionary-based (fast, lower OOV coverage). Libraries such as Jieba rely on prefix trees (tries) and greedy algorithms (maximum matching / DAG). These methods are CPU-efficient and lightweight but commonly miss new or domain-specific tokens (OOV words).
2. Statistical / sequence-labeling (better context, moderate cost). Approaches using HMMs, CRFs or perceptrons (adopted by tools like THULAC, LTP, pkuseg) label characters with B/I/E/S tags to model local context and transitions. They substantially improve OOV handling and achieve high F1 on news/text benchmarks, at the cost of higher computation and memory compared with pure dictionary methods.
3. Deep learning (highest accuracy, highest cost). Transformer/BERT-based models offer superior contextual understanding and segmentation accuracy, but their inference latency and memory footprint (often hundreds of megabytes to multiple gigabytes) make them unsuitable for real-time, low-latency assistive software like NVDA.

2. NVDA-Specific Requirements

NVDA requires low latency, a small memory footprint, and reliable offline operation — which rules out heavyweight deep-learning models. At the same time, very lightweight pure dictionary- or rule-based segmenters are also unsuitable: their lower accuracy and poor OOV handling lead to frequent mis-segmentation, inaccurate speech output, and increased user confusion in real-world documents. cppjieba therefore represents a pragmatic middle ground — native C++ performance and modest resource usage combined with a hybrid strategy (dictionary lookup plus HMM-based sequence labeling) that delivers appreciably better accuracy and OOV mitigation than pure dictionary methods, while remaining far lighter and faster than Transformer/BERT approaches — making it a sensible choice for improving Chinese text handling in NVDA without imposing large runtime or memory penalties.

Link to issue number:

Related to #4075 and a part of OSPP 2025 of NVDA.

Summary of the issue:

NVDA’s current word navigation mechanism relies on Unicode boundary rules through the Uniscribe API, which do not work well for languages such as Chinese due to the absence of explicit word delimiters.

Description of user facing changes:

None

Description of developer facing changes:

A tool to implement word navigation and braille output within Chinese content.

Description of development approach:

• Added cppjieba as a submodule.
• Added its wrapper and building script.

Testing strategy:

Confirm weather it can be successfully compiled and its segmentation function can be called by ctypes.

Known issues with pull request:

Code Review Checklist:

• Documentation:
  • Change log entry
  • User Documentation
  • Developer / Technical Documentation
  • Context sensitive help for GUI changes
• Testing:
  • Unit tests
  • System (end to end) tests
  • Manual testing
• UX of all users considered:
  • Speech
  • Braille
  • Low Vision
  • Different web browsers
  • Localization in other languages / culture than English
• API is compatible with existing add-ons.
• Security precautions taken.

@coderabbitai summary

[wmhn1872265132]

I downloaded the launcher built by GitHUB Actions for testing and it doesn't seem to work

[cary-rowen]

Imo, I suggest that initial development and debugging be carried out locally, especially during the stage when functionalities are not yet operational. If extensive testing by community early users is needed, at least new features should be functional.

[CrazySteve0605]

I downloaded the launcher built by GitHUB Actions for testing and it doesn't seem to work

Apologies for the insufficient explanation. This PR is intended as an initial step of the overall work. It does not implement any segmentation functionality yet, but merely introduces the ‘cppjieba’ module. Therefore, it has no impact on end users.

[CrazySteve0605]

Imo, I suggest that initial development and debugging be carried out locally, especially during the stage when functionalities are not yet operational. If extensive testing by community early users is needed, at least new features should be functional.

As many parts of the code need changes, splitting the work into smaller tasks might be more efficient, as the community can review them at the same time.

[seanbudd]

@CrazySteve0605 - can you please provide more information on why this library was selected over others? what were alternative options?

[seanbudd]

@CrazySteve0605 - could you please update the PR description with what was requested in #18548 (comment), some more information on the module chosen?

Please also make sure to mark the PR as ready for review if it as this stage.

We are hoping to see implementation of segmentation with an example TextInfo such as WordDocumentTextInfo or Gecko_ia2_TextInfo

[CrazySteve0605]

@seanbudd OK I see. I'm adding more details to make the main comment more informative. Instead of extending 'TextInfos', I'm creating a separate 'WordSegmentationStrategy' and 'WordSegmenter' that follow the strategy pattern, selecting an appropriate segmenter based on the Unicode fields present in the given text. I think this approach will be more extensible and easier to reuse in other potential components, such as braille output. I also plan to submit it as my next PR.

[michaelDCurran]

this pr is pretty much good. Just two small things in my last review.
We have decided that this and any other related PRs for this OSSP project will be merged into a special try-wordSegmentation-staging branch rather than master. So that all PRs can be reviewed and merged in to one working branch, which will then if successful, be merged to master all at the same time. Therefore I have switched the base branch of this pr to try-wordSegmentation-staging accordingly.

[michaelDCurran]

Please note, that I incorrectly had merged this as a squash commit, which caused problems for dependent PRs such as #18735.
I have now reverted that squash commit and merged this PR's branch unsquashed as commit 093a825