Skip to content

Commit 012cfa0

Browse files
authored
text_view: Expose the rendered body to the accessibility tree (#4056)
## Description Fixes #3437. A `TextView` had no accessibility role, so on macOS the native tree showed the controls inside and around it but none of its text. A screen reader could not read a chat transcript or any other rendered Markdown or HTML. `TextView` is now an AccessKit `Document` whose synthetic `TextRun` children carry the rendered text. AccessKit gives a client a node's text only through such runs (`supports_text_ranges`), which then answer `AXValue`, `AXNumberOfCharacters` and the text-range calls on macOS, the Text interface on AT-SPI and the Text pattern on UIA. The runs reuse the Input's `A11yTextRuns`, so no separate parser is involved: - The text is `TextViewState::rendered_text`, the same string plain copy produces, so it follows `set_text` and `push_str` as their parses land. - Runs are captured in prepaint and only while `window.is_a11y_active()`, so views without an assistive client do no extra work. The state keeps them until the revision, select-all or scale changes, so a long transcript is not split into characters again on every frame. - `A11yTextRuns` takes an optional selection now. A read-only view with nothing selected reports no caret. A select-all is reported over the whole text. The Input still passes its selection as before. - Controls inside the view (code block actions, images with labels) stay in the tree. AccessKit excludes non-run children from the document's text. This changes `gpui-base`, because the issue is in `crates/base/src/text/text_view.rs`. ### Not covered yet - Paragraph, heading, link, list and code semantics. The body is exposed as plain text, one run per line. - Partial selections. The view has no mapping from its selection to rendered-text offsets yet, so it reports none rather than a wrong one. - Character geometry. Runs carry no bounds, so hit testing a point in the text lands on the document. Prepared with AI assistance. I reviewed the generated code. ## How to Test ```sh cargo test -p gpui-base --lib -- accessibility_text_follows a11y_text cargo test -p gpui-kit --features test-support,gpui-fast --test integration -- a11y_text ``` - `accessibility_text_follows_the_rendered_text` checks that the runs hold the rendered text (not the source), follow `push_str` and `set_text`, and report select-all, and that unchanged text reuses its runs. - `text_view_body_is_in_the_accessibility_tree` renders a selectable `TextView` beside a label in a window and checks the tree. It is `gpui-fast` only, like the existing test there, because only that backend's test window builds a tree. It fails when the role is removed. - All `gpui-base` `input::` tests (488) pass after the `A11yTextRuns` change. - `cargo clippy -p gpui-base -p gpui-kit -- --deny warnings` passes. `--all-targets` reports a redundant clone in `text/inline.rs` tests, which is also on `main`. Not yet verified with VoiceOver in a native window. ## Checklist - [x] I have read the [CONTRIBUTING](../CONTRIBUTING.md) document and followed the guidelines. - [x] Reviewed the changes in this PR and confirmed AI generated code (If any) is accurate. - [ ] Passed `cargo run` for story tests related to the changes. - [ ] Tested macOS, Windows and Linux platforms performance (if the change is platform-specific)
1 parent 8515630 commit 012cfa0

5 files changed

Lines changed: 217 additions & 30 deletions

File tree

‎crates/base/src/input/base/a11y_text.rs‎

Lines changed: 60 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
//! An input's text as AccessKit text runs.
1+
//! An input's or a text view's text as AccessKit text runs.
22
//!
33
//! AccessKit gives an assistive client a field's text, caret and selection only when the
44
//! field's node has `Role::TextRun` children (`accesskit_consumer` `supports_text_ranges`):
@@ -40,7 +40,7 @@ impl<M: InputModeKind> InputBaseState<M> {
4040
&& layout.visible_line_byte_offsets.first() == Some(&0)
4141
&& layout.lines[0].len() == text.len()
4242
});
43-
A11yTextRuns::capture(&text, selection, window.scale_factor(), |bytes| {
43+
A11yTextRuns::capture(&text, Some(selection), window.scale_factor(), |bytes| {
4444
has_layout.then(|| self.range_to_bounds(&bytes)).flatten()
4545
})
4646
}
@@ -66,8 +66,9 @@ struct Run {
6666
/// The field's text runs and its selection, as character positions in those runs.
6767
pub(crate) struct A11yTextRuns {
6868
runs: Vec<Run>,
69-
anchor: usize,
70-
focus: usize,
69+
/// The (anchor, focus) character indices, or None where nothing is selected and there
70+
/// is no caret, as in read-only text.
71+
selection: Option<(usize, usize)>,
7172
}
7273

7374
/// One grapheme of the field: its bytes in the text and, when known, its laid-out bounds.
@@ -78,11 +79,11 @@ pub(crate) struct Grapheme<'a> {
7879
}
7980

8081
impl A11yTextRuns {
81-
/// `selection` is the active selection as byte offsets (anchor, focus); `bounds_of` gives a
82-
/// byte range's laid-out bounds in window coordinates, or None.
82+
/// `selection` is the active selection as byte offsets (anchor, focus), or None; `bounds_of`
83+
/// gives a byte range's laid-out bounds in window coordinates, or None.
8384
pub(crate) fn capture(
8485
text: &str,
85-
selection: (usize, usize),
86+
selection: Option<(usize, usize)>,
8687
scale: f32,
8788
bounds_of: impl Fn(std::ops::Range<usize>) -> Option<Bounds<Pixels>>,
8889
) -> Option<Self> {
@@ -118,19 +119,36 @@ impl A11yTextRuns {
118119
}
119120
Some(Self {
120121
runs,
121-
anchor: character_index(text, selection.0),
122-
focus: character_index(text, selection.1),
122+
selection: selection.map(|(anchor, focus)| {
123+
(character_index(text, anchor), character_index(text, focus))
124+
}),
123125
})
124126
}
125127

126-
pub(crate) fn push(self, builder: &mut gpui::A11ySubtreeBuilder) {
128+
/// The whole text, as the runs hold it.
129+
#[cfg(test)]
130+
pub(crate) fn text(&self) -> String {
131+
self.runs.iter().map(|run| run.text.as_str()).collect()
132+
}
133+
134+
/// The (anchor, focus) character indices of the selection, if any.
135+
#[cfg(test)]
136+
pub(crate) fn selection(&self) -> Option<(usize, usize)> {
137+
self.selection
138+
}
139+
140+
pub(crate) fn push(&self, builder: &mut gpui::A11ySubtreeBuilder) {
127141
let ids: Vec<_> = (0..self.runs.len())
128142
.map(|index| builder.synthetic_node_id(("text-run", index)))
129143
.collect();
130-
let anchor = position(&self.runs, &ids, self.anchor);
131-
let focus = position(&self.runs, &ids, self.focus);
144+
let selection = self
145+
.selection
146+
.map(|(anchor, focus)| accesskit::TextSelection {
147+
anchor: position(&self.runs, &ids, anchor),
148+
focus: position(&self.runs, &ids, focus),
149+
});
132150
let mut previous_continues = false;
133-
for (index, run) in self.runs.into_iter().enumerate() {
151+
for (index, run) in self.runs.iter().enumerate() {
134152
let id = ids[index];
135153
let mut node = accesskit::Node::new(accesskit::Role::TextRun);
136154
if run.continues {
@@ -140,21 +158,21 @@ impl A11yTextRuns {
140158
node.set_previous_on_line(ids[index - 1]);
141159
}
142160
previous_continues = run.continues;
143-
node.set_value(run.text);
144-
node.set_character_lengths(run.character_lengths);
145-
node.set_word_starts(run.word_starts);
146-
if let Some((positions, widths)) = run.geometry {
147-
node.set_character_positions(positions);
148-
node.set_character_widths(widths);
161+
node.set_value(run.text.as_str());
162+
node.set_character_lengths(run.character_lengths.as_slice());
163+
node.set_word_starts(run.word_starts.as_slice());
164+
if let Some((positions, widths)) = &run.geometry {
165+
node.set_character_positions(positions.as_slice());
166+
node.set_character_widths(widths.as_slice());
149167
}
150168
if let Some(bounds) = run.bounds {
151169
node.set_bounds(bounds);
152170
}
153171
builder.push_child(id, node);
154172
}
155-
builder
156-
.parent_node()
157-
.set_text_selection(accesskit::TextSelection { anchor, focus });
173+
if let Some(selection) = selection {
174+
builder.parent_node().set_text_selection(selection);
175+
}
158176
}
159177
}
160178

@@ -289,7 +307,15 @@ mod tests {
289307
use super::*;
290308

291309
fn capture(text: &str, selection: (usize, usize)) -> A11yTextRuns {
292-
A11yTextRuns::capture(text, selection, 1.0, |_| None).unwrap()
310+
A11yTextRuns::capture(text, Some(selection), 1.0, |_| None).unwrap()
311+
}
312+
313+
fn anchor(runs: &A11yTextRuns) -> usize {
314+
runs.selection.unwrap().0
315+
}
316+
317+
fn focus(runs: &A11yTextRuns) -> usize {
318+
runs.selection.unwrap().1
293319
}
294320

295321
fn texts(runs: &A11yTextRuns) -> Vec<&str> {
@@ -306,7 +332,7 @@ mod tests {
306332
fn a_single_line_is_one_run() {
307333
let runs = capture("abc", (1, 1));
308334
assert_eq!(texts(&runs), ["abc"]);
309-
assert_eq!(at(&runs, runs.focus), (0, 1));
335+
assert_eq!(at(&runs, focus(&runs)), (0, 1));
310336
assert_eq!(at(&runs, 3), (0, 3));
311337
}
312338

@@ -333,16 +359,16 @@ mod tests {
333359
fn a_trailing_break_gets_an_empty_last_line() {
334360
let runs = capture("ab\n", (3, 3));
335361
assert_eq!(texts(&runs), ["ab\n", ""]);
336-
assert_eq!(at(&runs, runs.focus), (1, 0));
362+
assert_eq!(at(&runs, focus(&runs)), (1, 0));
337363
}
338364

339365
#[test]
340366
fn a_selection_across_lines_maps_both_ends() {
341367
let text = "one\ntwo";
342368
// Reversed: anchor after "tw", focus after "o".
343369
let runs = capture(text, (6, 1));
344-
assert_eq!(at(&runs, runs.anchor), (1, 2));
345-
assert_eq!(at(&runs, runs.focus), (0, 1));
370+
assert_eq!(at(&runs, anchor(&runs)), (1, 2));
371+
assert_eq!(at(&runs, focus(&runs)), (0, 1));
346372
}
347373

348374
#[test]
@@ -359,6 +385,13 @@ mod tests {
359385
assert_eq!(at(&runs, RUN_CHARACTERS + 4), (2, 0));
360386
}
361387

388+
#[test]
389+
fn text_without_a_selection_has_no_caret() {
390+
let runs = A11yTextRuns::capture("ab\ncd", None, 1.0, |_| None).unwrap();
391+
assert_eq!(texts(&runs), ["ab\n", "cd"]);
392+
assert!(runs.selection.is_none());
393+
}
394+
362395
#[test]
363396
fn characters_are_grapheme_clusters() {
364397
let runs = capture("a🎉\nğ", (0, 0));

‎crates/base/src/input/mod.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ pub use token_presentation::{
4444
InlineTokenHoverListener, InlineTokenRenderer,
4545
};
4646
#[path = "base/a11y_text.rs"]
47-
mod a11y_text;
47+
pub(crate) mod a11y_text;
4848
#[path = "base/kind.rs"]
4949
mod kind;
5050
#[path = "editor/language.rs"]

‎crates/base/src/text/state.rs‎

Lines changed: 63 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ use std::time::Instant;
44
use std::{
55
ops::{Range, RangeInclusive},
66
pin::Pin,
7+
rc::Rc,
78
sync::{Arc, Mutex, OnceLock},
89
task::Poll,
910
time::Duration,
@@ -20,7 +21,7 @@ use gpui::{
2021
use crate::{
2122
AutoScroll, ElementExt, TextSelection,
2223
async_util::{Receiver, Sender, unbounded},
23-
input::{self, SelectAll},
24+
input::{self, SelectAll, a11y_text::A11yTextRuns},
2425
text::{
2526
CodeBlockActionsFn, CodeBlockHighlighterFn, LinkClickHandlerFn, LinkFn, MarkdownExtensions,
2627
RangeHighlight, RangeHighlightError, RenderedText, TableActionsFn, TaskToggleHandlerFn,
@@ -151,6 +152,9 @@ pub struct TextViewState {
151152
multi_click_selection: Option<TextViewMultiClickSelection>,
152153
selected_text_override: Option<String>,
153154
select_all: bool,
155+
/// The last text runs built for the accessibility tree, with the
156+
/// (revision, select-all, scale) they were built for.
157+
a11y_text_runs: Option<((usize, bool, f32), Rc<A11yTextRuns>)>,
154158
pub(super) auto_scroll: AutoScroll,
155159
pub(super) selection_adapter: TextViewSelectionAdapter,
156160

@@ -232,6 +236,7 @@ impl TextViewState {
232236
multi_click_selection: None,
233237
selected_text_override: None,
234238
select_all: false,
239+
a11y_text_runs: None,
235240
selectable: false,
236241
hover_cursor: None,
237242
selection_format: SelectionFormat::default(),
@@ -675,6 +680,30 @@ impl TextViewState {
675680
)
676681
}
677682

683+
/// The rendered text as AccessKit text runs, so a screen reader can read the body.
684+
///
685+
/// The runs follow [`Self::rendered_text`], which tracks the document as its parses land,
686+
/// so text set or streamed in reaches the tree with the frame that shows it. A select-all
687+
/// is reported as the selection; any other selection is not mapped yet, so the view
688+
/// reports none rather than a wrong one.
689+
///
690+
/// The runs are kept until the text, the select-all or the scale changes, so a long
691+
/// document is not split into characters again on every frame.
692+
pub(super) fn accessibility_text_runs(&mut self, scale: f32) -> Option<Rc<A11yTextRuns>> {
693+
let key = (self.committed_revision, self.select_all, scale);
694+
if let Some((cached, runs)) = &self.a11y_text_runs
695+
&& *cached == key
696+
{
697+
return Some(runs.clone());
698+
}
699+
let text = self.rendered_text();
700+
let text = text.as_str();
701+
let selection = self.select_all.then_some((0, text.len()));
702+
let runs = Rc::new(A11yTextRuns::capture(text, selection, scale, |_| None)?);
703+
self.a11y_text_runs = Some((key, runs.clone()));
704+
Some(runs)
705+
}
706+
678707
/// Replace the range highlights, whose ranges index the current rendered text.
679708
///
680709
/// Compute ranges from the current [`Self::rendered_text`] and set them in
@@ -2327,6 +2356,39 @@ mod tests {
23272356
});
23282357
}
23292358

2359+
/// A screen reader reads the rendered body, not the Markdown source, and
2360+
/// reads text set or streamed in once its parse lands.
2361+
#[gpui::test]
2362+
fn accessibility_text_follows_the_rendered_text(cx: &mut TestAppContext) {
2363+
cx.update(crate::init);
2364+
let state = cx.update(|cx| cx.new(|cx| TextViewState::markdown("Hello **world**", cx)));
2365+
cx.run_until_parked();
2366+
let runs = |state: &Entity<TextViewState>, cx: &mut TestAppContext| {
2367+
state.update(cx, |state, _| state.accessibility_text_runs(1.).unwrap())
2368+
};
2369+
2370+
let first = runs(&state, cx);
2371+
assert_eq!(first.text().trim_end(), "Hello world");
2372+
assert_eq!(first.selection(), None);
2373+
// An unchanged document reuses its runs.
2374+
assert!(Rc::ptr_eq(&first, &runs(&state, cx)));
2375+
2376+
state.update(cx, |state, cx| state.push_str("\n\n- item", cx));
2377+
cx.run_until_parked();
2378+
assert_eq!(runs(&state, cx).text().trim_end(), "Hello world\nitem");
2379+
2380+
state.update(cx, |state, cx| state.set_text("Other", cx));
2381+
cx.run_until_parked();
2382+
assert_eq!(runs(&state, cx).text().trim_end(), "Other");
2383+
2384+
state.update(cx, |state, cx| state.select_all(cx));
2385+
let selected = runs(&state, cx);
2386+
assert_eq!(
2387+
selected.selection(),
2388+
Some((0, selected.text().chars().count()))
2389+
);
2390+
}
2391+
23302392
#[gpui::test]
23312393
fn select_all_in_source_format_returns_source(cx: &mut TestAppContext) {
23322394
cx.update(crate::init);

‎crates/base/src/text/text_view.rs‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ use gpui::{
99
};
1010

1111
use crate::StyledExt;
12+
use crate::input::a11y_text::A11yTextRuns;
1213
use crate::text::TextViewFormat;
1314
use crate::text::markdown_ext::{MarkdownExtensions, MarkdownNode, MarkdownPlugin};
1415
use crate::text::node::{CodeBlock, TableData};
@@ -733,6 +734,9 @@ pub struct TextViewPrepaintState {
733734
clip_bottom: Option<Pixels>,
734735
/// The touch handles this view owns, with their hitboxes.
735736
touch_handles: crate::TouchHandleLayout,
737+
/// The rendered text for the accessibility tree, captured only while an
738+
/// assistive client is connected.
739+
a11y_text_runs: Option<Rc<A11yTextRuns>>,
736740
}
737741

738742
/// Absorbs sub-pixel layout jitter: a line ending within a pixel of the box
@@ -829,6 +833,22 @@ impl Element for TextView {
829833
None
830834
}
831835

836+
/// A document whose text runs carry the rendered body, so a screen reader
837+
/// reads the text itself and not only the controls inside it.
838+
fn a11y_role(&self) -> Option<gpui::Role> {
839+
Some(gpui::Role::Document)
840+
}
841+
842+
fn a11y_synthetic_children(
843+
&mut self,
844+
prepaint: &mut Self::PrepaintState,
845+
builder: &mut gpui::A11ySubtreeBuilder,
846+
) {
847+
if let Some(runs) = &prepaint.a11y_text_runs {
848+
runs.push(builder);
849+
}
850+
}
851+
832852
fn request_layout(
833853
&mut self,
834854
_: Option<&GlobalElementId>,
@@ -1084,10 +1104,18 @@ impl Element for TextView {
10841104
} else {
10851105
crate::TouchHandleLayout::default()
10861106
};
1107+
let a11y_text_runs = window
1108+
.is_a11y_active()
1109+
.then(|| {
1110+
let scale = window.scale_factor();
1111+
state.update(cx, |state, _| state.accessibility_text_runs(scale))
1112+
})
1113+
.flatten();
10871114
TextViewPrepaintState {
10881115
hitbox,
10891116
clip_bottom,
10901117
touch_handles,
1118+
a11y_text_runs,
10911119
}
10921120
}
10931121

0 commit comments

Comments
 (0)