ASSETS 2026 poster

See What I Mean: A Technology Probe of AI Assistants for Chart Exploration with Low Vision

Yotam Sechayk1, Lucy Jiang2, Yuhang Zhao3, Takeo Igarashi1

1The University of Tokyo, 2University of Washington, 3University of Wisconsin–Madison

Problem and research questions

Problem. Magnification enlarges details but fragments the chart and loses the overview. AI chart assistants mostly serve screen reader users, yet low-vision individuals prefer using their residual vision.

RQ1: How do low-vision individuals experience AI-supported chart interaction grounded in their vision?

Answer. Sharing their view helped under both personas: it complemented vision, reduced effort, and kept independence.

RQ2: Should AI support systems for low-vision individuals adapt their response content?

Answer. Adapted answers mattered: 8 of 12 participants preferred the AI that adapts its answers to low vision, and 9 of 12 followed its directions.

During exploration: ask, follow, point

Three moments from one participant (P12) exploring a world map with the low-vision AI.

  1. Ask

    Magnified view of a world map colored by gender ratio, showing North and Central America and the Caribbean. Europe is outside the view.
    • Participant: Which countries have the highest female population?
    • AI: “…Outside your view to the right: Russia, Ukraine, Belarus, and Portugal.”

  2. Follow guidance

    The participant panned right across the Atlantic. Canada and Greenland are on the left; at the right edge, about halfway down, Portugal appears in dark orange.
    • Participant: Can you tell me how to find Portugal on this map?
    • AI: “Portugal is visible on the right edge of your current view, about halfway down.”

  3. Point to confirm

    Magnified view of the Iberian Peninsula. A yellow circle marks the point the participant double-clicked, on Portugal, which is colored dark orange.
    • Participant: [double-clicks the country]
    • AI: “You are pointing to Portugal. It is colored dark orange…”

How the probe works: AI shares your view and interest point

Approach. The AI assistant sees what you see: your magnified view and point of interest.

Diagram of the ChartAEye probe in three parts. Left: a magnified bar chart of monthly government budget surplus and deficit, with the pointer on a green surplus bar, next to a chat panel. The chat shows a [new point] message, the AI reply “This is the surplus in July 2025, with value about 20B”, and the question “What is the second highest surplus?”. Middle: the AI input context, the full chart with a red box marking the user's view and a yellow dot at the pointed bar, plus a crop of the user's view. Two tags, Spatial and Semantic guidance, lead to the low-vision AI answer. Right: the two AI answers, given in full below.

Both personas receive the same input. Their answers to “What is the second highest surplus?”:

General AI (GAI): “The second highest surplus is 205B, in October 2025.”

Low-vision AI (LVAI): “The second highest surplus is 205B, in October 2025. It is outside your view, to the right, about one viewport width. The green bar you are pointing is the smallest surplus, 20B, and the highest, 257B, is visible at the left edge of your view.”

Technology probe study

12 low-vision participants, within-subjects design. Session order:

  1. Training
  2. Persona A: chart 1
  3. NASA-TLX and custom Likert questions
  4. Persona B: chart 2
  5. NASA-TLX and custom Likert questions
  6. SUS questionnaire and interview

RQ1: Seeing my view helped

Complement vision

“…when I asked, it just kind of confirmed that I didn’t have to go out of my way to go down and look and piece apart that information. It just kind of laid it out.”

Participant P5

Reduce effort

“usually I have to go scan a chart in, and then manipulate it, and then start looking up the information I’m questioning. This is more of a one-stop shop.”

Participant P3

Agency and independence

“It’s just giving me more agency because it’s not something I can necessarily see.”

Participant P11

“…there’s only so many questions you can ask someone before they ask, are you sure you know what you’re doing? … So it’s really important to be able to navigate this data independently.”

Participant P9

RQ2: Adapted answers mattered

Share of AI answers that include each type of guidance
GuidanceLow-vision AI (LVAI)General AI (GAI)
Spatial guidance71.2%7.8%
Semantic guidance84.8%29.1%

“much more specific about tying [the visuals] back to numbers or magnitudes … better suited to [their] personal needs.”

Participant P2

“…as my vision gets narrower… it’s gonna be harder for me to locate [countries]… with this technology, I could click on it, it could tell me, and I could keep moving.”

Participant P12

“describe some of the things on this map that appear very small… what in this chart is potentially visibly difficult to notice.”

Participant P11

Limitation

5 of 12 found the LVAI wordier, and it did not align with their in-the-moment needs.

“Sometimes I want an easy answer, value, name.”

Participant P1

Design recommendations: adapt both sides, what the AI sees and what it says

  1. DR1: Anchor answers on the chartPin, highlight, and annotate in the user’s view.
  2. DR2: Design for verifiabilityShow what the AI refers to, so users can check it.
  3. DR3: Scale answers to the actionQuick answers for clicks, detail for typed questions.
  4. DR4: Augment vision, keep agencyKeep every exchange user-initiated; the user’s view leads.