Exploration · Status: Ongoing
Live graphics: can I talk and have the visuals appear beside me?
Role: the presenter, the tester, and the person who decides what ships
AI used: Claude (Claude Code to build, Claude Haiku 4.5 inside the tool), Codex
Dates: 6 October 2026.
A still from the tool’s test page. A drawn silhouette stands in for me. The chart uses Mirror’s real discovery data: 14 replies, 28 to 29 September 2026.
The question
When I present Mirror on video, slides are the slow part. I wanted to know if I could just talk: say “here’s what people told us”, point at an empty spot, and have the chart appear there without covering my face.
Checked first
Others got there before me. RealityTalk (UIST 2022, with Adobe Research) shows graphics on live video when you say a keyword, and you move them by hand. Tableau Gestures puts charts over your webcam. Prezi Video has gestures too. In all three, the visuals are prepared in advance. Nobody I found let an AI write the content from free speech while you draw where it goes, so that is the part I tried.
What I built, in rounds
- Pointing. The camera tracks my hand and face on the Mac. A key drops a card at my fingertip, never on my face. I asked for a mirrored view, because pointing the other way felt wrong.
- Voice without a keyboard. Pinch and hold to talk, release to send. Speech is transcribed on the Mac by Apple’s own model. Nothing leaves the laptop at this step.
- Draw where it goes. While pinching, I trace a rectangle and the visual fills it. If the area is too small, it grows to what the content needs. A chart is never cropped.
- Real data only. Charts read from a data file. The AI can pick a chart but cannot write a number.
- Live mode. No pinch at all: after each sentence, the AI decides if a visual would help. Most of the time it says no, on purpose.
- Cleaning up. Visuals fade after 10 seconds, “keep that” pins one, “remove the last one” or a fast swipe removes one.
- Haiku, then Codex. Claude Haiku 4.5 makes the decisions. I also ran Codex in its place to compare.
Measured, not estimated
| Measure | Result |
|---|---|
| Speech to text, on the Mac | under 150 ms after I stop talking |
| Decision by Claude Haiku 4.5 | about 1 second |
| Decision by Codex, same prompt | 4.5 to 5.8 seconds |
| Cost per decision, Haiku with prompt caching | about $0.0013 |
| A generated picture of a mirror | 8 s (Gemini), 49 s (Codex) |
| Automated tests | 213 passing |
Before the Codex run, Claude estimated 10 to 30 seconds per decision. It took about 5. That is why the table only has measured numbers.
How I tested it
I rehearsed the Mirror pitch on camera six times and read the logs after each take. Every take found something the code could not:
- My pauses split sentences into fragments, so “there’s a chart on” and “how accurate the labels are” arrived separately. The AI now reads the last few seconds together.
- The laptop microphone from a distance returned mostly empty text. Earbuds fixed it.
- Pinching cut my first and last words. The tool now keeps listening a little before and after the pinch.
- In a small drawn area, a chart lost its values. The tool now sizes each visual to its content.
- When I pinched on a plain sentence, the AI declined. A pinch now always means “show something”.
Who did what
Claude wrote the plan, the code and the tests, and ran parallel agents against tests written before the code. I set the goal, ran every take, and made the calls: the mirrored view, earbuds, drawing areas, no cropped charts, fading visuals, swipe to remove, and Haiku over Codex.
What I learned
- Live in timing is not live in form. The AI understood what I asked for, but it can only show the six kinds of visual the stage knows how to draw. Asked for a red pie chart, it said exactly why it could not. TV graphics work the same way: templates filled live.
- A real take beats a test. Each rehearsal found a problem the automated tests had missed.
- Measure before deciding. Claude’s estimates of delay and cost were both off. The logs settled every choice.
- A fixed menu first. It kept every visual readable and on time. Letting the AI draw anything would cost seconds and reliability.
Next
Let Codex prepare custom visuals from a script before the talk, while Claude Haiku plays them live in about a second. Then test image generation with a faster provider.