Speak: Building an AI Tutor Around the Learner

The first obstacle: a learner who cannot speak right now

A new learner opens Speak and meets a tutor that asks about their goals—but they are somewhere they cannot speak aloud. This is a practical constraint on Magic Onboarding, a conversation that gathers information to personalize later lessons. It demonstrates the tutoring experience immediately while asking more of a new user than tapping answers on a form. Speak provides fallback paths and continues experimenting with the interaction.

That conversational tutor was the destination long before it was feasible. In the Latent Space conversation, published in July 2025, Andrew Hsu traces Speak’s thesis to a year of AI self-study beginning in 2016. He and his co-founder expected speech and language models to reach superhuman capability within five to ten years, enabling entirely software-based language tutors. Finding a product people wanted took much longer than expected, even as the long-term vision held steady.

Hsu had joined the first Thiel Fellowship class in 2011 at 19, already having finished college and entered graduate school. He planned to leave for startups and met his future Speak co-founder through the fellowship’s second cohort. Their eventual ambition was a tutor that could help learners reach fluency faster than a human tutor could. Hsu estimated that 80–90% of the necessary technology was available—an assessment of technical readiness, not demonstrated learning superiority.

Conversational onboarding trades ease for engagement

Speak found product-market fit teaching English in South Korea roughly two to three years before Whisper and ChatGPT arrived in 2022. Its custom speech-recognition models learned from non-native English speech collected through the app. Large language models, or LLMs, later supported more open-ended tutoring and feedback about meaning, extending the product beyond its fast speaking drills.

Magic Onboarding brings that conversational capability before account creation. Hsu describes two approaches the team had tried: - State-machine control: explicit progression through the onboarding flow, probably the version an interviewer had encountered. - A goal in the prompt: a more natural conversation directed toward learning enough about the user to personalize instruction. The second approach must cope with unrelated topics and deliberately antagonistic responses. Speak uses evaluations and fallbacks, aiming for a three-to-five-minute tutor conversation before the learner creates an account.

Memory is a separate summarization step. Speak stores responses and uses another LLM prompt to extract goals and preferences. The learner sees a compact account of what matters to them rather than the full transcript.

The experiment exposed a funnel tradeoff: install-to-sign-up conversion fell, while trial-start rate rose. Hsu had no final conclusion; the experiment was still running, with no numerical lift or trial-start denominator given. The unresolved question was whether demonstrating the tutor immediately justified the extra friction.

The 2018 rebuild: give the learner a next lesson

Speak positions AI tutoring as a third generation of language software, following Rosetta Stone’s CD-ROMs and mobile apps such as Duolingo and Babbel. Its teaching priority is functional fluency: speaking spontaneously in situations that matter. Role-plays might involve a Spanish conversation with an Uber driver. Repeated sentence-pattern drills aim to make responses automatic, rather than centering instruction on explicit vocabulary and grammar rules.

The original Red App offered free content packs across multiple languages and let users choose topics. That loosely directed experience failed to find traction. In 2018, Speak rebuilt around English instruction in South Korea, new lesson types, and guided courses. Opening the app already consumed motivation; one prominent next-lesson button removed another decision between intending to practice and practicing.

Speak also moved from a free version to a premium product, partly to select learners already motivated to improve their English. The company attributes subsequent growth to several years of accumulated product lessons. From 2022 onward, improving models enabled a second phase: fuller tutoring, including direct wording feedback using models such as 3.5 Turbo to suggest more natural expressions.

By the conversation, Asian markets remained Speak’s mainstay. It was expanding in Japan and Taiwan, teaching English in 40 additional countries, and already offering Spanish and French. Speak estimated that roughly 6% of South Korea’s population had tried the app, without specifying the methodology. The companion article reports more than $50 million in annual recurring revenue. A business offering, begun about a year before the conversation as an experiment, was also growing, though no separate revenue figures were given.

South Korea tested demand for direct conversation

Why learn a language if real-time translation becomes good enough? Speak’s response rests on two distinctions: - Translation can require waiting for meaning. The speaker uses a sentence-final German verb to argue that word-order differences can force a translator to wait for a full sentence. This is an illustrative argument, not a measured latency result. - Learners want direct human connection. Speak describes Asian users seeking personal development and the ability to speak a shared language face to face. It anticipates integrating translation into learning, rather than treating translation as the whole experience.

South Korea offered a demanding test of that demand. The founders nearly chose Taiwan, but their Korean first employee—a former college roommate of the co-founder—helped explore Seoul as a translator and facilitator. English academies and extensive classroom instruction gave learners abundant alternatives. The founders’ hypothesis was that success against established human instruction would demonstrate a product they could take elsewhere.

Serving those learners from San Francisco required attention to localization down to individual button labels. Speak credits its Korean colleague as essential and recalls early users being surprised the company was American. The team believed this natural wording had a substantial effect, though it did not quantify one.

The early product had no LLM component. Teachers and an internal content team developed the Speak Method, drilling sentence patterns and combining them into more complex patterns, with custom tooling supporting production. A native, high-performance mobile app and careful polish were central priorities.

The engineering team grew through referrals: an initial iOS hire in Slovenia led to a cluster of colleagues in the same city and eventually a physical office. The first hire was chosen as the strongest candidate, rather than for cost savings. Daily synchronous meetings sustained the arrangement despite inconvenient time zones. By the conversation, roughly 90% of core product development was in San Francisco; the retrospective credited the remote engineers while acknowledging how burdensome the arrangement had been.

Whisper expanded tutoring while custom recognition kept drills fast

Whisper’s arrival in 2022 produced a memorable office test. Four people listened, eyes closed, to a clip of a beginner English learner in Korea and could not understand it; Whisper reportedly recognized it correctly. The speaker took this as a glimpse of the superhuman capability the founders had anticipated. It was a striking single comparison, not a general benchmark.

Previously, a core exercise showed text, played a teacher’s reference recording, and asked the learner to repeat it. Improving speech and language models made spontaneous responses, wording feedback, and explanations of mistakes possible. Speak retained two paths because their requirements differed: - Custom streaming automatic speech recognition (ASR): fast recognition fine-tuned for the repeated lesson loop. - Whisper with LLM feedback: recognition of spontaneous speech followed by feedback on wording and meaning. Whisper was not streaming in the setup described.

The resulting strategy was to exhaust useful model capabilities, then revisit blocked features as models improve. At the time, Speak was building a real-time voice platform for specialized lessons. More immersive conversations were expected to complement the existing curriculum, probably serving more advanced learners.

Scale curriculum production and assess uneven fluency

Manually written scripts and video lessons filmed in an LA studio created the next scaling constraint. Expansion required more than additional target languages: a language pair specifies both the learner’s native language and the language being learned. Speak’s ambitions were 100 times more content, ten times more languages, and eventually 100 times more language pairs—production goals, not achieved multipliers.

Speak described a tutor agent, a curriculum-writing agent, and an LLM pipeline that scaffolds curricula and writes individual lessons. The goal was to support more markets and languages while keeping the team small.

Personalization required a more detailed account of ability. Someone might comfortably order at a Mexico City taco stand yet struggle to discuss their family. Speak was developing a knowledge graph of learner proficiency, a structured representation of known and unknown vocabulary, sentence patterns, and clusters of recurring mistakes. That representation aims to capture what a learner can handle in the situations they care about.

Some assessment components and subscores were live; the intended Speak score would aggregate a sufficiently complete profile. A number such as 54 has little meaning alone, so the teaching goal was levels tied to things learners can do. The discussion offered no validation results or mapping to an external proficiency standard. Exam preparation remained a possible future product.

Generating courses also requires judging them. Pedagogical judgment was difficult to turn into explicit criteria: the content team already struggled to explain subtle differences between effective lesson formats when training human writers. Speak’s evaluation approach remained a combination of work in progress and human expertise: - Content-team review: people continued reviewing syllabi, curricula, and individual lesson lines. - Model-graded evaluations: models judging outputs were one component of an evaluation suite under development. - Reinforcement fine-tuning: experiments using internal data for curriculum or lesson-writing agents were still early. The hoped-for result was much greater course output from the team; a hundredfold productivity gain had not been established.

Private practice first; pronunciation and visuals where they help

Speak prioritizes everyday conversational language, stopping short of a focus on Gen Z slang. Its English curriculum taught standard American English at the time, a pragmatic choice based on team size and user demand. More differentiated accent and dialect instruction remained an ambition.

The core teaching position is that imperfect pronunciation need not prevent effective communication. Speak encourages frequent speech in a private setting where learners can make mistakes without the embarrassment they might feel even with a paid teacher. Its separate English pronunciation coach used a Meta model fine-tuned on Speak’s phonetic transcript data. It handled individual words; full sentences and additional languages were planned.

Learning could also draw on activities outside the app. The conversation considers browser extensions that modify web pages and a tool that masks words in YouTube transcripts, without resolving their effectiveness. Speak for Business uses the same underlying tutor through a different distribution channel. A hypothetical employee preparing to transfer to a US office illustrates why actual work documents might inform lessons. A Mac app or browser integration was under consideration, with implementation undecided. Hardware that listens during the day and supplies later language feedback was a more speculative possibility.

The medium must fit the practice setting, too. Teacher videos were engaging, but users also wanted to walk with a locked phone in their pocket. Speak had tested more audio-oriented formats and saw promise in voice with optional visuals. One participant’s difficulty understanding native-speed German after two years of study motivated a related proposal: controllable, AI-generated listening experiences resembling podcasts, integrated into the app.

A proposed multimodal tutor would coordinate speech, text, and images: reveal a word’s spelling as it is introduced, or generate a beach scene and ask questions about it in the target language. What appears, and when, depends on what the learner already knows. Faster image generation was a forecast underlying this vision. Generative video had received little attention because engineering capacity was focused on more accessible, high-impact work.

Real-time tutoring needs multilingual audio, viable costs, and patience

Speak had received early access to OpenAI’s real-time API the previous year and envisioned audio becoming the primary interaction across the app. Language tutoring adds bidirectional code-switching: both learner and tutor may mix languages within a sentence. A learner practicing Spanish may ask part of a question in English, and the tutor must pronounce both languages appropriately. This was described as a difficult text-to-speech capability available in only a few models at the time.

Routing each utterance to a language-specific model does not address those within-sentence switches. The response to that proposal was that concatenating separately generated pieces could sound unnatural. Speak wanted controllable audio generation that handles multiple languages natively.

No real-time API experiences were in production at recording time, mainly because of price. Customer support can justify costs against hourly labor; Speak wanted learners to practice conversations and role-plays for hours a day if they chose. A launch was described as close, possibly before episode release, but remained prospective.

The concrete design under development was a three-to-five-minute instructional lesson in which a model teaches a concept and invites interaction at selected points. It required several pieces of supporting engineering: - Lesson orchestration: alternate between passive listening or viewing and short live conversations. - Real-time transport: provide the communications infrastructure for interactive portions. - Cost control: keep the experience economically viable at Speak’s scale. The team warned that implementation mistakes could cost millions of dollars, describing a prospective risk rather than a reported loss.

Measure latency from the learner’s last words to the tutor’s first audio. Request-to-first-audio timing starts the clock too late because it excludes deciding whether the learner has finished. Voice activity detection, or VAD, helps determine when speech has ended; poor turn detection can add roughly another second to the wait.

Semantic VAD was described as estimating, at each token, whether the speaker is done. But behavior suited to ordinary English conversation can interrupt a learner who hesitates for ten seconds or more inside an unfinished response. The tutor must distinguish thinking from yielding the floor. Speak was working on this problem, expected custom turn detection to be necessary, and considered it the dominant factor in perceived latency.

Personalization modifies a shared curriculum backbone

Speak applies the same willingness to revisit model capabilities to its engineering culture. AI coding tools are expected to be a default path, and hiring probes candidates’ initiative in trying them. A disappointing experience with an earlier tool is a reason to experiment again as capabilities change.

The broader educational ambition was beginning to appear in business use, with management, hospitality, and communication skills adjacent to English proficiency. Math and biology remained possible future applications. An interviewer raised a trust constraint: practice may become less candid if learners believe employers are evaluating them through it. The response accepted that concern; it was not a description of an existing employee-evaluation feature.

Speak’s learner representation is customized around vocabulary and sentence patterns. It works alongside careful level placement and a structured curriculum backbone. Beginners through intermediate learners need many of the same concepts; needs diverge more sharply later. The system modifies the shared path according to individual strengths and weaknesses. Personalization therefore combines an assessment of the learner with a designed progression through the subject.

Model capability still has to become everyday value

The closing stories return to reaching consumers. Speak paid more than $100,000 for speak.com, withholding the exact price, to support its ambition for a globally scalable brand. In Seoul, its main teacher had become recognizable on the street, alongside the company’s billboards, television commercials, and influencer campaigns.

An earlier failed startup, Bloom, pursued adult learning through interactive JavaScript widgets but never found product-market fit. The participant recounting that effort described conversational LLMs as transformative for personal learning: curiosity could lead immediately to an explanation and targeted follow-up questions. The appeal was the ability to keep exploring, though this was a personal assessment of learning value.

Open-ended tutoring also exposed uses outside the intended lesson. After GPT-4-powered role-plays launched in 2023, early users entered custom scenarios the speaker considered questionable. The behavior was anticipated, but seeing actual logs made it concrete. Later models were judged better at refusals, without a quantitative comparison.

Even identifying the core customer was difficult. Speak served parents learning with children, older adults, and people studying for work, while finding a Korean sweet spot among professionals aged roughly 25–45. Broad demand made segmentation difficult, not irrelevant.

The final exchange separates model progress from adoption. One participant sees enormous real-world inertia and too few new consumer AI applications at scale, calling for more builders to turn capabilities into everyday value. Another sees gradual adoption as useful advance warning of AI’s effects. These are interpretations of the moment rather than measured adoption findings.

The closing response to that uncertainty is practical: when AI anxiety rises, return attention to the users.