Building Snipd: From Podcast Playback to Usable Knowledge

Capture an insight without interrupting listening

An interesting podcast moment arrives while you are running, driving or walking the dog. Capturing it can mean taking out a phone, saving a timestamp and hunting through the audio later. [Snipd](https://link.snipd.com/Cx7S/swyx) reduces that work to a headphone gesture: triple-tap after a passage, and the app saves and summarizes it into a note called a snip. Its purpose is to help people who listen to learn capture knowledge they can use.

The initial concept was a TikTok-style feed of podcast clips. Expecting listeners to consume clips readily but hesitate to create them, the team concentrated on effortless capture. After launch, it found that people used snips for discovery while still preferring full episodes—and created snips enthusiastically. That qualitative feedback shifted the emphasis toward learning and knowledge capture.

One participant had previously made a personal mixtape podcast from five- to ten-minute excerpts with commentary. Each entry began as an Overcast URL and timestamp in a notes app, followed later by downloading and clipping the MP3, recording an introduction and outro, and publishing. Snipd’s gesture simplified the capture step in that laborious workflow.

From banking models to search within an episode

The Snipd guest’s path began with mathematics and economics at ETH, specializing in quantitative finance. Modeling and statistics appealed more than finance itself. A friend recommended a machine-learning course; a weekend reading its slides became roughly a year of reading and building models. After experimenting with machine learning at a bank, the guest joined an early-stage Zurich startup and spent five years building its AI team.

Projects included product recommendations for bank sales teams and a text-processing model that made cryptic credit-card transaction descriptions readable. Later, two friends working full-time built Snipd’s precursor over a Hack Zurich weekend. Their question was concrete: could natural-language search find a moment inside a podcast?

Their demo searched for smoking weed in the roughly two-and-a-half-hour Joe Rogan episode featuring Elon Musk, then played the relevant passage. Retrieval used audio; video made the stage presentation memorable. They won, but the stronger signal came afterward: participants asked to use the prototype and described adjacent podcast problems. The founders subsequently left their jobs to build Snipd.

At the time of the conversation, Snipd had four technical team members: two handling AI and backend work, and two building the frontend. Alongside iOS and Android, its Apple Watch app served listeners such as runners leaving their phones behind. It supported downloads and streaming with internet access, concentrating on listening and snipping; users could review their captures later on the phone.

The Watch also exposed a hosting dependency. During the conversation, a participant encountered a playback error for a Substack-hosted episode. Snipd reported that the problem also affected Apple’s Podcasts app and displayed a specific explanation after identifying it. The underlying cause remained unexplained.

Make episodes navigable—and connect them

Searchable transcripts motivated one listener’s switch from Overcast. Snipd adds transcription, speaker names, short guest biographies, attempted image lookup, and chapters with titles and descriptions. Speaker diarization first groups speech blocks belonging to the same voice; assigning a person’s name to those blocks is a separate task.

Book discovery turns an extracted reference into another listening opportunity. A large language model (LLM) scans the full transcript for book mentions. The pipeline combines the Perplexity API with other model calls to retrieve a cover, author and short description, then checks for episodes featuring that author. This supports the guest’s own habit of listening to an interview with a writer for an overview before pursuing a book.

That connection depends on distinguishing being mentioned from appearing as a guest. Early results surfaced too many absent people: frequent discussion of Sam Altman or Elon Musk could overwhelm actual interviews. Some listeners welcomed episodes about a person, but Snipd wanted guest appearances separately identifiable. The team reported improved identification without accuracy figures.

Within an episode, listeners can search keywords or ask the chat to locate a topic, produce key takeaways or create a note. A snip preserves the summary alongside the original transcript and audio in a knowledge library, keeping the captured idea connected to its passage.

Dynamic ads break the timestamp assumption

A timestamped transcript assumes the listener receives the audio that was transcribed. Dynamic ad insertion breaks that assumption: a server can insert different advertising into an episode’s MP3, changing its timeline. Word-level timestamps from an earlier file can then point to the wrong passage.

Snipd resynchronizes the listener’s audio with the existing transcript on the fly. The guest describes fuzzy matching at approximately a byte-level representation, rather than matching transcribed words or requiring exact matches. The comparison is to Shazam for podcasts, without specifying the fingerprinting implementation. Snipd detects dynamic content; it does not automatically skip it.

Choose models and infrastructure by task

The backend was predominantly Python on Google Cloud. Flutter and Dart provided one mobile codebase for Android and iOS, which Snipd credited with fast development. The team had no plans to move away, although the Watch required native development and synchronization with the Flutter app—a tradeoff the guest relayed from the frontend engineers.

The first release preceded ChatGPT and Whisper. Snipd initially hosted open-source models and attempted fine-tuning, using wav2vec for transcription. The guest credits the wav2vec paper with encouraging the move into audio: it was the first work they had personally encountered bringing transformers to audio and continuous data, suggesting that features beyond their immediate reach would become feasible.

By the conversation, Snipd used Whisper and still self-hosted transcription and diarization to control audio-processing costs. It automatically processed selected podcasts; paying members could request others. An LLM combined speech blocks with guest names and biographies to assign speaker names.

Most remaining processing used hosted APIs, primarily OpenAI and Google’s Gemini models, with Perplexity for web search. Snipd had tried Google Search Grounding, but had not compared quality in depth because Perplexity already worked. Its selection rule was to identify the capability a task requires, then find the best price at that level. Processing costs determined how broadly features could be offered across podcasts.

Use podcast structure to improve speaker separation

Snipd’s modified open-source diarization pipeline creates speech embeddings—representations of speech that it groups into speakers. The team changed the clustering algorithm and added podcast-specific heuristics. Controlled, high-quality recordings help, as does episode structure: someone speaking for thirty seconds in an hour-long episode is more likely to be an advertising voice than a host or guest.

Those cues may transfer poorly to everyday recordings with arbitrary environments and conversational structures. Snipd also combines transcripts, LLMs and diarization outputs to adjust where one speaker stops and another starts. Challenged about errors an LLM might introduce, the guest acknowledged that diarization remained imperfect and described improvement over the preceding year without measurements.

The listener’s most immediate delight was simpler: click transcript text to jump to the audio. For the guest, that reaction illustrated the discipline consumer AI builders need: keep returning to the user’s task. A straightforward navigation feature can deliver substantial value without sophisticated new modeling.

A prompt is only part of the interface

A request for custom summary prompts exposed a product gap. Snipd pointed toward its existing chat, but the listener wanted a preferred summary already waiting in a readable format. Move beyond the chat box was the guest’s lesson: making a capability accessible through prompting does not finish the interface. Regenerating whole-episode summaries for each listener also adds cost, so broader personalization depended partly on expected price declines.

Clickable chat citations show the engineering behind that interface. The model receives the full transcript and instructions to return timestamp references in a particular format; the frontend renders them. A quick demo is easy, but format violations can make production output look broken. Snipd uses numerous regular expressions to repair those violations.

Streaming shapes that repair strategy. Displaying text as it arrives reduces perceived waiting time, but complicates sending it through a second LLM for correction. The APIs under discussion supported streaming output, not streaming input into another model. Controlling the models might permit feeding emitted token sequences onward, but the participants considered that machinery unjustified for this feature.

The guest described evaluation as informal vibe evals: accepting occasional imperfect outputs to iterate quickly. This was a stated startup tradeoff, not a measured reliability result.

Give stronger models smaller decisions

Snipd’s year-end Wrapped experience used saved snips to generate a playful personality, a learning scorecard and a standout quote. When quote selection sometimes returned something dull, the team split the work: 1. A cheaper model processed the large input and extracted five candidate quotes. 2. A stronger model selected one candidate. The expensive model handled a small judgment task. The guest reported that the approach worked for quote selection, without a quantitative evaluation.

A second LLM also challenged book extractions that included unmentioned books, non-book items or invented titles. Snipd used a similar check for speakers, though the exact validation was unspecified. No post-check accuracy was given.

Model preference and production choice could diverge. The guest preferred Claude’s phrasing and personality, and the team used Claude 3.5 Sonnet extensively in Cursor and for brainstorming. Cost led to other models for many product tasks; the team had experimented little with Haiku.

Infrastructure followed two demand patterns: - Episode preprocessing: transcription, diarization and chapter generation ran once per episode through a continuous queue. Release patterns made demand relatively predictable. - User-triggered inference: requests needed real-time responses and arrived in spikes and quiet periods. Hosted APIs reduced the work of managing fluctuating capacity. The preprocessing queue was a custom database-backed system, with backend workers picking up jobs and returning results. Google Pub/Sub handled other orchestration.

Some large-input tasks might eventually justify open-source models, but the guest offered no percentage saving: iteration speed and learning from users took priority over infrastructure work. The same pragmatism applied to replacing the audio pipeline. Informal Gemini 2.0 Flash experiments suggested transcription and diarization cost substantially more than the self-hosted system. The guest expected a multimodal model to replace separate stages once the economics worked; no workload sizes, prices or accuracy comparison accompanied that expectation.

Turn the end of an episode into a learning trigger

Future content directions included native audiobooks and AI-generated listening material, such as combining deep research with NotebookLM-style podcast generation. For discovery, the guest imagined listeners stating immediate goals to the recommendation system: set aside enjoyable cat videos for two hours and focus on AI. Such control fit Snipd’s stated learning purpose.

Voice offered a more specific opportunity. The guest’s starting question was what event causes someone to use an app: wanting to travel, for example, prompts opening Airbnb. A reflection feature needs an occasion too. Asking listeners to return later through reminders adds friction; an episode ending provides an existing moment to build around.

The proposed interaction asks listeners to choose one personally important takeaway, even from an episode containing many useful ideas. Selecting one requires judging relevance and considering an action. The guest practiced this personally and wanted to support it in the product, while acknowledging that reflection requires effort.

A two- or three-minute AI conversation could begin when an episode ends, keeping the listener in the audio habit while shifting toward reflection and application. A participant initially unconvinced by podcast chat found that purpose more compelling. The interaction remained undecided, and no measured learning improvement was reported.

Other possibilities remained exploratory: browsing snippets, discussing one and retrieving relevant passages from the full episode; or collaborating with creators so listeners could converse with an AI version of Swyx through Latent Space in Snipd. Neither was presented as an existing integration.

Creators, discovery and the cost of switching

From a creator’s perspective, one participant urged prioritizing YouTube for its recommendations and established viewing habits. Snipd’s guest favored backgroundable video, crediting the concept to Gustav Söderström: primarily listen, then look when a clip, slide or demonstration needs visuals. More engaging discovery was the biggest proposed benefit for Snipd.

The creator distinguished three audiences: listeners, people adding snippets and annotations, and podcast creators seeking discovery for their work. Snip counts served as a personal proxy for active engagement—a deliberate capture or share—alongside incomplete download visibility.

Production offered another possible direction. Latent Space had recently started a shorter, remote, YouTube-only series using Riverside. A participant found its automatic switching between active-speaker and group views useful, but reported rough editing, weak thumbnails and suggested shorts, plus manual downloading and republishing. The desired workflow was record, press a button, finish publication. Reducing that effort, the participant argued, could help busy practitioners publish without learning production tools. It was a proposal for Snipd.

The closing invitation described Snipd’s freemium offering and offered a show-notes link for a free month of premium access. Migration remained a barrier: one participant waited four to six months after a demo before switching because the import process did not preserve partially listened-to episodes or personal rankings. The practical advice was to use the switch to prune accumulated subscriptions and refocus listening.

The conversation closed by recognizing a small technical team building a consumer app as a useful example for other builders, alongside the larger venture startups and business-software companies often featured on the podcast.