AI Is Eating Search: What Websites Must Deliver

The website chatbot hid a discovery problem

Banks and insurance companies kept asking Robert McCloy and his co-founder Chris for a ChatGPT-style website widget. The two were exploring ideas after leaving Hearsay, but Robert doubted that visitors wanted to find a website and then adopt another chatbot. [Scrunch AI](https://scrunchai.com/) grew from investigating the discovery problem behind that request: how would customers encounter a business through AI?

Swyx questioned whether technologists’ dislike of pop-ups represented mainstream users, pointing to Intercom’s business success. Robert distinguished support within an existing relationship from discovery on a first visit. Existing customers may welcome a business’s support assistant; people exploring options may prefer staying in ChatGPT, Claude, or Perplexity. His evidence for that preference came from conversations with people he knew.

The opportunity became concrete when the founders showed marketing executives how ChatGPT Search described their businesses. Their brands were appearing in a customer experience they could not readily see or understand. Scrunch’s thesis became broader than improving search rankings: help businesses understand how AI changes discovery and customer interaction. In the July 2025 conversation, Alessio disclosed that Decibel had invested in the financing round being announced with the episode.

Training and retrieval offer different points of control

Scrunch starts by simulating consumer prompts across AI platforms. Robert described coverage of ChatGPT’s search and trained-in knowledge, Gemini, Perplexity, Claude, and Meta AI, with more platforms being added. Selection considers audience value as well as usage: a smaller platform can matter disproportionately when its users overlap with a business’s customers.

The source of an answer determines how readily a business can influence it: - Trained-in knowledge: Information incorporated during model training is harder to act on when an answer is wrong. - Search-backed information: Retrieved pages provide a more actionable surface for updates. Robert argues that commercially relevant queries increasingly use search for timeliness and to reduce hallucinations.

For training crawlers such as GPTBot and CCBot, Robert recommends considering how quickly information becomes stale. A seasonal sale can end while a model continues recalling it. Scrunch therefore advises customers to consider withholding short-lived promotions from training crawlers.

Restricting access can also restrict discovery

Bot paywalls introduce two separate questions: - Payment infrastructure: Can publishers and consuming platforms make compensation work end to end? - Bargaining power: How much access can publishers restrict when they depend on search referrals? A strong direct audience gives a publisher more freedom to impose restrictions. For a business dependent on discovery through other platforms, the tradeoff remains unresolved.

One participant favors free distribution and model training in the hope of gaining authority and referrals. AI browsers and local tools complicate the access question further: negotiating payment with a centralized platform differs from preventing a user’s model from consuming material already available in their browser.

The broader proposition is that AI can replace portions of web browsing. Users delegate reading and synthesis instead of clicking through pages themselves, for some uses of the web. Referral effects vary too. A participant reports that Lenny’s newsletter received more traffic from ChatGPT than Twitter over a period beginning May 10 and ending the week before the conversation, while their own publication still received substantially more from Twitter.

Search rank gets a page considered; relevance gets it read

The technical account still starts with conventional search indexes and ranking methods. Search engine optimization, or SEO—work intended to improve a page’s visibility in search results—therefore remains relevant. But retrieval is only the first hurdle: an AI system also selects which results to read and use.

The described pipeline has four stages, with details varying by platform: 1. Generate queries: A large language model turns the request into one or more keyword searches. 2. Retrieve candidates: Searches run in parallel, or potentially sequentially with a reasoning model. 3. Select for relevance: The system reorders or filters conventional search results. 4. Read and synthesize: Selected pages supply content for summarization and answer generation. Reranking here means reordering or selecting results; it does not imply a dedicated reranker model.

A high-ranking page can disappear at the selection stage. The reported failure mode is a title or description optimized for human clicks—such as urgent sale language—that says little about the page’s contents. An assistant may instead select a lower-ranked result with more descriptive metadata. Metadata must explain relevance before the content gets read.

The page itself then needs enough detail to support a recommendation. Apparel illustrates the distinction: photographs remain important to shoppers, while text explaining who buys particular leggings and for what purpose gives the assistant useful context. The recommendation is to disclose who the product serves and what it helps them do, often with the aim of sending a shopper to a page with checkout. Improvements to visibility and purchasing are reported without a quantified uplift.

Referrals reveal demand without revealing the prompt

Could a website rewrite its landing page around the query that brought an AI-referred visitor there? The proposal runs into a missing input: websites receive almost no information about the originating ChatGPT prompt. The response also anticipates privacy limits on future sharing, because conversations can contain sensitive health and personal information.

The destination page still offers indirect evidence of demand. Developer infrastructure customers reportedly noticed ChatGPT referrals arriving at posts about particular problems and published more content on those topics. The approach was described as effective, without quantified results: identify the problem attracting visitors, then supply more useful answers.

Hallucinated features might offer another demand signal. Participants recall an unverified example of a company building a feature ChatGPT had invented, comparing it with implementing methods repeatedly hallucinated by a coding assistant. Clearer product descriptions might have prevented the error, though the company had not been closely examined. The immediate content problem is less speculative: reviews of startup homepages found elaborate visual effects that left the basic question—what does this company do?—unanswered.

Experiments can help identify what works, but the surrounding infrastructure is changing quickly. Scrunch provides a feedback loop for observing how content changes affect performance. The discussion attributes some volatility to immature code connecting models, search, and the web, including less developed abuse defenses. Tactics that happen to work under those conditions may not last.

Agent experience depends on the customer’s context

Scrunch’s broader category is agent experience: how AI systems discover, consume, and interact with a business’s content and infrastructure on a customer’s behalf. Search visibility is the entry point. The proposed goal is to measure more of the customer journey as it moves into assistants.

Personalization makes that measurement harder. Memories and explicit preferences can affect what ChatGPT searches for, which sources it prefers, and how it presents information—even when the underlying search index is not personalized. One speaker reports different results with instructions requesting concise, senior-engineer-level answers versus a default session, an observation rather than a controlled comparison.

A single universal visibility score could therefore become inadequate. Scrunch describes grouping prompts by customer persona, such as senior engineers or product managers, to model what those audiences are trying to accomplish and assess their experience.

Completing work can absorb the searches that preceded it

High prompt volume does not necessarily mean high commercial value; role-playing is offered as one example. Conversely, a task labeled “generative” may have substantial commercial intent. An unnamed study using opt-in consumer browsing data reportedly found extensive generative use of ChatGPT and suggested that traditional search might be relatively protected.

The counterargument is that asking AI to complete work can replace researching how to do it. Writing a proposal, building a financial model, or producing runnable code can absorb earlier informational searches. Someone already solving a problem inside ChatGPT or Cursor may be closer to taking a commercial action than someone merely exploring options.

Business software procurement makes this tangible. A buyer required to compare several products can delegate the work to ChatGPT or Deep Research. The comparison can follow the buyer’s requirements rather than the features vendors emphasize. Product descriptions become inputs to a purchasing workflow.

These usage observations combine consumer panels with qualitative observation and follow-up questions to people who reported discovering a business through ChatGPT. The speaker cautions that panel participants may not represent high-value professional users, including senior engineers. The examples explain a commercial mechanism without establishing how common it is across all users.

Fix content and delivery before chasing tricks

Asked about prompt injection—placing instructions on a product page to manipulate the consuming assistant—a participant says it often works at the time of the conversation. The response predicts penalties as systems mature, while acknowledging few reports of bans so far. The concern centers on immature integration code between search, pages, and models: exploiting current behavior is a fragile strategy.

The recommended improvements address more durable needs: - Supply missing context: Clear descriptions, facts, examples, and frequently asked questions help models make accurate comparisons and recommendations. - Keep content coherent: The speaker favors understandable structure and clean HTML over excessive attempts to optimize topic similarity. - Separate documentation packaging from discovery: An AI-oriented text-file format is described as useful for loading existing documentation into a model’s context window, but not something ChatGPT retrieved by default at that time. The speaker reports little discoverability benefit.

The clearest engineering failure is useful content absent from server-rendered HTML. Participants describe Next.js pages whose content depended on useEffect and was unavailable to AI ingestion. Their practical diagnostic is to disable JavaScript and inspect whether the content remains. In the systems discussed, client-side rendering could prevent an assistant from receiving the information at all.

Programmatic SEO—generating pages at scale—needs a similar distinction. Remixing abundant public information may occasionally win citations, but competitors have the same tools. Publishing knowledge particular to the business offers a stronger rationale. Customers reportedly turn recurring support tickets into how-to pages, matching the problem-solving questions people bring to ChatGPT. The pages are said to help users and reduce support questions, though no measured reduction is given.

Deep Research makes consistency more consequential

The discussion distinguishes three retrieval patterns: - Single search: Retrieve results, select or reorder them, and summarize an answer. - Parallel searches: Issue several searches simultaneously to gather material. - Sequential research: Use a reasoning model to identify unanswered questions and launch follow-up searches. Deep Research is distinguished here by sequential follow-up: earlier findings influence what the system investigates next. Its content-ingestion requirements are described as broadly similar to ordinary search, including the problem with JavaScript-dependent content.

More retrieved material can introduce more contradictions. A website may contain current pricing alongside old articles quoting obsolete prices. Reading more pages exposes the assistant to conflicting answers and can produce the wrong price. One speaker also reports cases where Deep Research was less useful than ordinary search because the additional material did not contribute to the desired understanding.

Related links or suggested searches might guide follow-up research. The response acknowledges some influence from earlier results, but favors including the needed information directly when feasible. Branching topics and limits on how much text a model can take into context prevent a single page from containing everything. The proposed compromise is a focused page with a complete answer for a particular query or persona.

The same constraint limits giant consolidated documentation files that exceed common models’ context windows. Conventional search over focused website pages already offers a way to select manageable pieces, provided those pages answer users’ actual questions.

Measure conversions and keep responsibilities clear

The case-study discussion prioritizes conversions over traffic alone. Clerk, the user authentication company, is presented as a customer whose strong documentation and targeted content work improved AI referrals. The speaker tentatively recalls roughly sixfold traffic growth and ninefold conversion growth. No timeframe, baseline counts, conversion definition, or consistent boundary between ChatGPT and other AI channels is supplied. These are reported growth figures, not a conversion-rate comparison.

The proposed mechanism is readiness to act: people asking implementation questions already have a concrete problem, and useful content brings them closer to solving it. The work is divided accordingly: - The customer creates the content: Clerk’s team uses its knowledge of developers and their needs. - Scrunch supplies feedback and experimentation: It helps identify what works and is developing more structure for running experiments and delivering content to AI platforms.

Technical delivery beyond traditional search remains an area of ongoing work. NLWeb is mentioned as part of that exploration, without an implementation explanation.

Robert reports typically seeing double-digit traffic improvements within one or two months for businesses actively updating websites and publishing content. Results vary by vertical, with developer tools a particularly good fit; he explicitly declines to promise sixfold growth generally. The companion article separately claims conversion rates two to four times higher for AI referrals than traditional search, without specifying the sample, measurement method, or conversion definition.

Follow audience affinity, then investigate what works

Robert’s closing platform comparison distinguishes three reasons to pay attention: - Consumer attachment: He sees ChatGPT as having the strongest durable consumer relationship. - Distribution: Google can put AI directly in search users’ paths, while Meta benefits from its existing reach. Robert lacks good statistics for the then-new Google AI Mode. - Audience fit: He recommends Perplexity for technology and some finance audiences and highlights Claude’s smaller but passionate user base.

Consistent usage matters more than a momentary ranking. Robert describes ChatGPT as consistently strong and Perplexity as maintaining steady usage at a smaller scale, compared with more pronounced peaks and troughs for Gemini, DeepSeek, and Grok. These are qualitative comparisons, without underlying figures presented. He also considers ChatGPT’s position conditional on users continuing to like the product: perceived changes in the wrong direction can provoke substantial pushback.

His final recommendation is to make publishing compatible with AI and understand how it appears in answers: investigate what works, avoid snake oil, and take the change seriously. He closes with Scrunch’s then-open roles investigating how AI and the web interoperate, and how businesses can reach customers through that changing environment.