Enterprise AI Search Starts with the Missing Document

Finding the presentation from three meetings ago

You remember a presentation from three meetings ago: its subject, but not where it was shared—or whether it was shared at all. That is the problem Deedy Das uses to explain Glean: helping employees find workplace information with only a partial memory of what they need.

In this Latent Space conversation, published in April 2023, Das traces Glean’s origins to experience with Google’s internal MoMA search. Google had direct API access to workplace documents and their permissions, allowing it to index and rank that information. Leaving that environment made the missing capability conspicuous. Glean began with this retrieval problem; large language models (LLMs) offered additional ways to address it.

Das identifies three changes that made enterprise search more practical and urgent: - Distributed Elastic handled much of the core search infrastructure. - Richer APIs in hosted workplace applications exposed content and permissions. - Information spread across more applications, while remote work increased the need for internal search. Access to company information mattered as much as improvements to the search engine.

The interface still had to support compact queries. At the time, Das says, natural-language questions represented only a small fraction of Glean searches. Employees often entered a project name, an acronym, or a person’s name.

Give employees a reason to return

A useful search engine does not automatically become a habit. Das argues that employees can try a newly introduced tool once and forget it exists. Glean’s expansion into an employee portal addressed that retention problem by giving people reasons to return before they had a specific search in mind.

The portal offered several routes into company information: - Go Links: memorable document addresses using Google’s internal go/ convention. - Feed: documents trending within an employee’s sub-organization, plus a limited set of recommendations. - Mentions: tags and notifications collected from Slack, Jira, email, and GitHub. Mentions had just launched, and Das acknowledged that it remained somewhat buggy.

Employee enthusiasm also has to survive enterprise purchasing. Buyers typically justify spending through revenue gains or cost reductions; time saved searching makes a softer productivity argument. A company already functions without internal search, and its value may become apparent only after adoption.

A hypothetical support-tool pitch illustrates the difference: improve ticket resolution by 20%, translate that into $100,000 in savings, and charge $50,000. Search requires a longer argument—from searches, to minutes saved, to productive work. The figures illustrate a sales pitch, not a product result.

Hybrid retrieval, then organizational ranking

AI branding cannot establish search quality. A few impressive queries followed by a bad result reveal little about an engine’s dependability. The lesson offered from Google Search is rigor in tuning ranking: sustained evaluation, painstaking refinement, and intellectual honesty about failures.

Search becomes more tractable when divided into specific problems. Query understanding determines what the user means. Synonym matching broadens which words can produce a match. Document understanding assesses the material itself, including whether it is useful content or spam designed to appear in search results. Improving these components builds the overall experience.

Glean uses hybrid retrieval, combining several ways to find results: classical information-retrieval signals, synonym matching, acronym expansion, and vector search. A further ranking and personalization layer decides which results matter to the employee.

Organizational context helps make that decision. An engineer may want documents published or coauthored by immediate teammates rather than Salesforce material. Role, team, authorship, and interactions with colleagues inform personalization. The claimed advantage comes from combining those signals with careful ranking; adding an LLM and embeddings alone does not establish excellent search.

Asked whether Glean chat is coming, the respondent declines to confirm a product. The company is experimenting with modern AI and language models, with launches guided by what users find useful.

Match the interface to the information need

The respondent reports moving a substantial share of personal queries from Google to ChatGPT, including questions they previously would not have thought to ask online. Yet the useful comparison is between query classes—different kinds of information needs. In the products available at the time, each interface had advantages.

- Lists and fresh information: Google remained useful for song lists and current movie showtimes. - Exploration: researching a subject could require several links and news sources to navigate, rather than one generated paragraph. - Specific synthesis: an obscure Doom 3 question about a particular level and boss illustrated the appeal of assembling scattered information into one answer. The Doom example was a prediction about where synthesis could help, not a demonstrated accuracy result.

Coding was the respondent’s largest personal ChatGPT use case. A coherent answer provided a convenient starting point compared with parsing Stack Overflow threads. Their tolerance for hallucinations depended on being able to verify the answer themselves.

Freshness brings search back into the architecture. Continually retraining a language model was, in the speaker’s assessment, too slow and expensive to incorporate rapidly changing information. Retrieval-augmented generation supplies fresh context by searching in the background and using the results to generate a response. Bing is the example given: its conversational interface still depends on retrieval.

That interface could change the economics of the web. The speaker worries that people who read an answer and leave will click fewer sponsored links and visit fewer of the websites supplying information. Reduced traffic and advertising revenue could weaken publishers’ incentive to create material. They also doubt that subscriptions or ads inserted into chat would readily replace search advertising. These are concerns about future behavior and incentives.

Language as an interface to tools

Smarter answers still face a latency constraint. Across products the speaker had worked on, higher latency reportedly correlated with lower engagement, approximately linearly over part of the observed range. This was an experience-based observation without accompanying measurements; the expectation was that optimization would reduce delays.

Tool use broadens the architecture beyond search. Drawing on reading rather than firsthand use of Bard, the speaker describes LaMDA querying search, a calculator, a compiler, or a translator, then revising its response using their feedback. They connect this approach to Toolformer: language handles interaction and phrasing, while tools supply computation or factual information.

The motivating concern is that storing factual associations in model parameters cannot scale as a substitute for retrieval and computation. Evaluation must also distinguish memorization from other capabilities. One participant raises possible memorization of public exam answers in GPT-4 benchmarks; another recalls a claim about different performance on older and newer Codeforces problems. Neither supplies measurements or a protocol establishing that effect.

Asked to compare approaches to tool use, the respondent declines to rank them. Their recommendation is empirical: try prompts and tool patterns, measure whether they work, and retain successful approaches. Predicting behavior in advance remains difficult.

A practical alternative is to use large models to create synthetic training data for smaller models. For named-entity recognition—tagging entities in a sentence—the proposal is to generate examples with GPT-4, then train a conventional task-specific model on them. The large model helps obtain data; the smaller model performs the eventual task. The suggestion that this could compress years of work into weeks is an estimate without a specified dataset, accuracy target, or measured comparison.

Document Q&A eventually needs an index

Having tried Perplexity and Neeva, the respondent sees them primarily as search-result synthesis products. Their skepticism concerns retrieval quality, compelling use cases, and monetization: would those products offer enough to displace established search engines or ChatGPT? This assessment belongs to the conversation’s 2023 context; the respondent already acknowledges that Perplexity may be changing direction.

A hypothetical startup illustrates the larger concern. Founders become excited about LLMs, consider consumer products, move to enterprise for perceived stability, and choose document Q&A because demos make it look feasible. Prospective customers then explain that asking questions of individual documents is not their workflow.

The request to answer questions across all company documents exposes the underlying work. The system must ingest and index them, converging on enterprise search with an LLM on top. The recommendation is to start with a customer problem, talk to users, and judge the product by whether it solves that problem.

A host offers two examples of focus: Mickey Friedman’s image-generation work for product placement and branding, and Jasper’s initial targeting of writers on Facebook. The investing discussion applies a similar test to founders: technical experience, a thoughtful approach to the problem, and understanding of the market and business matter more than enthusiasm for LLMs alone.

From shared chatbot outputs to a talking portrait

Poe suggests another use for chat: conversations can generate content useful to people beyond the original user. The speaker sees potential in learning from shared chatbot outputs, while remaining unsure whether the product idea will endure.

Image generation produces a more immediate sense of surprise. The speaker recalls their mother finding GPT-4 interesting but reacting much more strongly to generated images. Their excitement about image and video generation comes with uncertainty about revenue opportunities. Praise for ElevenLabs similarly reflects limited experimentation and examples, rather than a detailed evaluation of voice synthesis.

The speculative product is a talking digital portrait, inspired by Harry Potter’s moving pictures. Image generation, voice recognition, language modeling, text-to-speech, and personalized imagery would let someone greet a display and receive a reply in the depicted person’s voice and conversational style. The speaker believes the components exist; bringing them together remains a proposed experience.

Price the final run—and the work that precedes it

Infrastructure costs can be invisible to engineers inside large companies. The speaker recalls working on Google Search without knowing its operating costs beyond public information. Startup builders need that visibility: understanding how infrastructure spending grows with use is essential to the economics of serving customers.

Disclosed accelerator counts and training duration can support approximate model-cost estimates even when papers omit the bill. The speaker estimates $4 million for LLaMA and $27 million for PaLM, describing these as upper bounds that might be roughly twice the actual cost. Negotiated bulk compute pricing is one source of uncertainty.

The essential distinction is final training cost versus total development cost. The estimates exclude hyperparameter tuning, architecture exploration, training-data experiments, failed runs, and debugging. A rough tenfold multiplier is offered as a common rebuttal, not an established measurement. Funding a successful run and funding the work that discovers it require different budgets.

Local models create another route to experimentation. The speaker credits local image generation with enabling forks and specialized fine-tunes, and C++ implementations plus inference optimization with making moderately capable language models runnable on personal computers without buying GPUs. Faster, accessible models could allow individuals to pursue ideas beyond company research programs.

That access carries an unresolved tension. Local text generation could enable sustained, varied spam and harassment; advocates of individual access also object to reserving the capability for companies such as OpenAI.

The speaker identifies several promising directions: Runway ML for generative video editing, streaming speech recognition, better text-to-speech, and an open alternative to ElevenLabs. Combining modalities could also support editing video or assembling it from images. Luma’s generative 3D work is intriguing, though its practical applications seem less clear to the speaker.

Watermarks and the fingerprints of adjusted scores

The respondent supports text-watermarking research aimed at making generated writing identifiable. They favor standards across providers, so detection would extend beyond each provider’s own outputs. Their motivations include recognizing machine-written assignments and avoiding training on a model’s own generated material, which they describe as harmful. They offer no particular method or effectiveness result.

A separate investigation of Indian examination results turns to statistical traces of intervention. In the investigator’s account, exam performance can determine both college access and field of study. Results were accessible with only a roll number, enabling collection and analysis of the distribution of marks.

Two reported patterns stood out: - Missing near-failing scores: the passing grade was 33, but no scores appeared from 29 through 32. The investigator interpreted this as marks being raised to passes. - Gaps among passing scores: roughly 30 values between 33 and 93 were absent, including 91 across the subjects and years examined. The second pattern initially looked like a code bug. The investigator ultimately interpreted it as evidence of undisclosed adjustments to final marks.

Another board reportedly showed a large spike at 95. The proposed mechanism was a fixed score increase capped at 95—for example, adding ten points for a harder mathematics exam. That cap would collect different original scores at the same final value. The objection concerned opaque adjustments to consequential scores, rather than a general rejection of examinations.

The investigator dates the findings to 2013 and says later password protection prevented further scraping. Their suspicion that the practice continued is not a follow-up finding. Another participant compares the anomalies with scrutiny of investment returns: unusual distributions can provide clues about intervention and reasons to investigate how the numbers were produced.

The useful assistant knows when to hand work back

The lightning round returns to learning and building. One participant recommends AI Pub for periodic roundups, preferring reading at their own pace to time-consuming communities. Asked about the following year, the respondent predicts larger gains in model cost and latency than in quality, which they believe is nearing a maximum—a forecast made in this 2023 conversation.

The proposed communication assistant has a precise requirement: automate routine messages and flag important ones for the user. Deciding which messages warrant automation is central to a useful balance between automatic replies and suggestions.

Another participant reports building a Gmail prototype that authenticates with an inbox and pre-drafts replies. Its prompt includes personal role and workplace context for users to customize. But a scheduling instruction fails to stay conditional: asked to suggest a time or provide a calendar link when someone asks to schedule, the model adds a calendar invitation to every reply.

The failure makes the missing capability concrete: generating the requested text is insufficient when the system cannot restrict it to the intended context. The proposed GitHub assistants likewise specify work around generation—checking issues for missing information, drafting follow-up requests, and closing issues after roughly 30 days without a response. A separate proposal would find vulnerabilities and submit patches as pull requests. These remain requested products, not demonstrated implementations.

Das closes with an invitation to bridge research and end users: build things, read original papers, and understand how the systems were made. He recommends Andrej Karpathy’s YouTube explanations as one route into that material.