Phind’s path from retrieved answers to coding models

A fast model still needs a reason to be used

Around 2017, recognizing an object with Google Lens could mean waiting for an image upload and a server response. Michael Royzen built SmartLens to run recognition locally on an iPhone, avoiding that round trip. Inspired by an on-device computer-vision API at Apple’s WWDC, he filtered ImageNet 22K into approximately 17,000 categories and fine-tuned the Inception V3 image-recognition model.

Royzen recalls it working better than the first Google Lens, particularly because it avoided the latency of mobile connections at the time; he gives no quantitative comparison. TechCrunch coverage brought a usage spike that later tapered off. Some monthly subscribers remained at the time of the interview, despite the app requiring no servers.

In college, he turned the small, fast classifier into an enterprise product without first talking to users. An invoice-processing API—extracting structured fields from unstructured documents—led him to Hugging Face, BERT, and Longformer’s larger context window, the amount of text it could process at once. He recalls working with roughly 4,000 tokens instead of the 512-token limit of his earlier models. The technology worked, but few people used the product, and he lacked the personal interest to sustain it.

A question-answering demo becomes a search system

In summer 2020, Royzen encountered Yacine Jernite’s long-form question-answering notebook. He remembers a question about why boats are white. The answers were unreliable, but the interaction stayed with him: the user could ask a question without supplying examples or context. The system retrieved information before answering.

That arrangement is retrieval-augmented generation, or RAG: retrieve relevant material, then give it to a model that generates an answer. Royzen recalls BART as the notebook’s answering model, with retrieval options using Elasticsearch over Wikipedia or a FAISS-powered index. He fine-tuned a larger BART model on ELI5, a dataset of human-written Reddit questions and answers. The examples taught it the form of an answer, even while its reasoning remained limited.

The resulting model mainly summarized: answer quality depended on retrieved context. Once he reached BART large, Royzen concentrated on the index. Roughly $10,000–$15,000 in leftover AWS credits funded processing of Common Crawl. He filtered its pages using an Alexa popularity list—tentatively recalling the top 10,000 sites—to exclude the web’s long tail and limit the project’s scope.

The prototype reportedly indexed approximately 350 million pages in Elasticsearch and produced promising general-knowledge answers. Hosting costs and college coursework pushed it onto the shelf after fall 2020. Development resumed in fall 2021 with [BigScience T0](https://huggingface.co/bigscience/T0). A T0 3B model, fine-tuned on ELI5 and connected to the index, worked well enough to sustain development and launch on Show HN in January 2022 under the name Hello.

Search before trusting the model’s confidence

Work on factuality in question answering, undertaken for the builder’s UT Austin honors thesis, helped establish a product rule: always search, rather than letting model confidence decide whether to search. A model could confidently misunderstand a question without recognizing its knowledge gap.

One reported failure made the problem concrete. Asked how to run llama.cpp, ChatGPT browsing treated the name as an ordinary local C++ file and offered GCC compilation advice without looking it up. The consequential mistake happened before retrieval: the model acted as though it understood the task. Phind’s search-first policy aimed to avoid relying on that judgment.

In June 2022, around entering YC, the team focused on programmers. It expected basic summarization and current-events answers to become cheaper commodities. Improving model economics offered two strategic choices: - Same intelligence, lower price: deliver an existing capability more cheaply. - More intelligence, same price: spend the available budget on harder reasoning, debugging, and multistep problems. Phind chose the second, connecting a reasoning model to codebases and the internet.

[Phind](https://www.phind.com/) became a system for getting developers unstuck. A user could submit a question or broken code; Phind would locate relevant internet or codebase material and pass it to an answering model. The pitch promised to resolve an issue in 15 seconds instead of 15 minutes, though this was an ambition rather than a measured result.

Context gathering shapes the interface

Two usage patterns shaped the product: - Find the relevant documentation: answer how to accomplish a task with a particular tool by retrieving the useful portions of its documentation. - Repair supplied code: diagnose and fix code pasted by users, a behavior that emerged organically as Phind increasingly used GPT-4. The VS Code integration made code repair smoother by finding context inside the codebase. It could also search the internet, and users could supply material manually when automatic selection needed help.

The team saw no immediate need to own the editor. Its desired features, including suggested diffs, had been achievable through a VS Code extension using supported functionality or workarounds. The web interface served design, planning, and conceptual exploration before coding began. The intended workflow ran from an idea to a working product, with the editor covering only part of it.

Phind reported users solving problems that ChatGPT with GPT-4 had failed to resolve, suggesting documentation access or multistep reasoning as possible explanations. For a long period, Phind itself relied primarily on GPT-4, with web search providing its principal added value. These observations came without a controlled success rate, but they explain why the team treated retrieval as part of answer quality, even when the underlying model was the same.

Stronger answering models changed the product

After entering YC, the team abandoned its cumbersome search index and used Bing for raw search data, launching as Hello Cognition. Paul Graham proposed the spelling Phind during a domain-brainstorming session. The founders initially disliked its spelling difficulty and dated feel, but eventually bought the domain and quietly renamed the company in late December 2022.

The programmer-focused version from summer 2022 included code-snippet search because its models generated incomplete snippets. Better embeddings and HTML parsing subsequently improved context retrieval, but the team associated each major version with a stronger answering model. In February 2023, it switched to OpenAI after judging its own models insufficient. Traffic and retention reportedly improved, although GPT-3.5 still struggled beyond documentation-style how-to questions.

GPT-4 enabled the reasoning behavior the team had wanted, leading to the April Hacker News launch. Phind offered free access without login to demonstrate the product. Repeated launches and a Fireship Short brought attention, but usage was expensive. Unofficial APIs exploited that free access; OpenAI reportedly helped Phind strengthen its Cloudflare bot controls.

Specialized training offers control over capability and cost

Rising costs made running its own answering model attractive again. Phind’s hypothesis in 2023 was that the gap between open and proprietary models would narrow enough for application-specific training data to overcome it. The team predicted that startups could use their own data to outperform proprietary alternatives for their applications in 2024. This was a forecast, not an established outcome.

Users also wanted to ask questions about entire codebases. Longer context, faster responses, and lower cost were explicit model goals. The team argued that placing raw files in a sufficiently large context window could work better than selecting chunks through retrieval, reducing the chance of omitting needed material. It supplied no comparative measurement for that claim.

The training strategy centered on high-quality data. One example, tentatively identified as BigCode’s Octopack, used human-written code changes as supervision: give a model a file’s pre-commit state and the commit message, then train it to produce the resulting code. A requested change paired with its implementation supplied a concrete signal for learning coding behavior.

The Phind model started from Code Llama 34B, retaining its architecture and training on many additional billions of tokens using Phind’s infrastructure. Code Llama already included long-context pretraining. The discussion described a 16K position limit and an adjustment to its positional encoding intended to support sequences up to 100K tokens. That longer reach was theoretical support, not demonstrated effective use of the entire context.

Beating GPT-4 depends on what you measure

As of October 5, the team reported leading the BigCode Leaderboard, particularly outside Python. The claim lacked specific model versions and metrics needed for a reproducible comparison. Phind also questioned whether HumanEval, a coding benchmark, represented its users’ work.

The critique separated two problems: - Narrow task coverage: HumanEval captured only a limited subset of programming work. - Possible training-data contamination: a speaker suspected GPT-4 had encountered the benchmark during training after observing exact docstrings and example values in its output. That was an inference, not proof of training-data provenance.

Phind emphasized two application-level checks: - Score representative user questions: GPT-4 evaluated answers on an internal dataset intended to reflect actual Phind usage. With its temperature setting at zero, scores were reportedly mostly deterministic across runs. - Release to users: the team reported little perceived difference from GPT-4 for most Phind searches, while acknowledging reasoning problems where GPT-4 remained stronger. Personal comparisons sometimes favored Phind’s concision or implementation, but no controlled win rate was given.

Additional code training reportedly helped the model solve non-code riddles involving objects moving over time. The team interpreted this as emerging reasoning capability, without a benchmark or measured effect size. Yet producing a correct answer once did not mean producing it consistently: the model could succeed on one attempt and fail on another.

That inconsistency motivated a proposed reinforcement-learning experiment using correctness as feedback. Generate several answers to a problem, identify the correct ones, and train the model to favor them. The aim was to make a behavior the model already exhibited more reliable. It remained an experiment to try, not a reported training result.

Long conversations need explicit memory choices

Phind’s pair programmer asked clarifying questions and supported tasks requiring multiple searches or reasoning steps. Users were trying to move from a basic idea to working code in threads of 60–100 messages. Those conversations made it difficult to decide what should remain in the model’s context.

Message prioritization let users preserve important context. It arose from requests to pin messages so they would stay in the conversation context. The team said the feature substantially helped long conversations, though it offered no quantitative evaluation.

Execution was the next proposed extension. Phind already offered Run in Replit; a native sandbox could let it execute code and iterate on the result. The team expected coding tools to converge on similar problems from different starting points: Replit from a browser editor, Phind from a model connected to relevant resources. A broader technical-reasoning product beyond programming was another longer-term possibility.

An investor introduction leads to GPU access

During YC’s summer 2022 batch, the founders had just graduated and chosen the startup over jobs secured through earlier internships. They planned to build in New York near college friends, despite mentors urging them to consider San Francisco. While in New York, they secured a 7 p.m. office-hours slot with Paul Graham and made a same-day round trip to meet him at YC’s Mountain View office.

Their pitch imagined search delivering direct answers. In the founder’s recollection, Graham agreed to invest before seeing the demo. The meeting extended into dinner, where Graham imagined future generations receiving answers without knowing conventional search results. The funding round then came together quickly, and Graham introduced the team to Ron Conway. The founder’s lesson was that accelerator opportunities required active pursuit, including asking for the investment.

When Conway visited the team’s San Mateo workspace, its immediate need was GPU access: obtaining enough quota was already difficult. Within hours, he reportedly contacted Nvidia’s CEO, leading to an introduction to an AI vice president and a dedicated support team. Access did not require owning hardware; Phind wanted help finding capacity that fit its needs and budget.

The team valued Nvidia’s help comparing cloud capacity and possible hardware purchases, rather than pushing a single provider’s long-term commitment. Software support mattered too: Nvidia reportedly added streaming generation for T5-based models in FasterTransformer. Phind used those models until switching to GPT around February or March 2023.

Local deployment brings a size–quality tradeoff

Companies were asking for a Phind-like system within their own environment, on hardware they controlled. That demand motivated interest in efficient local inference, though a completed deployment was not described. Reducing numerical precision offered two approaches: - INT4 and INT8 quantization: use four- or eight-bit integer representations to reduce model size, with quality loss the speaker considered relatively small but noticeable. - Mixed-precision FP8: combine eight-bit floating-point formats with different precision-versus-range tradeoffs, retaining 32-bit floating point for some layers. Efficient hardware support was a prerequisite.

Nvidia reportedly claimed no degradation from its mixed-precision strategy demonstrated with the H100. Whether that held in practice remained an open question for the speaker.

Product engineering and model research reinforce each other

The learning approach behind this work started with understanding what a system does, then examining how it works and why its authors made particular assumptions. Those assumptions became candidates for improving speed or accuracy. A proposed use for Phind was to support that process as a technical research assistant: ask clarifying questions and help a learner formalize assumptions.

Phind’s product policy was answer quality first for demanding technical users. It ran GPT-4 or an equivalent by default, accepting slower responses to simple questions. Its model-agnostic interface could benefit as the underlying model improved, linking production engineering to applied model experiments. The hiring pitch sought people interested in both reliable user-facing software and experimental training work, without insisting on a rigid role title.

Correctness requires diagnosing the whole system

The shift from summarizing retrieved material to reasoning about users’ code happened faster than the team expected. Guaranteeing correctness remained unresolved: both the logic and the implementation have to work.

Grammar-constrained decoding was one exploratory direction: restrict generation so output conforms to a specified grammar. That concerns output structure; the discussion demonstrated no guarantee that the answer’s meaning or code behavior would be correct. A participant suggested LMQL, but its connection to the unidentified paper was unconfirmed, and the respondent had only skimmed the abstract. Phind described reinforcement learning for correctness as further along in exploration, without reporting a measured improvement.

The immediate engineering practice was more concrete: evaluate retrieved context separately from the answer. Phind measured whether answers were right or wrong rather than directly measuring a hallucination rate. Separating context quality from answer quality helped locate failures: - Bad context, bad answer: investigate retrieval and context selection as likely causes. - Good context, bad answer: investigate the model’s interpretation or conclusion. These cases called for different corrective steps.

Two further proposals grew out of internal development problems: - A structured evaluation platform: use GPT-4 to help make model evaluation more quantitative, potentially within LangChain. - A formal prompting language: provide an abstraction above natural language for the extension’s increasingly complex instructions, perhaps combined with grammar constraints. Both remained abstract ideas rather than demonstrated solutions.

The closing advice returns to the demo that started the project: pursue an idea you cannot stop thinking about. The founder argued that a personally meaningful technical possibility sustains daily commitment more readily than job boredom or a general desire to start a company. Phind’s longer-term ambition was to assemble people who believed that making software would increasingly center on human problem-solving, with AI taking on more implementation. Shared conviction, in that account, was what could carry the team through difficult periods.