From Borrowed GPUs to Lexica: Making AI Capabilities Useful

A chatbot needs a GPU

Sharif Shameem wanted to train a chatbot on his iMessage history. The messages were available in a SQL database; he needed a GPU to train a character-level recurrent neural network. A friend’s NVIDIA 1080 Ti was earning roughly $5–6 a day mining Ethereum. Shameem offered $10 a day for a week of access. The model was poor, but the rental worked: he obtained the compute he needed, and his friend earned more than mining paid.

The experiment grew out of his computer vision work at MITRE. His broader career path included leaving the University of Maryland without graduating because a side project consumed his time. A future investor’s experience of leaving school and later returning helped him see that choice as reversible; Bryan Caplan’s The Case Against Education also influenced him. At the interview, he described himself as on indefinite leave.

VectorDash turned miners’ GPUs into a rental marketplace for machine learning researchers. Shameem recalls launching within months and attracting more than 120,000 NVIDIA GPUs to the platform, though that count does not establish simultaneous availability or utilization. After the Ethereum crash, he says usable GPUs rented for about $0.20 an hour while still paying miners more than mining would earn.

Cheap compute did not solve the customer problem: - Students and researchers had little money, even at attractive prices. - Startups had cloud credits, weakening the appeal of a cheaper provider. - Larger companies distrusted the platform with their data, making trust more consequential than price.

VectorDash pivoted to streaming users’ games from distributed GPUs to Macs and PCs. Hosts sometimes sat only blocks from players, which Shameem credited with unusually low latency. But upload bandwidth stranded the compute. He estimates typical US upload capacity at roughly 35 Mbps and a 4K stream’s requirement at 15–20 Mbps, recalling that a host with 60–100 GPUs could consequently use only about one for streaming. These were operating recollections, not a measured utilization benchmark; the bottleneck was that more GPUs did not bring more network capacity.

Make generated code visible

After VectorDash, another chatbot drew Shameem toward GPT-3. It would ask about goals during the week and turn informal text replies into a table tracking activities such as running or calorie intake. He wanted a language model to make the largely preprogrammed interaction feel more natural. An extravagant capability claim on YC’s internal forum prompted him to seek access; he says Greg Brockman responded to his public request within hours, and he was experimenting that afternoon.

A simulated conversation with Steve Jobs seemed somewhat plausible, but code completion made the stronger impression. Shameem supplied JavaScript function headers, copied the generated implementations, and found that examples he ran worked. His informal boundary was logic that fit into one sentence: requests whose full logic could not be expressed that compactly exposed more failures.

He consolidated his testing loop—generate in the playground, copy into VS Code, run, and reload a React page—into one browser page. A prompt triggered generation, compilation, and rendering. For very basic JSX React components, the output became an inspectable interface. Showing code turn into a usable result also made the capability easier to communicate.

After an overnight build, Shameem posted a roughly 30-second demo. He recalls waking to about 20,000 likes and 100,000 viewers, then publishing more demos over the next week. The API had already existed for roughly a month, he recalls, though access remained restricted. Rendering the result made that existing capability tangible.

Lexica: search before generation

Lexica began as a better way to browse Stable Diffusion images and learn prompts. Discord search showed roughly five images at a time. Lexica’s builder reports scraping about ten million images into a database and building the website in about two days, launching roughly a day after Stable Diffusion’s model weights became available.

The gallery made exploration accessible through a shareable URL; running the model locally required a GPU and environment setup. An interviewer offers two further explanations: search avoided the generation wait, and people already knew how to search. The builder agrees that a one-word search could retrieve better-looking images than generating from that word alone.

The original search used Postgres full-text search over prompts. Inspired by the visual search engine Same Energy, the builder added CLIP-based semantic search: CLIP maps images and text into a shared representation, allowing the system to compare their similarity. Lexica indexed those image representations, or embeddings, and retrieved nearby matches. Search could now follow image content beyond the words in its original prompt.

When the embedding collection outgrew the regular database approach, the index moved to FAISS, a library for nearest-neighbor search. GPUs computed the embeddings, while CPUs served searches. An eight-core server was described as fast enough, without latency or throughput measurements. The builder reports substantially greater search usage after the change; the same index also let users upload an image and retrieve visually related results.

Adoption changed the company’s priorities. The builder had set one million users as the threshold for considering full-time work, expecting it to take years. Lexica reportedly reached it in a week and a half. The company pivoted from Debuild, whose growth was described as slow and linear, and raised about $5 million from Daniel Gross, separately participating in AI Grant. By the conversation, Lexica’s builder estimated more than five billion images served monthly—deliveries to users, not newly generated images.

The next feature came from watching users switch between Lexica and a separate Stable Diffusion interface: find an image, copy its prompt, paste it into the generator, and change a word or two. The builder experienced the same frustration. Integrated generation removed the handoff between discovery and creation, and users reportedly welcomed it.

User preferences shape the training distribution

The lesson drawn from AI Grant was a gap between available capabilities and useful products. The proposed priority was to build compelling applications and get people using them, with less emphasis on leading evaluation scores. A participant connects this to research incentives: familiar benchmarks can direct attention and funding away from work serving different goals.

For Lexica’s diffusion models, the training explanation centers on selecting the distribution the model learns—the kinds of images represented in its training data. The speaker emphasizes data quantity, quality, and filtering: training on a better selection should teach the model to produce images resembling that selection. Much of the work, in this account, lies in choosing data rather than changing the architecture.

Lexica’s traffic supplied preference signals. If someone liked one image while another appeared beside it, the system could infer that they probably preferred the first. Those pairwise comparisons supported rankings, including Elo scores. Aesthetic models—models that predict image preferences—helped filter larger training datasets. Possible objectives included predicting whether someone would like an image or assigning a score from one to ten.

The discussion treats large-scale training as learnable by capable programmers, while acknowledging a slower start. Practical resources include targeted searches, Stack Overflow, relevant GitHub issues, fast.ai, the Deep Learning book, and Andrej Karpathy’s courses. Open collectives such as Eleuther and LAION are credited with making their training runs easier to learn from by sharing details.

Estimating substantial costs before pressing Run creates a less familiar burden for software engineers. The response favors fine-tuning existing models, adapting them to a task with additional examples. Low-rank adaptation is mentioned as a way to make that process practical on one GPU. At Debuild, the team explored collecting examples and feedback from its GPT-3-powered product and using them to adapt smaller open-source models to the same domain.

The team tried GPT-J, but the speaker recalls Flan-T5 working best for that use case, provided prompts were small enough. The broader claim was approximately state-of-the-art domain performance at ten to one hundred times lower runtime cost than an API. Those remain informal estimates without a specified workload, benchmark, model size, or cost-accounting basis.

Browser agents need context and precise controls

Debuild’s browser-agent exploration began with a concrete workflow: a user would supply an API documentation link and describe a desired call. The agent would browse and summarize the documentation, potentially also creating and registering an API key. That required both a readable page representation and a way to act on the page.

The early demo worked around GPT-3’s then-2,048-token context limit—the amount of text available to the model at once—with a repeated procedure: 1. Serialize the webpage into a compact representation. 2. Split pages that remained too long into chunks. 3. Use sub-prompts to evaluate actions within those chunks. 4. Have a top-level node select an action, execute it, and repeat. The speaker estimates that a sixty-second demonstration cost roughly $100. Repeated inference and the small context window made the approach impractical at the time.

GPT-4 renewed interest through greater capability and more context: the conversation describes 8,000 tokens as available and 32,000 as forthcoming. But more space did not eliminate the choice of what to preserve: - All page text could exceed the context window. - Only buttons and inputs saved space but discarded information needed to choose an action. - A terminal browser’s text frame preserved readable text while allowing graphical character blocks to be removed; it was described as promising without a quantified evaluation. - Chrome’s accessibility tree, the representation used by screen readers, could help but often lacked useful annotations.

Choosing a control creates another ambiguity. A calendar day labeled 30 might be easy to match, but a friend list can contain many identical Delete buttons. The action target must stay connected to its surrounding content. The proposed interface borrows from Vimium: pressing F overlays short character labels on interactive elements, and entering a label activates the corresponding control. Combining those addresses with readable page content could give the model context and precise control.

One participant argues that reasoning is already sufficient and that representation and control are the main remaining engineering problems. Knowing when to ask for help remains a separate behavior to teach. The proposed approach collects browsing trajectories, finds where agents went wrong, replaces those steps with requests for assistance, and fine-tunes on the edited examples. For a closed model, an instruction to ask when uncertain is suggested instead. Neither approach is evaluated in the exchange.

Vision could preserve what text loses

A Yelp map makes the representation problem concrete: its spatial context is difficult to serialize into text. Participants expect multimodal input—here, images alongside text—to help with graphics-heavy pages. That is an anticipated benefit, not a demonstrated browser-agent result.

The discussion distinguishes similarity from visual reasoning. CLIP supplies a shared representation useful for retrieval and clustering. Mini-GPT4 enters as an example of connecting vision and language models to answer questions about images; participants informally describe it as combining BLIP and LLaMA. This raises an unresolved question: what does joint multimodal training provide beyond connecting separately trained components?

A tentative recollection of vision-enabled GPT-4 outperforming text-only GPT-4 on unspecified tests prompts speculation about learning extra structure from images. Another participant offers a simpler explanation: a test contained a chart. The first accepts it. Asking about an image of a pipe or requesting a bug fix from a VS Code screenshot are proposed as further probes, rather than reported successes.

Ship early enough to discover the wrong objective

An interviewer turns to Sharif’s Startup Manual and its advice about shipping roughly every other day. The response emphasizes the smallest application that is already useful. Lexica’s first version is recalled as one React screen backed by Postgres, showing a randomized image grid; the speaker is unsure whether that very first version included search. Its useful job was helping someone write better prompts.

The development advice separates three decisions: - Launch when the application becomes useful. Actual users can shape the roadmap; the first release starts that conversation. - Look for spontaneous recommendations before focusing on growth. Users telling others to try the product is proposed as evidence that they love it. - Do not require the idea to sound like a business immediately. Typeform illustrates how an apparently modest utility—attractive web forms—can support a substantial company. The speaker believes two extra months of feature development would have cost Lexica much of its launch momentum. That is a counterfactual judgment, rather than a measured comparison.

Aperture V1 showed that photorealism was the wrong objective for part of the audience. Its marketing tweets attracted attention, but the image model proved less popular than expected. Some users preferred Stable Diffusion 1.5 because their work involved art and unusual abstract images that fit it better.

Prompt following posed a distinct technical problem. The speaker attributes much of Stable Diffusion’s difficulty with complex requests to its CLIP text encoder, which seemed to steer image sampling broadly without capturing more complicated intent. T5 is proposed as a better encoder, but no benchmark accompanies the comparison.

Aperture V2 changed the data selection to better represent art. Images from Lexica helped train aesthetic models that ranked artwork; those models then filtered larger datasets for training. V3 extended the approach with substantially more data. V1 had taken about a month, and the speaker regarded its lessons as useful for subsequent versions.

Numbered releases generally corresponded to different base models, while some fine-tunes shipped without a version increment. The proposed rule was behavioral: a clear change in what the same prompt produces should warrant a new version. The speaker acknowledges that this informal scheme does not fit conventional semantic versioning neatly.

Unusual ideas become tangible through demos

Asked what else would be worth building, a participant proposes tools whose users are language models: browser access, payments, credit cards, and control of physical robots. A Boston Dynamics Spot collecting an order from high-level instructions is the imagined example.

A detour into consumer genomics gives the episode its title. A participant describes exporting 23andMe data to an interpretation service that links genetic variants to published studies, then finding reported associations with late sleep timing, risk-taking, and optimism personally explanatory. Another participant’s “founder gene” response is a joke, not an established genetic finding. No specific variants or studies accompany the anecdote.

The conversation returns to an actionable way of encouraging ambition: demonstrate a possibility so others can pursue it. BabyAGI and other GPT agent projects are described as limited in practical usefulness at the time but effective at inspiring experimentation. A Stanford demonstration with a miniature Sims-like environment serves a similar role; one participant recalls agents visiting bars in the afternoon despite instructions against it.

Games make the imagined consequence vivid. A participant describes non-player characters in The Last of Us Part II referring to one another by name and pleading for their lives, characterizing that behavior as conditional rules. Participants imagine language-model-driven characters with persistent lives, individual personalities, and open-ended conversation. More convincing agency could make encounters more affecting—and games harder to put down.

Choose the audience, then remove the friction

In the lightning round, one participant names ChatGPT as their most-used AI product and Cursor’s integrated code iteration as the product they are most excited about: describe a change in natural language and work on it directly in the editor. The enthusiasm is prospective. They had used Cursor only a little, had not fully switched, and made a possible switch conditional on continued improvement.

The advice for coding products is to choose a specific type of code and a clear audience: - A low-code tool should avoid concentrating too heavily on attracting developers. - A VS Code alternative should focus entirely on developers.

Another participant describes a smaller integration: a forked ChatGPT menu bar app, kept live and opened with Command+Shift+G for quick queries. The immediacy comes from opening an already-running interface, not faster model inference. Like the earlier browser-based code demo, it removes steps between an intention and the useful interaction.

Text-to-video enters as a forecast: a participant expects a quality leap during 2023 comparable to the jump from DALL-E 1 to Midjourney, while explicitly acknowledging that it has not happened yet. GPT-4’s visual capabilities are the surprise they regard as already present. Aperture’s interest in video remains exploratory, rather than a product commitment.

Agents that can finish administrative work

The closing product request returns to browser agents, now framed around willingness to pay. A participant wants to assign a task and launch many agents against it, tolerating failures if some produce useful outcomes. The example of 80 failures and 20 partial successes among 100 agents is hypothetical. The desired work is concrete: register for a software service, book a call, or handle an email thread to obtain access without making the user attend a demo.

The proposed system needs write access to the world, with distinct capabilities: - Browser control: use graphical interfaces and register accounts. - Communication: maintain an inbox, send useful messages, and ask other people for help. - Payments: use a credit card with an assigned balance. - Escalation: return to the user when it gets stuck. The participant contrasts this ambition with their impression that ChatGPT plugins then appeared read-only, and speculates that cards for GPT agents could become a service within a year.

The final advice is to build and share unusual ideas instead of assuming someone has already tried them. The speaker calls for more strange, interesting products and fewer variations on GPT-powered documentation search. A compelling demo can make a possibility visible; a usable product lets other people experience it.