Playground’s Bet: Build the Model and the Graphics Editor Together

From traffic alerts to a different kind of computer

A sudden traffic spike in one region is easy to miss when a product generates more data than anyone can inspect. Suhail Doshi recalls Mixpanel building a machine-learning feature to flag such surprises, alongside predictions of user churn and conversion. Around 2015 or 2016—he is unsure of the year—the team explored what its accumulated data could predict. Most of the work, he says, came down to logistic regression. It did not feel transformative, and the team stayed at four or five people.

Doshi, the former Mixpanel co-founder and CEO and founder of Playground AI, began studying fast.ai material during that period and later revisited it several times. His next company, Mighty, pursued a computer streamed from a data center. It started with the browser: as more applications moved there, relocating it seemed a way to free users from managing RAM, disk space, hardware upgrades, and data locality.

But moving execution did not remove its bottleneck. In Doshi’s assessment, too much web-app work still depended on single-threaded JavaScript and the speed of an individual CPU core. More data-center resources could not deliver the new kind of computer he wanted. He thought an enterprise business might remain viable, but abandoned Mighty when he stopped believing in that larger ambition.

AI renewed the idea because models offered workloads designed for massively parallel computation. Doshi imagined remote models turning additional compute into capabilities a local machine could not supply. He expected local and remote models to coexist, with deployment depending on the task: a requirement for 30 or 60 frames per second could make network execution difficult. Mighty’s reported achievement of 30-frame-per-second video streaming made him optimistic, though he supplied no latency measurements or network conditions.

Graphics offered a way beyond the prompt box

The transition began with an address-bar experiment in Mighty’s Chromium fork: could a model predict where a user wanted to go? Discussions with OpenAI explored GPT-3 and GPT-3.5, with low latency central to the idea. Meanwhile, DALL-E 2 and Stable Diffusion made image generation compelling enough to reconsider the business. The participants remembered their impact differently: one emphasized DALL-E 2’s viral images, while another found Stable Diffusion more personally significant.

The search ranged across hosted model APIs, training, optimization, and analysis. OpenAI’s pre-ChatGPT playground suggested another opening. Its text box and generation settings appealed to engineers but demanded considerable understanding from users. Early product ideas included prompt organization and libraries. Graphics supplied a stronger thesis: visual work should support visual interaction, with controls that could grow beyond text entry.

The initial market was described as largely hobbyist, attracted to a novel toy. The commercial thesis depended on increasing utility: if the tool could perform demanding graphics work, businesses would have reasons to adopt it. Mixpanel’s movement from smaller customers toward enterprise supplied an analogy. Creative work also supplied personal motivation—the combination of AI and an artistic tool was something worth staying up until three in the morning to build.

Playground v2: improve the model, share the starting points

[Playground v2](https://blog.playgroundai.com/playground-v2/) had been available for only two days when the conversation took place. The rationale for training from scratch began with a product limitation: Stable Diffusion could produce enjoyable artwork while still getting hands and eyes wrong. Waiting for external progress meant waiting for more useful workflows. The respondent compared image models to an early GPT-2 or GPT-3 moment—exciting generation capabilities whose business utility still needed to develop.

The release also contributed to a community praised for quickly turning research into working features. Playground reported sharing weights from both 256-pixel and 512-pixel pretraining stages to help academic groups with limited GPU access. The releases served different purposes: - `base` models: pretrained starting points for further research. - The [`aesthetic` model](https://huggingface.co/playgroundai/playground-v2-1024px-aesthetic?ref=blog.playgroundai.com): the version aligned after pretraining to produce attractive images.

The desired research extended beyond better individual samples: - Character consistency: preserve a person’s facial identity when placing them in another image. - Coherent editing: support multiple editing tasks, rotations, and changes across images without losing coherence. - Style and image understanding: improve style transfer and reasoning about image contents. Instruct Pix2Pix and EmuEdit were cited as promising work, with EmuEdit described at the time as a Meta paper without an available model.

The explanation for v2’s improvements emphasized data cleaning and systems engineering: better captions, dataset selection, filtering, and the data used for alignment after pretraining. An interviewer cited a 2.5× preference over Stable Diffusion XL on a 1,000-prompt benchmark, but did not establish the rating protocol or define the ratio. The explanation that followed described incremental improvements across the training pipeline.

A tentative v2.1 plan targeted late 2023 or early 2024, using community complaints to prioritize repairs. Blurry or misshapen eyes were already a target, with better data proposed as the remedy. Changes to the noise used in training were also candidates for improving color dynamic range; these were proposed experiments, not reported gains.

A unified model and an evolving definition of quality

An interviewer raised a practical alternative: generate an image, then repair its face, eyes, or hands in separate stages. The respondent favored a unified model over an expanding repair pipeline, arguing that specialized corrective models add computation and generalize poorly. ControlNet tiling—used here to generate higher-resolution versions of image regions—was acknowledged as an existing workflow but regarded as a temporary workaround. The analogy was to language models growing from narrow systems into models capable of many tasks.

That preference did not settle the architecture. Image compression was raised as a possible information bottleneck, with pixel-based generation and vision transformers among the alternatives worth exploring. These remained open questions; no architectural replacement was shown to solve fine-detail failures.

[MJHQ-30K](https://huggingface.co/datasets/playgroundai/MJHQ-30K?ref=blog.playgroundai.com) addressed another problem: automatically evaluating aesthetic quality, using a dataset the respondent confirmed came from Midjourney. The rationale was to compare against the strongest available product, even when Playground lost. Midjourney was the speaker’s preferred reference at the time; beating an older model mattered less than approaching the alternative users wanted.

Aesthetic quality was also a moving target. Community experimentation could uncover unexpected styles, sometimes after a model had circulated for about a week. Pixel art and photorealism had familiar labels; names such as ProtoVision and DreamShaper identified qualities harder to describe. Like musical genres, visual styles could split into subgenres or become tiresome through repetition.

One interlocutor wondered whether businesses would eventually need less stylistic diversity than consumers. The response accepted that possibility while defending curiosity-driven experiments. Voluntary hobbyist use signaled attraction before a monetization path was clear. Starting with a toy could reveal opportunities that a narrow assessment of immediate business utility would miss.

The holiday-card test: does the edit actually work?

A holiday photograph made the gap between generation and useful editing concrete. The desired operation was easy to express: select a child’s mouth and make the child smile while preserving the photograph’s fidelity and coherence. This remained an unmet quality requirement. The respondent described repeated attempts producing crooked features or artifacts, attributing part of the difficulty to the mouth occupying a small region with little image information.

A proposed workaround was to enlarge the region, identify it as a mouth, edit it, and shrink it again. The extra manipulation exposed the product problem: users were wrestling with the model. The team could develop visual judgment through sustained exposure, but expressing that judgment in words remained difficult.

Playground’s initial evaluation used roughly 1,000 difficult prompts, some written by the team and others generated, followed by user assessment. A horse riding an astronaut tested an unusual relationship; misspelled words tested recovery of intended meaning. These prompts exposed an evaluation trap: when every model fails the task, preference can reward the prettiest failure.

PartiPrompts supplied a common comparison set for academic researchers. The broader goal was a benchmark that tracked user satisfaction and real-world performance. The proposed feedback loop was: 1. Observe failures in images users actually make. 2. Identify attributes to measure, such as smiles, lighting, or color contrast. 3. Occasionally ask users to assess those attributes after generation. 4. Validate that the measurements approximate real-world performance before publishing a reusable benchmark.

Users were reportedly generating millions of images daily. The speaker estimated that occasional feedback requests could yield tens of thousands of evaluations per day, but this was prospective. Publishing the validated benchmark remained an intention for the following year.

Small text, compatible tools, and safe browsing

Playground v2 prioritized image quality over text synthesis—rendering writing inside an image. The respondent praised Ideogram subjectively and described DALL-E’s text rendering as inconsistent. The desired advance went beyond decorative lettering: tiny, readable text on a milk carton outside the scene’s focal point. An interviewer suggested directing the system to that region instead of requiring success in a single generation; the respondent agreed without specifying an implementation.

Community experiments supplied inspiration, including logo masks mapped onto food or vegetables so the imagery read as text. Playground wanted a unified model with comparable abilities, while preserving access to existing tools. V2 retained Stable Diffusion XL’s architecture to avoid fragmenting the ecosystem. Adapting ControlNet was described as potentially requiring a small code change and retraining a ControlNet for the model; shared architecture did not imply automatic compatibility with existing trained weights.

Safety involved several distinct responsibilities: - Training-data filtering: the team reported excluding images classified as not safe for work to reduce the burden on downstream filters. - Capability misuse: combining nude generation with consistent facial identity raised a specific abuse concern. - Browsable content: artistic legitimacy and individual taste did not settle what belonged in a feed used by classrooms of children, even when images contained no nudity. A recent safety sprint had addressed these problems, but moderation still required judgments beyond detecting nudity.

Make generation less blind without removing control

The interviewer described controls for choosing a seed to make generation partly repeatable, setting image counts, and retaining generated images on a canvas. A generation box could span images for outpainting, extending the generated area. The larger ambition combined a graphics foundation model with an editor designed around its capabilities. Photoshop supplied a familiar reference for power and configurability.

The respondent compared prompt boxes to terminals: conventions such as parentheses for weighting words required users to learn syntax just to make a picture. Selecting headphones and asking for their removal, or expanding an image with less typing, illustrated a more visual direction. Playground’s name itself paid homage to GPT-3’s playground, the interface that helped inspire the company.

Intuitive interaction still needed expert control. Prompt expansion, which rewrites a user’s prompt, remained optional because some users wanted help while experienced users preferred their own wording. Playground accepted a potentially narrower audience in exchange for power and configurability. The interface was still evolving: better models could make an interaction that felt appropriate six months earlier feel broken.

Waiting for generation could be worthwhile when users could not otherwise create the image. Repeated blind sampling imposed a different cost. Preview rendering showed a rough image before the final generation, using consistency models within the familiar prompt-to-image workflow. When the user proceeded, the system tried to keep the final image coherent with the preview. Borrowing the idea of viewport rendering, the feature aimed to reduce repeated guesses; no measured reduction in retries was reported.

A mood board could express what a prompt cannot

LoRAs entered the discussion as a scalable way to customize a model’s artistic style, compared with heavier DreamBooth fine-tuning. The product argument was that users needed ways to request originality beyond naming an artist. In the respondent’s interpretation, artist names often substituted for visual qualities users recognized but could not otherwise request.

Training such customizations still demanded technical skill. The proposed alternative was a mood board as the style interface: supply five or six reference images and have the model infer the intended style, even when the images differed substantially. Combining a Tron image with several other influences illustrated why a broad label such as cyberpunk might miss the desired subgenre. This remained a proposed capability.

Playground’s feed helped users build a visual vocabulary through informal daily competition, likes, encouragement, and reusable prompts. Requests for prompting lessons were interpreted as requests for better composition, lighting, and aesthetics. Prompt sharing might become less central as tools grew more complex, but teaching users and celebrating their art remained reasons for the feed.

Affordable inference, fragile training

Playground ran its own infrastructure primarily to control compute costs and keep the product affordable. The respondent distinguished three engineering problems: - Inference operations: running models for users felt comparable to managing a large API server fleet. - Model performance optimization: making the models run efficiently was a newer and harder discipline. - Distributed training: scaling a training cluster was substantially harder than expected because a failed node could crash the entire run. The reported stack used Slurm for scheduling, PyTorch, and PyTorch Lightning. Resilience to node failure was a central scaling problem.

Follow the project far enough to learn what it needs

The closing research question widened the scope beyond graphics: might AI have more useful modalities—forms of information—than familiar categories such as vision, language, and audio suggest? A physics foundation model representing Earth’s fluids and physical interactions was offered as a difficult possibility. The discussion questioned whether thinking in words and tokens constrained the imagination, while accepting that tokens and bytes were ultimately units of information. Mind uploading and imperfect models of human consciousness followed as speculative interests.

The practical learning advice was to work through Karpathy’s classes, identified in the conversation as Zero to Hero, and write code to cement understanding. For self-taught founders, the recommendation was to choose a project exciting enough to sustain the work and learn its details while building. Learn model training when the project requires it; otherwise, go as deep as necessary. Sustained motivation mattered more than prescribing the same educational path to everyone.

The recording closes with Doshi’s own creative work. In the editing-room outro, Swyx introduces a track Doshi shared, “It All Started With a Feeling,” as the episode’s closing music.