tldraw: How a whiteboard became an AI input device

Two boxes and a good-looking arrow

Place two boxes anywhere on a screen and connect them with an arrow. Move them together, pull them apart, or make them overlap. What should a good-looking arrow do? Steve Ruiz developed Perfect Arrows by repeatedly testing those cases. Geometry supplied the tools; visual judgment supplied the requirements. GIFs of moving boxes and arrows made that process public.

That work eventually led to tldraw, a reusable whiteboard engine whose canvas could hold interactive web content as well as drawings. It became an unexpectedly useful AI interface: users could sketch a prototype, generate a working version, and draw corrections directly over the result.

Ruiz’s route into software began in studio art. He earned a master’s in visual art at the University of Chicago, moved to the UK, and around age 30 sought better income and closer collaboration. Experience building WordPress portfolios gave him a technical foothold. He specialized in prototyping interactions whose behavior could only be discovered by building them.

The code-oriented version of Framer, around 2017–2018, taught him programming and the distinction between exploratory and production code. A subsequent year doing education at Framer developed his ability to explain technical behavior through tutorials and short demonstrations. At Play, he worked on design tools that used actual platform interface components to bring prototypes closer to finished products.

From digital ink to a reusable canvas

Perfect Freehand tackled variable-width digital ink. A raster approach builds the appearance of a continuous stroke from tightly packed brush stamps, varying their size between sampled pointer positions. Ruiz wanted a geometric representation instead: take input points, return a polygon surrounding the stroke, then outline and fill it.

He could not find an available implementation that met his speed and consistency needs. Posting imperfect renderings attracted people who had attempted similar work and specialists in digital ink; their exchanges helped develop the library. At the interview, Ruiz reported its use in Canva, draw.io, Excalidraw, and tldraw.

Integrating the ink into Excalidraw and building tools to inspect it left Ruiz with reusable canvas code. His experimental tool Globs exposed the next limitation: it offered a workspace that users could pan and zoom, but could contain only globs. tldraw began as a general-purpose whiteboard engine, initially intended largely for himself.

The work extended beyond drawing shapes. Canvas products inherit expectations about selection, undo and redo, camera movement, and modifiers such as Option-drag to clone. Ruiz turned these individual behaviors into public demonstrations. By spring 2021, they were attracting recruiting interest from companies building creative software.

The demand was for infrastructure

The founder arranged to join Adobe after six more months on the project. Then an unnamed communications company needed to move its C++ touchscreen whiteboard into the browser for phone and laptop users. Without a team to rebuild it, the company paid $75,000 for source access. That funded the project’s first six months without making the founder its contractor.

Initially, tldraw operated as sponsorware: GitHub sponsors could use an early version. The November 2021 launch made the app, library, and canvas public under the MIT license. The founder reported roughly 40,000 launch-day visitors. The consequential surprise was demand from teams adding canvases to existing applications, including products that would never have built a whiteboard themselves.

Using React throughout let web teams customize the canvas with React components and integration code. The founder reported teams shipping production integrations within two or three weeks—an account of their experience, not a controlled productivity benchmark. The analogy was Mapbox or Google Maps: infrastructure that makes a valuable feature practical for teams unable to build the underlying system. Customer relationship management and sales workflows already improvised in Miro suggested possible applications.

Demand led the founder to decline the Adobe start, raise a seed round from Lux, Amplify, and angels, and assemble a London team around a whiteboard software development kit, or SDK, for other product teams. Sponsorship had reached about $200,000 cumulatively by fundraising; recurring sponsorship was separately estimated at about $5,000 per month. Companies were sponsoring partly because there was no purchasing path yet.

Make It Real turns output into input

Around OpenAI’s Dev Day, a Figma developer independently built a tldraw demo that converted a website sketch into a rendered page. The team expanded that idea into Make It Real, then released a tldraw starter kit for language-model integrations. An outside experiment had returned to the project as a new direction for development.

Make It Real was a thin, hastily assembled layer over the SDK. It sent a wireframe to GPT-4 with Vision, asking for one HTML file containing the markup, styles, and JavaScript for a working prototype. The prompt framed the model as a developer interpreting designers’ sketches, notes, and screenshots, following an earlier prompt credited to Sawyer Hood.

The canvas already used HTML and CSS, so it could contain playable web content and allow users to draw over it. Generated HTML returned in an iframe, an embedded web page that remained interactive on the canvas. In the color-slider demonstration, the presenter reported that the slider moved from orange to pink and displayed a hex code. Clicking the code showed a clipboard-copy confirmation.

That made the result itself the next prompt: 1. Annotate the generated interface with text, arrows, or a reference screenshot. 2. Select the result and its annotations together. 3. Generate again, with instructions explaining that the markings request changes to the previous output. This revision added missing labels and a full-width button to the slider while reportedly preserving its behavior.

Each request included an image and separate canvas text items to reduce dependence on visual text recognition. The presenter thought revisions also included the previous HTML, but was uncertain about that detail. Results could be opened through hosted, bookmarkable links or sent to StackBlitz and CodeSandbox for inspection and further development. Replit support was still forthcoming.

A timer separates appearance from behavior

A familiar application can make a visual demo misleading. A model may generate a calculator from its name alone, so success with an elaborate drawing does not establish what the drawing contributed. The kitchen-timer experiment compared four inputs: a name, an interface sketch, a state chart, and the sketch plus chart. Here, the state chart described stopped, running, and complete states and the button actions that moved between them.

The live results separated several kinds of success: - Name only: A box labeled “kitchen timer” produced a seconds-entry field and controls that could stop, resume, and reset the timer once a number was entered. - Interface sketch only: Without the application name or behavioral specification, the output resembled the drawing, but the controls did not work. - State chart only: Time-adjustment controls worked and the timer entered its running state, but counted upward. - Sketch plus state chart: The result more closely matched the intended interface, with working start, stop, and reset controls.

An annotated revision corrected the upward-counting version while preserving its appearance. The prompt supplied a simple convention for keeping instructions distinct from interface elements: red marks were annotations. Together, these examples showed how the canvas could carry both appearance and behavior, then localize a correction when one was wrong.

Iteration could also clarify the designer’s requirements. While specifying a drag-and-drop list, the presenter realized that part of the difficulty was not knowing how the interaction should work. The canvas provided somewhere to resolve that uncertainty. Nor did input need to originate as canvas shapes: an imperfect photograph of a paper wireframe, with a pen across one screen, reportedly yielded a working responsive website.

Keep alternatives and corrections in view

A canvas holding screenshots, text, websites, sticky notes, and drawings becomes a multimodal prompting workspace: a place to combine visual and textual instructions. It also supports parallel prompting, running several requests and arranging their results together. Image generation could branch into four variations, continue from a favorite, and retain the alternatives as visible history.

Spatial arrangement supplied lightweight organization. In the timer examples, inputs sat on the left, results on the right, and revisions farther right. That relationship was understandable without an explicit representation in the application’s data model.

The host raised a separate question: would a developer keep and extend the generated code? Layout problems made that uncertain. The demo also required a single artifact ready to run, so it did not support React output or multipage applications. Those were restrictions of this implementation, rather than requirements of visual prompting.

A proposed extension would provide an existing React application and an annotated screenshot, then request a diff across multiple files. That remained theoretical, with limits on how much input and output the model could handle. The deeper argument concerned expressing intent and correcting mistakes: users rarely supply every requirement initially, and incremental feedback can preserve useful work while moving toward the intended result.

Supply what an implementer would need

The presenter’s rule for choosing prompt materials was to imagine specifying work for a developer collaborating asynchronously. Screenshots of Stripe or Linear could communicate style; flowcharts could describe behavior. Pasted JSON could supply data, and annotations could ask generated code to fetch an endpoint and display its results. The endpoint needed no special registration inside the tool.

In the live dog-image example, the demonstrator drew an image area and button, then added instructions to fetch a dog from a pasted endpoint when clicked. The generated code reportedly called the correct endpoint, handled its JSON response, and displayed the image. No response schema was supplied. The participants suggested that familiarity with a common tutorial API might explain success; the example did not establish reliability with unfamiliar APIs.

This made the separate array of canvas text strings consequential. Small lettering can become hard to read after image processing; sending the strings alongside the image helped recover instructions without relying entirely on pixels. The same prompting approach suggested other output targets—configuration JSON, setup scripts, or SQL initialization from entity relationships—but these remained proposed extensions.

When the interaction is the product

Drawfast placed a generated image beside a drawing and refreshed it as the user moved shapes. The demonstrator described rapid feedback without a measured latency. Moving objects and adding highlights became ways to manipulate the generated image, and experimentation led into making stories.

The appeal did not depend on improving the original artwork. When a video transformed the demonstrator’s daughter’s mermaid drawing into a generic illustration, critics preferred the child’s drawing. The demonstrator agreed about its artistic interest while defending the transformation as play. The host suggested stop-motion-like animation, while noting that consistency still needed work.

Lens extended that interaction into a shared space. It built on Together, where collaborative drawings scroll upward and disappear, creating an ephemeral handwritten chat room. Lens captured a square from the center and sent it to an image-generation model with a scene prompt, such as a marketplace or futuristic cityscape. Shared marks continually changed the scene. Participants described roughly ten updates per second during the live demo, without formal benchmark conditions.

The model was referred to as LCM; its expanded name is unclear in the recording’s transcription. The implementation used Cloudflare Workers for updates, but screenshot capture and upload depended on a laptop in the speaker’s living room remaining on. Lens’s intended value lay in the aesthetic and social experience of changing images together, rather than retaining any particular frame.

Build the canvas for many products

The founder’s case for continuing the SDK centered on defensibility. A subscription app combining drawing and image generation was conceivable, but easy to replicate. The proposed durable asset was the extensible web canvas and its refined interactions: the accumulated behaviors that otherwise consume a team’s effort before it can explore its own product.

The preferred strategy supported many directions, including virtual collaborators and richer image interactions. The same canvas input could also be sent to different models for comparison. Make Real used OpenAI at the time; Gemini was discussed as a future alternative, without a comparison result.

At the interview, tldraw 2.0 was already in a long-running prerelease. Its planned launch primarily introduced commercial licenses for funded or revenue-generating products, with no substantial or breaking changes expected. The founder considered the software stable after extensive use and fixes; legal discussions still preceded the licensing change.

The technical distinction remained the use of ordinary React components, allowing interactive widgets and embedded pages to participate in the canvas. The founder called that decisive for many inbound users and described a friendly, reciprocal relationship with Excalidraw. The broader prediction was that collaborative whiteboards would spread through productivity applications much as Kanban boards had.

That view also shaped the discussion of AI inside design tools: assistance such as autocompleting a design could be more useful in Figma than generic image generation. tldraw’s ambition was broader still—a canvas as an input device enabling thousands of products beyond design, whiteboarding, or collaboration.

Curiosity, relationships, and a richer input box

The closing AI wish concerned relationships: help keep up with a growing network, remember birthdays, and recognize who could help with a current problem or benefit from a connection. The host proposed a Twitter-based relationship manager that could search conversations and shared interests before proactively suggesting connections. Lindy came up as an example of scheduling and research assistance covering part of that ambition.

The founder advice returned to studio art: treat curiosity as a prioritization signal. A strong desire to work on something may reflect accumulated observations about a field, conversations, and emerging needs. At tldraw, interest helped select code, features, partners, and problems. The qualification was to preserve that motivation while also working backward from desired outcomes.

A participant recalled writing a tldraw Substack post 72 hours into Make Real’s viral surge to preserve what the experience felt like while it was happening. The conversation then returned to a concrete interface idea: put a drawing surface inside a conversational input box, allowing users to sketch requirements or annotate photos before sending them.

A possible ChatGPT integration remained an aspiration. The proposed interaction was already clear: give people somewhere to express visual intent, inspect the response, and mark up what should change next.