ComfyUI: Engineering the Workflow Behind AI Art

The second pass that needed a different model

ComfyUI began with a concrete problem: generate a small image, upscale it, then refine it with a different model. Comfyanonymous was experimenting with the high-resolution fix in AUTOMATIC1111, a workaround for diffusion models’ limited generation resolution at the time. He edited its code to change the second pass’s sampling method, settings and step count. Switching models went further: he judged that building his own interface would be easier than extending the existing one.

ComfyUI made these stages expressible as connected nodes on a visual canvas. Users could chain models and control individual operations, extending image generation into complex, sometimes parallel image and video workflows. Model chaining was one of its earliest capabilities.

Comfyanonymous discovered Stable Diffusion in October 2022, with experience in software engineering, Python and basic automation, but none writing PyTorch code. Node interfaces were also new to him; he chose one because it seemed the clearest way to represent the diffusion process. He began coding on January 1, 2023, and released the first version on January 16. The name came from people describing his generated pictures as comfy.

A mountain here, a fox there

Area conditioning—applying different prompts to different image regions—brought early attention on Reddit. Comfyanonymous’s example was compositional: put a mountain in one place and a fox in another. Precise placement remained difficult, but the workflow gave separate instructions to separate parts of the image.

The mechanism works at every diffusion step. It evaluates each region with its own prompt, evaluates the whole image with another prompt, then averages the results. Regional instructions therefore shape generation throughout the process. Comfyanonymous recalls MultiDiffusion appearing roughly a month later with the same idea.

Using different models within area composition requires a shared latent space, the representation in which the diffusion computation operates. Different SD 1.5 models are offered as compatible examples; SDXL and SD 1.5 have incompatible latent spaces for this operation. Comfyanonymous had not implemented that multiple-model variant. He also credits latent-space computation with Stable Diffusion’s popularity, describing computation directly in pixel space as too slow.

Model chaining also mattered to SDXL’s developers. In the account given here, Stability hired the tool’s developer in June 2023 to help experiment with its base and refiner models. The base was trained normally; the refiner was trained on lower timesteps. Internal tests reportedly found better image quality when they were chained, though no evaluation method or numerical result is given. The refiner was a full diffusion model that could also generate images independently.

Choosing models means choosing an aesthetic

The model recommendations distinguish creative variation from consistency: try SD 3.5 for the former and Flux for the latter. These are qualitative preferences, not benchmark rankings. ComfyUI also supports closed-source model APIs through custom nodes—extensions that add operations to a workflow—including some official provider integrations. Ideogram is one example affirmed in the conversation.

New releases do not automatically replace older models. SD 1.5 is described as retaining many users and SDXL as another major choice, while SD 2 offered too little improvement to attract broad adoption. Stable Cascade receives a different assessment: a worthwhile improvement over SDXL whose momentum suffered when SD 3 was announced roughly a week later. The speaker tentatively recalls a delay for red teaming and speculates that an earlier release could have improved its reception.

The participants disagree about adoption: one sees users abandoning models before exploring their capabilities; another sees them switching only for substantial improvements. Migration also depends on the workflow. Many graphs may remain broadly compatible across models, but the prompt may need a complete rewrite. Preserving the operations does not necessarily preserve the instructions that produced the desired image.

Evaluation reflects the artistic task. ComfyUI users are described as generating images and judging which look good, with more systematic checking concentrated on model development. Technical capability and consistency do not guarantee aesthetic appeal: the argument is that indiscriminate training on internet images would reproduce many qualities users do not want in generated art.

Prompt embeddings, weighting and model adaptation

Textual inversion customizes generation by learning a new prompt embedding from example images. A prompt’s tokens become vectors, which pass through an attention-based text encoder. Textual inversion learns another such vector—effectively a new word—to steer generation toward the concept represented by those images.

One participant reports successful internal textual-inversion experiments with the T5 XXL text encoder at Stability. Transfer to other image models using T5 XXL remains an untested possibility. A narrower precedent illustrates the compatibility issue: SD 1.5 embeddings are said to work more weakly in SDXL because only one of its two text encoders shares SD 1.5’s CLIP L encoder. Their influence is further diluted in SD3’s three-encoder arrangement.

The image model is trained to work with its text encoder, so an arbitrary encoder replacement is constrained. LongCLIP is a compatible extension discussed here: a CLIP fine-tune for longer prompts, supported in core ComfyUI at the time. Its context length is given as 256 tokens, compared with regular CLIP’s 77.

A separate workaround for long SD 1.5 prompts splits the input into chunks, encodes each chunk, then concatenates the outputs. It is functional but imperfect. Its practical justification is that users expect concepts near the end of a long prompt to appear, even when detailed adherence to the entire prompt is weaker.

Prompt weighting modifies the encoded representation. ComfyUI’s described method obtains CLIP outputs for an empty prompt and the actual prompt, then interpolates between them according to token weights. The explanation attributes its effectiveness in SD 1.5 to CLIP L’s shallow architecture: input-token positions retain a close relationship with output concepts. This method is said to lose effectiveness with deeper encoders and fail on T5 XXL. The proposed explanation for few Flux complaints is tentative: users may compensate with richer natural-language descriptions.

LoRA instead learns a compact change to model weights. It trains two smaller, low-rank matrices whose product supplies the weight update, reducing the training burden relative to full-model fine-tuning and making adaptations convenient to share. LoHA and LoCon are described as alternative ways to represent learned weight differences.

ComfyUI applies the LoRA update directly to model weights before sampling. The reported cost is a small initial delay to apply the update, after which sampling runs at the original speed. No timing measurements accompany that implementation account.

The canvas rests on a local inference backend

ComfyUI deliberately prioritized power while other projects pursued ease of use. Most early engineering effort went into making local execution work well. Its listed capabilities included re-executing changed portions of a workflow, an asynchronous queue and memory management; the frontend received greater attention later.

The original canvas used an existing JavaScript node-interface library to minimize frontend work. The architectural preference was a defined API between frontend and backend. The critique of Gradio is that it encourages interface and backend logic to become entangled, making maintenance harder. It is still recommended for quickly demonstrating existing machine-learning code, and another participant notes that the architectural judgment is contested. A similar impression of Streamlit rests on only brief familiarity.

GPU memory management is identified as the hardest backend problem. Initially, ComfyUI fully unloaded one model before loading another. That worked for small models, but repeated transfers became costly as models grew. The replacement strategy estimates the next sampling operation’s memory needs, evicts enough loaded models to make room, and preserves the rest in GPU memory to avoid unnecessary reloads.

Keeping more data resident introduces another boundary. The Windows NVIDIA driver is reported to page overflowing GPU allocations into system RAM, allowing a workload to continue while becoming extremely slow. Paging is one possible explanation for sudden slowdowns. The objective is using GPU memory fully without triggering paging or allocation failure. PyTorch’s high-level abstractions limit precise control, and memory freeing remains delegated to Python and PyTorch.

Hardware support depends on the surrounding software stack. CPU execution is straightforward through PyTorch. At the time of the conversation, AMD is described as usable on Linux but substantially harder to use on Windows, with a good Windows PyTorch build for AMD’s ROCm stack identified as the missing piece. The comparison with similarly priced NVIDIA hardware is qualitative, without configurations or benchmarks.

Expose the controls, then learn their effects

ComfyUI exposes sampling at different levels of complexity. Four sampler nodes are described, from a regular sampler with basic settings to an advanced custom sampler with separately exposed components. Users can choose how much of the generation process to control directly.

Two controls offer practical starting points: - Sampling steps: reduce the count until image quality deteriorates beyond what the task tolerates. Fewer steps mean fewer iterations of the diffusion process. - `CFG`: this guidance control scales the difference between positive and negative predictions. Contrast is offered as an intuition: lowering it may help when an image looks burnt out. The recommendation is to change parameters and inspect their effects. Knowing the formula alone does not establish which setting produces the desired image.

Setting CFG to 1.0 is said to disable the negative prompt and make sampling twice as fast. The speed claim comes without benchmark or implementation conditions, so it is not a general timing guarantee.

Custom nodes and video extend the workflow

ComfyUI’s backend can serve other interfaces. A Krita plugin brings compatible models into Krita. A recalled community project put a Wolfenstein 3D-style game inside ComfyUI, accepting generated textures as inputs that replaced textures in the game. Both connect generation to a larger creative process.

A YouTube download node illustrates the broader pipeline: ingest data, apply filters and generate outputs. Easy custom-node development helped the ecosystem grow, but also produced overlapping node packs. Discovery required manual work: the ComfyUI Manager maintainer reportedly searched GitHub daily and added nodes by hand. The Comfy Registry was intended to formalize discovery and reduce that burden.

Video adds a representation choice. Stable Video Diffusion is described as adapting SD2 with temporal attention and video training while retaining 2D latents, an approach compared with AnimateDiff. Both compress spatially but not temporally. Mochi instead uses 3D latents and a temporal compression component, called a temporal VAE, that also compresses along time. The distinction concerns what the representation compresses; all these approaches generate video.

Mochi was implemented in ComfyUI because it seemed to its implementer the best available video model at that point. CogVideo also received a favorable assessment. These were qualitative judgments rather than results from a common benchmark.

From an experimental tool to an ecosystem

The closing discussion returns to ComfyUI’s growth. After the January 2023 release and area-composition Reddit post, a March video by a YouTuber identified as Olivio is credited with the first substantial burst of attention. Adoption changed the work: the tool built for experiments left less time for experimentation, as user-facing features took priority.

SDXL then made local execution a distribution advantage. In the developer’s account, Stability released code before the checkpoint, allowing ComfyUI support to appear early. SDXL 0.9 checkpoints circulated through signup-based early access and subsequently leaked. Their use in ComfyUI produced feedback and fixes before SDXL 1.0 arrived.

The developer attributes the resulting growth to making SDXL usable on ordinary GPUs, describing the competing AUTOMATIC1111 implementation as rushed and inefficient. That account gives no comparative measurements or precise minimum hardware requirements, but it makes the practical adoption argument clear: users needed a way to run the model on computers they already owned.

At the time of the conversation, core development was reported to rest with one developer, while the team concentrated on the previously neglected frontend and was hiring. The planned V1 would package ComfyUI with an improved interface and easier Windows installation; Mac support remained a hope. No public release date was committed.

The business direction combined continued open-source development for local model execution with potential revenue from cloud inference and enterprise offerings. Other startups building on ComfyUI were welcomed: their use could attract people into the ecosystem even without direct contributions.

The final technical question is text. Custom nodes already supported tasks such as prompt enhancement, but adding text functionality to core remained a low priority. Development stayed focused on diffusion models—with an opening for a good, openly available text diffusion model, which the developer said he would probably implement if one appeared.