Guardrails: turning LLM outputs into application contracts

When stronger prompting stops helping

Shreya Rajpal’s applications worked well often enough to be exciting, but failed often enough that she could not deliver the user experience she wanted. Her main control over the large language model (LLM) was its prompt: stronger instructions, capitalization and more exclamation marks. Guardrails grew from a need for a better interface: specify what a good output looks like, then check and correct the response against those requirements.

By this May 2023 conversation, Rajpal had left her previous job to work full time on the open-source project. She described Guardrails AI as two connected pieces: a specification framework and code that enforces it. An output contract could describe a dictionary’s structure and field types, or reach into meaning—for example, whether a summary faithfully represents its source documents.

RAIL, introduced as a reliable AI markup language, expressed that contract. Guardrails used it before generation to construct the prompt, then afterward to validate the response and apply configured corrective actions. Rajpal had released summary faithfulness and coherence checks that morning. These were described as semantic guarantees, but their reliability was not quantified: the strength of any such guarantee depended on how well its validator could check the property.

Why put the contract in XML?

Guardrails began with Python classes. For the common task of describing a desired dictionary, Rajpal found that approach too verbose. Declarative markup reduced the upfront work, although she acknowledged that code offered flexibility markup might lack.

XML appealed because it kept field metadata separate from child fields. A field could carry a description and validity criteria while containing a nested structure. Those descriptions and criteria also flowed into the prompt, giving developers more room to communicate intent than a field name alone.

Failure handling borrowed from HTML event handlers. A failed criterion could trigger reasking—another request to the model, with parameters specifying how to seek a correction. Other failures could receive different treatment according to their importance.

The design goal was a low floor and high ceiling: approachable for people accustomed to prompting in English, yet expressive enough for complex workflows. An early request for a visual editor made XML’s document-tree representation attractive. Syntax preferences still depended on the audience: SQL familiarity could help business analysts, while reported enterprise feedback favored XML. A code-first interface and alternative markup formats were being explored.

RAIL also allowed embedded scripts to register custom validators and generate dynamic values, such as arrays. The design drew on JavaScript embedded in HTML, attaching validators to fields much as inline styles attach to elements, and Pydantic’s validation approach. The specification was intended to support programming with models beyond natural-language instructions.

Contracts at workflow boundaries

In a chain of model calls, each call could have its own RAIL configuration, with a library such as LangChain composing the calls. An early community project reportedly used a separate specification for each node. This supported workflow composition; importing reusable specifications or providing a package system remained a separate, unanswered question.

Agents made these boundaries harder to define. A proposed approach drew on a reported Drive.ai architecture: learned components worked alongside more deterministic components, with guarantees at their interfaces. An agent could similarly be divided into smaller tasks with checks on each output. But automatically generated goals may require automatically generated correctness criteria, making it difficult for developers to define acceptable behavior in advance. Model-written RAIL specifications were still a proposed experiment.

The argument for a separate validation layer extended beyond agents. Longer contexts, lower latency and cheaper inference were expected to improve model capabilities, but would not by themselves establish application guarantees. During Guardrails development, prompting that had produced structured outputs reportedly stopped working after an internal model update. Guardrails was intended to absorb that prompt-maintenance burden for its users.

The discussion also cautioned against treating temperature=0 as a fixed-seed reproducibility guarantee, without offering a model-specific test. The broader concern was the long tail of inputs and application requirements that training data and tests could not exhaustively cover. The proposed architecture combined LLMs with deterministic checks, rules, heuristics and traditional machine learning, rather than relying solely on models to check one another.

Text2SQL: generate, execute, repair

Structure and types were described as Guardrails’ most visibly popular features. [Text2SQL](https://shreyar.github.io/guardrails/use_cases/text2sql/text2sql/)—generating SQL from a natural-language request—showed how much more a validator could do. The feature called bug-free SQL checked whether a query could execute in a sandbox and used errors to guide repairs.

The workflow combined prompt context with execution feedback: 1. Accept a database connection string or schema file and create a sandbox SQL environment at startup, avoiding repeated setup for each query. 2. Add schema information and similar examples from serving history to the prompt. 3. Execute the generated SQL in the sandbox and collect errors. 4. If reasking is configured, send those errors to the model to request corrected SQL.

The example discussed repaired problematic table names and predicates. Its execution check established whether the query could run in that sandbox. The developer argued that combining validation, schema context and retrieved examples substantially improved on a plain-English request, though no comparative benchmark was given.

Reasking was one corrective action, with alternatives chosen according to the error and output: - Repair it programmatically. - Substitute a default value. - Filter an invalid value from structured data. - Request another model response. These options did not all fit SQL equally well. Fuzzy matching an incorrect table name against existing names was proposed as a local repair; the example actually used another model request. Combining local repairs with reasking was expected to reduce cost and latency, but savings were not measured.

Other SQL checks covered application restrictions: excluding predicates, checking column presence and limiting exposed columns to avoid sensitive data. Similar restrictions were suggested for tables. Executability and permitted data access were separate properties of the output contract.

Making failures visible and quality checkable

The validation loop retained an execution history: prompts, raw model outputs, checks performed, validation results and corrective actions. The developer described keeping this history in memory and writing it to a file. The interface grew out of building applications in notebooks, where inspecting each stage mattered. Deeper investigation tools were still under consideration, and a question about machine-readable logs and Datadog integration remained unresolved.

The proposed service-level agreements (SLAs) for outputs concerned content and quality. Guardrails could not guarantee response time while depending on external API calls. Quality commitments began with decomposing a request into checkable components: marketing copy could become a title, a short summary and a detailed announcement with required talking points.

Each component could carry constraints on length, readability or reading time. Subjective qualities such as wit were harder to validate, especially when the evaluating model lacked a reliable grasp of the desired quality. The contract was only as strong as the check: a requirement had to be expressible and evaluable, programmatically or with a model in the loop.

Quality also depended on the product. One content-creator tool builder reportedly found sanitized language undesirable because it undermined authenticity. Custom validators could express that preference—even requiring profanity. Guardrails provided a framework for application-specific requirements, not a universal definition of acceptable language.

Business constraints supplied another case. A prompt might instruct a bot to avoid mentioning competitors, yet violations could go unnoticed in production. Explicit post-generation validation offered a way to check compliance. For agents interacting with multiple services, however, responsibility for defining and owning those constraints remained unresolved.

Building the framework with its users

Guardrails’ integration strategy favored collaboration with other frameworks. Its developer reported LangChain and LlamaIndex integrations and a minimal model interface: a user-supplied callable with string input and string output. Developers could bring their preferred API component while Guardrails focused on the output contract.

User failures and feature requests tended to take priority over other backlog items. Intensive conversations after launch tested whether the framework fit problems beyond its developer’s own. The XML-versus-code question remained open, with a forthcoming code-first RAIL API shaped through a request for comments, collaborators and community feedback.

A useful distinction emerged between open source as distribution and open source as development. Guardrails had contributors and community requests, but its developer still handled most work supporting those requests. Cultivating outside contributions was a next goal. The business model was unsettled; the full-time commitment rested on the conviction that making LLM applications useful at production scale was a necessary problem to solve.

Engineering judgment: collaboration and experimentation

A career detour brought two lessons from working with Ian Goodfellow at Apple. A participant recalled his unusually creative feedback on machine-learning problems, then a smaller incident during their first or second week: after roughly 45 seconds of setup trouble while pair programming, Goodfellow asked colleagues on Slack for help. That challenged the new hire’s instinct to solve everything alone. The lesson was to optimize for solving the problem, including through collaboration.

Research interests ranged from model constraints and evaluation to efficient inference, longer contexts and fine-tuning with less data. On efficiency, the respondent favored combining methods and experimenting with the tradeoff between smaller, faster models and retained performance. The practical starting point was the bottleneck; the right combination depended on the use case, architecture and hardware.

Code generation makes the feedback loop tangible

The lightning round began with Copilot as the respondent’s favorite AI product, alongside enthusiasm for an experimental extension tentatively identified as Copilot Labs. Another participant described a voice component combining Whisper with Copilot, useful when typing was painful. From assistance, the conversation moved to iterative code repair: generate code, test it and use failures to guide the next attempt.

[AutoPR](https://github.com/irgolic/AutoPR) was described as a community-built GitHub Action using Guardrails: an appropriately labeled issue triggered creation of a pull request from the issue text and additional context. Wolverine, credited to BioBootloader, illustrated a repair loop that ran code, encountered an error, requested a corrective diff, replaced the affected code and ran again. The SQL validator encapsulated a related generate–execute–ReAsk workflow.

The prediction for the following year was more restrained: turning such demonstrations into consistently good production experiences would take longer than people expected. Demonstrating a possibility and delivering it reliably were different engineering milestones.

The email assistant that invents a fundraising pitch

The closing product request made that gap concrete: expand a short statement of intent into a concise email in the user’s tone, informed by previous correspondence, supplied links and connected data. Another participant had built an autonomous drafting tool using the current thread and standing instructions, including declining companies outside developer tools and requesting the sender’s Calendly for scheduling. Retrieving correspondence across threads remained a desired capability.

The builder used public figures with substantial online writing, including Paul Graham, as imperfect tone references. Their informal estimate was that 20% of drafts were ready to use, 70% needed editing and 10% were inexplicably off target, with no sample size or evaluation protocol. Drafting replies to every incoming email with GPT-4 also accumulated costs, prompting the suggestion to generate only for labeled messages.

The consequential failures involved applying conditional instructions in the wrong context. A rule about declining out-of-scope investment pitches caused the tool to invent fundraising context for unrelated messages. A scheduling rule produced replies arranging sales calls the recipient did not want. Recognizing the sender’s request did not establish the recipient’s intent.

That exposed the difference between draft generation and draft expansion. The existing tool received a thread without per-email guidance and ran while its user was away from the inbox. The requested alternative would begin with a brief draft expressing the user’s intent and expand it into a complete response. The two inputs defined different tasks, even though both products appeared to be email writers.

A proposed reusable guardrail would restrict action items to the email’s context, exclude unmentioned context and accept configuration for typical follow-up actions. It had not been implemented. The design challenge was to turn individual failures into checks that could be reused and composed across applications.

The closing recommendation was to build around tasks and outcomes, combining LLMs with existing systems, rules, heuristics and traditional machine learning according to the product’s needs. The email example gave that advice a precise consequence: before enforcing an output contract, an application must determine which task—and whose intent—the output should serve.