From the Reversal Curse to DSPy: ICLR 2024, Part 5 of 5

Can prompting reverse a fact learned in one direction?

A model has learned a family relationship in one direction. Can prompting help it retrieve the relationship in reverse—for example, identify someone’s son? This problem opens Part 5 of the ICLR 2024 conversation. The reversal curse concerns this difficulty with facts learned in one direction. The respondent says they have not seen strong, robust results from prompting alone at inference time, when the model answers without further training.

The discussion considers three interventions: - Change the training objective. Blank infilling predicts tokens given the remaining tokens, without a causal mask restricting the available context to one direction. The speaker proposes this as mitigation, without suggesting it guarantees that the model learns everything. - Train on reversed facts. In Deductive Closure Training, a model assembles facts it knows, reverses them within the context, and then trains on those reversed versions. This changes training rather than recovering the reverse relationship solely at test time. - Search candidates at inference time. For a question about someone’s son, the model might enumerate celebrities until a candidate lets it check the familiar forward relationship. The speaker expects this could work in suitable domains; it is a proposed workaround, not a measured result.

Facts learned during training differ from facts supplied in the prompt. A speaker discussing the reversal-curse paper as their own recalls readers posting screenshots of successful reversal after placing the relevant facts directly in context. Those examples did not address the intended finding. The speaker acknowledges that the paper’s scope was insufficiently clear and believes its popularity partly reflected an appetite for evidence that language models were unintelligent—a misleading interpretation of the results.

Anthropic’s influence-function work, which examined influential training documents, was concurrent work, the respondent clarifies, rather than a response to the reversal-curse paper. They describe a related observation: influential documents presented relevant facts in the same order as the model’s response. They explicitly stop short of calling this causal evidence.

DSPy separates program structure from prompt wording

DSPy builds and optimizes language-model programs: one or more model calls, potentially chained together to perform a task. It separates three responsibilities: - Signatures define inputs and outputs, such as a question going in and an answer coming out. - Modules select the prompting technique, such as simple prediction or chain of thought, which adds a reasoning component. - Optimizers search for better prompt content, including task instructions and demonstrations placed in context.

For a simple question-answering task, DSPy turns the signature and module into a concrete prompt containing task instructions and a template for the fields the model should produce. This basic prompt can run before optimization. The optimizer then searches for instructions and examples that improve its performance.

Generate candidates, test combinations, reuse the template

The optimization workflow described in the conversation has three stages: 1. Propose instructions. Another language-model program receives grounding material, including a generated summary of the training dataset and the program’s signature, then proposes alternative instructions. 2. Generate demonstrations. A teacher model performs the task. In the example, GPT-4 produces an answer and a reasoning trace. If the final answer is correct, the full trace is retained as a few-shot demonstration—an example included in the prompt. 3. Search combinations. The optimizer tests combinations of instructions and demonstrations, using Bayesian optimization to search for effective interactions between them.

The demonstration filter assumes that a correct final answer indicates a useful reasoning trace. It does not independently validate each intermediate step, even though the retained trace becomes an example for future calls.

The concrete output is a reusable template containing instructions and few-shot examples. Future calls use that optimized template. DSPy is also described as supporting fine-tuning, but the procedure explained here produces prompt content.

Optimize pipelines that lack intermediate labels

The abstraction becomes more useful when a task requires several stages. In the question-answering example, a model generates a search query, the query retrieves relevant documents, and another stage uses those documents to answer the original question. The presenter compares this structure to PyTorch: prompts play the role of layers, while a forward pass specifies how their inputs and outputs connect. Developers can define modules and combine them into more complex prompting strategies.

A training dataset may supply only initial questions and final answers. An answer label does not supply a good retrieval query, leaving intermediate stages without their own labeled examples. DSPy’s optimizers bootstrap demonstrations for those stages, reducing the work of handcrafting and testing examples throughout the pipeline.

DSPy therefore combines program abstraction with optimization: developers specify prompting techniques, inputs, outputs, and connections; the framework constructs templates and helps search for effective prompt content.

The metric determines what optimization rewards

A questioner raises a practical concern: prompt optimization involves discrete choices, many combinations, and potentially sparse reward signals. Can a limited search budget find a better prompt? The respondent reports improvements of roughly 10–15 points over an initial program from randomly bootstrapping demonstrations and testing which work best. The task, metric, model, evaluation dataset, and budget are unspecified, so this report does not establish an expected gain for another application.

From their own DSPy experience, the questioner identifies a harder prerequisite: an evaluation function that provides a useful selection signal. Writing one can be difficult, especially for beginners. The respondent agrees that the metric determines which demonstrations count as good. Automated search still depends on a definition of successful behavior.

The questioner also argues that DSPy’s README emphasizes few-shot selection while underexplaining instruction optimization. The respondent, who works on that area, agrees it needs updating. They describe a new optimizer that supports three choices: - Instructions alone: optimize the wording of the prompt. - Demonstrations alone: optimize the few-shot examples. - Both together: jointly optimize instructions and demonstrations.

Joint optimization gave the largest improvement in the respondent’s account, though no quantitative comparison or evaluation conditions accompany that claim. The instruction optimization discussed here concerns prompt wording; it does not establish model-weight training for this optimizer.

Healthcare motivation, open-source origins, and output formats

Healthcare applications motivated one contributor’s involvement in DSPy. They found prompt optimization useful for retrieving clinical information and extracting insights from clinical notes, but manual prompt engineering was tedious. That experience led them to develop better instruction generation and joint optimization across full multi-stage pipelines.

At the time of the conversation, DSPy was an open-source project originating in Stanford labs, with Berkeley participation, rather than a startup. The contributor credits Omar with starting it and describes a large Discord community, open-source contributors, and company employees who contribute improvements while using DSPy at work.

The closing technical question concerns SGLang as a backend for enforcing output formats. The contributor tentatively says DSPy supports it for that purpose, then defers to others for confirmation and implementation details, leaving the integration unresolved in this exchange.