Open Models in 2024: From Downloadable Weights to Reusable Recipes

Why operate a model yourself?

An application needs its model to stay fixed, or must run on an edge device. Those requirements make control over deployment consequential, even when a hosted API offers convenient access to a powerful model. These are among Luca Soldaini’s early examples in the 2024 open-models session: a local model can meet engineering needs that benchmark rankings do not capture.

The choices expanded substantially in 2024. Soldaini highlights [Qwen](https://buttondown.com/ainews/archive/ainews-qwen-2-beats-llama-3-and-we-dont-know-how/), [DeepSeek](https://buttondown.com/ainews/archive/ainews-deepseek-v2-beats-mixtral-8x22b/), Llama 3 and fully open efforts such as OLMo. Some approached the performance of leading closed models, in his assessment, though benchmark selection could make the gap look larger or smaller.

For researchers, access enables experiments on model behavior, evaluation and inference that Soldaini argues would otherwise be impossible. For builders, he also identifies retrieval as a strong use case for local models, though his comparison with closed models comes without a specified task or benchmark.

Openness also supports the tools around a model: serving, efficiency and hosting. The same collaboration reaches into training. For successive OLMo versions, Soldaini’s team first looks for useful existing datasets; its later training stages also use outputs from other models. Shared components accelerate the next model, extending the value of a release beyond people who run it.

Open licenses and accessible reproduction

Soldaini treats the Open Source Initiative’s 2024 Open Source AI Definition as a significant development. In his account, it calls for freely available model weights, openly licensed code and no clauses prohibiting particular uses. He says Llama and some Qwen models fall outside the definition because of restrictions on use or requirements for naming outputs and derivatives.

His objection concerns training data. The definition, as he describes it, permits sufficiently detailed information to reconstruct a data pipeline and produce an equivalent system without requiring release of the data itself. He considers both sufficiently detailed and equivalent too vague.

Even complete documentation can leave reproduction out of reach. Soldaini offers a hypothetical data-recreation process costing $10 million: another team could understand every step and still be unable to perform it. Formal open-source status and practical accessibility are distinct; he wants open AI’s data requirements to do more for the latter.

Clearer recipes, higher compute barriers

By Soldaini’s account, 2024 brought clearer recipes for curating data for initial training and for post-training—the later work that adapts a base model. He singles out progress in reinforcement learning. Researchers had a better sense of the path toward competitive models, but executing those recipes at the scale of closed labs remained expensive. Despite better knowledge, he considered open-model research more resource-constrained than in 2023.

Compute needs vary sharply with the work: - Pre-training: Soldaini’s personal rule of thumb is that roughly 1,000 GPUs might permit some research, with 10,000 preferable. - Post-training: the discussion puts useful work at around eight GPUs when a suitable Llama model can run, while estimating an open replication of OpenAI o1 at roughly 10,000 GPUs. - Inference and evaluation: substantially smaller entry points remain possible, including evaluation with one GPU. These are informal estimates, with neither hardware nor training duration specified. They illustrate the range of research budgets rather than configurations from which to price a project.

Release the pipeline so others can reuse its parts

A fully open release provides the recipe and pipeline, allowing another team to reuse individual components, build a different model or improve the original. The release described here includes model weights, training data, training and inference code, training logs, and every intermediate checkpoint—a saved model state from the training run. The final model is one useful artifact among many.

An outside group reportedly reused the released pre-training data for an open replication of BitNet. Reuse also flowed into OLMo 2: much of its first-stage training data came from the multi-institution [DCLM initiative](https://buttondown.com/ainews/archive/ainews-apple-dclm-7b-the-best-new-open-weights/). In both cases, adopting another group’s data work freed a team to concentrate on a different part of the system.

The AI2 projects extend this approach across model development: - OLMoE provides a fully open mixture-of-experts model and its development artifacts. - Molmo provides a recipe for converting a text-only model into a multimodal model, applied to Qwen, OLMo and OLMoE checkpoints. - Tulu 3 provides a post-training recipe for adapting a base model, applied to OLMo and Llama, with an open Qwen replication effort also reported. The transferable recipe matters: its usefulness extends beyond the model released alongside it.

OLMo 2 combines data lessons from OLMoE with lessons about making models adaptable from Molmo and Tulu. It is presented as the best fully open language model at the time, though without evaluation details for that ranking. Its development illustrates how reusable knowledge accumulates across projects.

Open training data is becoming harder to obtain

Compute is only one constraint. Work involving Shane Longpre examined changes in website accessibility across Common Crawl, a publicly available collection of crawled web content, including comparisons between 2017 and 2024. The concern is that blanket crawling restrictions make previously accessible material harder to collect for pre-training.

An internal AI2 crawling project found a practical failure in the process for respecting website preferences. Sites sometimes blocked requests without advertising the restriction in robots.txt, the file used to express crawling preferences. In some cases, even that file could not be fetched. A crawler could no longer reliably check the published preferences before acting.

The speaker also criticizes blocking through infrastructure services such as Cloudflare and speculates that some content owners may be unaware of the restrictions applied. The larger concern is unequal access: established labs may already possess data that newcomers can no longer obtain. Openly accessible training data can become scarce before training data as a whole does.

Safety and policy require participation

The closing policy argument supports assessing AI risks and making openly distributed models safer. The speaker rejects treating AI as an alien technology with wholly unprecedented dangers, arguing instead that its risks are rooted in familiar software problems, including deployment in sensitive areas such as medicine. Work on open-model safety must balance accessibility with safe use.

The speaker takes a stronger position on biological risk, arguing that research—including work by Cohere researchers—undermined earlier concerns and characterizing the alarm as lobbying. That remains the speaker’s assessment; no study methods or results are given to support a general dismissal of biological risk.

Speaking explicitly in a personal capacity, the speaker opposes California’s SB 1047 and praises the open-source community’s late effort to explain its potential negative effects. The appeal is for sustained policy participation: advocacy attracts less enthusiasm than building models, but it also shapes whether the ecosystem can thrive.

Incentives and multilingual progress need collaborators

An audience question asks how incentives can keep open models open. The response endorses competitions such as the ARC Challenge, especially when participants build on open models and leave work that subsequent researchers can reuse. But model development is expensive and risky, making durable incentives difficult to design.

The speaker distinguishes investment in companies currently releasing open models from a commitment to long-term openness. Commercial incentives can support releases while they serve a company’s position. Each firm can make a locally advantageous decision without producing the best collective outcome.

A second question raises a perceived performance gap between open and closed models in low-resource languages. The response expresses an intent to improve multilingual support and predicts progress in 2025, without establishing that comparison through measurements. It identifies a concrete dependency: collaborators who know where useful regional language resources can be found.

A speaker originally from Italy gives a personal example: years in the United States have weakened their knowledge of how to access Italian libraries and other local content. Speaking a language does not necessarily preserve that knowledge. Cohere’s outreach to regional communities is presented as a useful way to recruit collaborators who have it.

Mistral’s year: models, customization and licenses

Mistral’s retrospective begins with its founding in Paris in May 2023, followed by Mistral 7B in September and a mixture-of-experts release in December. February 2024 brought Mistral Small, Mistral Large and the Le Chat interface. The presentation also covers a text embedding model, which converts text into vectors, and model distribution through Google Cloud, AWS, Azure, Snowflake and IBM.

Spring added another mixture-of-experts model and Codestral, presented as supporting more than 80 programming languages, without accompanying benchmark results. Model customization became available through a hosted fine-tuning service and the open-source Mistral Fine-tune codebase.

Later releases broadened the deployment options: a 3B model positioned for edge devices, an 8B model presented as a stronger replacement for Mistral 7B, and Nemo 12B, developed with NVIDIA. Mistral Large version two supported native function calling. Pixtral 12B and Pixtral Large handled both images and text, while Codestral Mamba explored the Mamba architecture and another research model targeted mathematical problems.

For deployment, the presenter distinguishes two license groups: - Mistral research license: free for exploration, with a purchased license required for enterprise production. - Apache 2.0 open-weight models: permit customization, fine-tuning and production use. Examples include Pixtral 12B, Mistral Nemo and Codestral Mamba. The license determines what a team can do with a release. The presenter also recommends migrating from three legacy models that were no longer receiving updates.

Le Chat puts the capabilities to work

The first Le Chat demonstration starts with a receipt: calculate the amount owed for coffee and sausage, exclude the other purchases, and add an 18% tip. The presenter identifies prices of 9 and 22 and reports that the model added them and calculated the tip. The task combines reading an image, selecting relevant line items and arithmetic; the final total and currency are not stated.

The presenter promotes these image-understanding and optical character recognition capabilities—reading text from images—as available through both Le Chat and the API. Le Chat is described as free at the time of the talk.

Next, Canvas generates an interface using PyScript to execute Python in the browser. The presenter tries import this and reports successful execution. A generated Tetris game follows, including an attempt to play it. These live examples show an interactive workflow: generate code, try it, ask for an explanation and request updates. They do not establish coding reliability across tasks.

The demonstration closes with web search for recent AI news and image generation using Black Forest Labs’ Flux Pro, prompted with researchers in Vancouver. Image generation is also described as free at the time.

In the event’s closing remarks, Laura Hamilton invites prospective founders to connect and brainstorm with her team, whose scope spans data, developer tools, AI infrastructure and AI applications.