ESM and the Data Barrier to Programmable Biology

Search for a protein that binds

Finding a protein that binds a chosen target is a concrete test of programmable biology. ESM-C approaches design by searching a predictive model for molecules that satisfy specified criteria. The team reports designing protein binders, including single-chain antibody formats called scFvs. The premise is that a general model of proteins can support design through search.

In the Latent Space conversation, Alex Rives traces his work on protein language models to summer 2018. He credits his team with training the first transformer language model for protein biology and describes new capabilities emerging as successive generations grew. His conviction rests on scaling as an empirical finding: experiments can justify further investment even when theory cannot reliably predict what a larger model will learn.

There is a biological reason to expect sequence prediction to teach structure. Consider two amino acids—also called residues—that touch when a protein folds. Evolution cannot freely change one without preserving compatibility with the other. Across related proteins, those physical constraints leave correlated patterns in their sequences. Sequence variation carries indirect evidence of folded structure.

ESM applies that intuition across protein diversity. Its masked language modeling objective predicts amino acids hidden at different positions using the remaining sequence as context. Accurate reconstruction may require learning the biological constraints that determine which amino acids evolution can preserve.

Representations, structures, and an atlas

The proposed protein world model combines three components: - ESM-C learns representations of protein sequences. - ESMFold2 predicts structures from those representations. - Sparse coding extracts biological features, helping connect proteins across evolution. Together, they support structure prediction, discovery, and the inverse problem of finding proteins with desired properties.

The accompanying atlas contains 6.8 billion nonredundant protein sequences and 1.1 billion predicted structures. To obtain structural coverage, the team clustered sequences at 70% sequence identity and predicted a structure for each cluster center. Rives expects other members to share its fold with small variations. This is representative coverage, rather than an individual structure prediction for every sequence.

The feature space provides another way to navigate the collection. Rives describes gene-editing systems that are distant in sequence but share functional patterns and structural relationships. Bringing them together makes the atlas useful for finding connections that sequence similarity alone may obscure.

Amino-acid prediction learns reusable biological features

Sparse autoencoders (SAEs) extract features from model representations. The team trained them across the layers of ESM-C’s 300-million-, 600-million-, and 6-billion-parameter models, with the deepest analysis focused on the largest. The reported features form a hierarchy from biochemical properties and structural building blocks to broad functional concepts. That organization emerged from amino-acid prediction without explicitly supplying the hierarchy.

One example is the nucleophilic elbow, a functional structural motif visible in crystal structures. A single reported feature activates for this motif across evolutionarily diverse protein families with very different overall structures. Some may have evolved the motif independently; the observation is that the model reuses one feature across those contexts.

The proposed explanation is compression through reusable biological variables. An amino acid’s identity depends on other positions in the sequence. A model can represent recurring constraints as hidden variables and reuse them to predict amino acids in different contexts. This remains a hypothesis about why shared features emerge.

The conversation connects this intuition to Zellig Harris’s 1954 paper Distributional Structure: the contexts in which a word appears reflect its meaning. In the biological analogy, structure, function, interactions, and evolutionary history constrain where amino acids occur. Learning those contextual patterns can expose the underlying biology.

The companion article adds examples of features that separate function from overall shape: - DNA binding across different folds: shared features activate in structurally distinct proteins that bind DNA. - Membrane integration across protein classes: features identify membrane-spanning segments in different kinds of proteins. - Bonded versus free cysteines: features distinguish cysteines participating in disulfide bonds from those that do not.

Metagenomic diversity changed the scaling behavior

An interviewer raises a data-distribution objection: available sequences may disproportionately reflect medicine, human biology, and disease. The response agrees that which data scales matters. A controlled, replicated experiment serves a specific hypothesis. Learning a general protein representation instead calls for observing amino acids in as many evolutionary contexts as possible.

ESM2 showed diminishing returns in structural representation quality as parameters and compute increased from roughly billion-parameter to ten-billion-parameter scale. It trained on UniRef, which combines sequencing resources and clusters sequences to reduce redundancy. ESM-C added billions of metagenomic sequences. The generations were approximately comparable in parameter scale, but ESM-C received more compute, making this a comparison with multiple changes, not a controlled test of data alone.

Metagenomic sequencing samples genetic diversity from environments such as hydrothermal vents, soil, the deep ocean, and the human gut. Researchers sequence genetic material and infer protein sequences from it. The breadth comes with noise: sequences may lack organism assignments or wider genomic context, and fragments may encode only partial proteins.

With that expansion, diminishing returns reportedly disappeared over the explored ESM-C scales. The team interprets this as evidence that ESM2 had been data limited. Smaller runs estimated the best representational fidelity attainable at a given compute budget, allowing extrapolation to larger models. The conversation supplies no numerical fidelity metric or prediction errors.

ESM-C is described as essentially a standard transformer with training-efficiency changes. Its aim is to learn biological organization from data, with fewer explicitly built-in biological assumptions than the discussion attributes to AlphaFold.

One important distinction concerns multiple sequence alignments (MSAs): comparisons of related sequences that expose correlated mutations, providing evidence of which residues lie near one another in three dimensions. ESMFold2 does not require these as inference inputs. The companion article nevertheless says ESM-C’s data includes a dataset generated with AlphaFold2, creating an indirect dependence on alignment-derived information. Alignment-free inference does not imply alignment-independent training.

Two routes to programmable proteins

The conversation distinguishes two approaches to specifying what a protein should do: - Prompt through explicit tracks. ESM3 lets biologists specify structure and function, learning from sequence, predicted structural, and functional patterns. - Search a predictive model. ESM-C supports finding molecules whose predicted properties satisfy a design goal. The response treats both as useful. Whether post-training could give ESM-C ESM3-like programmability remains unresolved.

The antibody example makes search concrete. An scFv is a single-chain antibody format that brings heavy- and light-chain components together into a target-binding interface. The team reports encouraging design success rates and binding affinity at levels needed for therapeutic function. Numerical affinities, assay conditions, and efficacy results are not given; these are promising binding properties, not demonstrated treatment outcomes.

Reformatting the designed scFvs is proposed as the quickest route to full antibodies. Direct design of full antibodies had not been tried. The broader thesis is that therapeutic design can emerge from searching a general model of sequence, structure, and function.

In response to an AlphaFold3 comparison, the team claims substantially better antibody performance. Its proposed explanation is that antibodies benefit less from evolutionary information than structural-topology prediction does, while ESM-C’s representation captures useful antibody properties. The recorded comparison includes no benchmark conditions or numerical results.

Discovery turns model connections into experiments

The same representation can guide searches for useful natural proteins. Known, distantly related gene-editing systems reportedly cluster together consistently with existing knowledge. Unknown proteins also appear in the map, suggesting that some could be new gene-editing systems. A suggestive cluster is a hypothesis to validate. The conversation reports that Feng Zhang’s group used the first ESM Atlas to find a new gene-editing system, illustrating the intended workflow.

The motivating example is a heat-stable polymerase from bacteria living in a thermal hot pool, which enables PCR. Connecting known proteins to unknown ones could reveal further building blocks for green chemistry, medicine, and climate-related applications. Those remain prospective uses of better discovery tools.

An interviewer proposes mapping every pair of human proteins as a starting point for a virtual cell. The response calls that valuable without committing to build it. ESMFold2’s alignment-free inference makes large-scale prediction attractive: sequence-to-structure inference is described as taking seconds, although no hardware, input-size, or accuracy conditions accompany that latency claim.

The longer-term direction is structure determination through models and measurements together. BioHub is developing cryo-electron tomography systems intended to improve contrast when observing cells at the atomic level. An experimentally resolved structural map of interactions remains a future ambition with substantial technical hurdles. Computational predictions can provide a proxy while the measurement technology develops.

Close the experimental feedback loop

The proposed scientific workflow combines scaled data generation, predictive representations, reasoning, and feedback: 1. Evaluate hypotheses digitally using models that predict experimental outcomes. 2. Select a small set of experiments from the larger space of possibilities. 3. Measure outcomes and update the models, then use the improved understanding to choose further experiments. The full capability remains prospective, with an ambition to consider thousands to hundreds of millions of hypotheses in parallel.

BioHub aims to build two complementary kinds of infrastructure: - Data generation for accurate predictions: produce the observations needed to learn useful biological representations. - Experimental feedback for continued learning: let models select experiments, examine outcomes, and revise their understanding. Open-ended biology makes feedback harder than in enclosed computational domains. The conversation imagines training with experimental verification, but does not specify an implemented method.

BioHub’s stated institutional role is to build scientific tools and support open science, combining experimental biology, measurement technology, and AI. The work described spans atomic imaging, light sheet microscopy of developing organisms, inflammation maps across space and time, and cellular reprogramming. The long-term disease vision is to connect molecular events to physiology for individual genomes—an exceptionally difficult problem across biological scales.

A virtual cell must predict a new intervention

Moving from molecules to cells raises the standard for success. A useful model must predict a new intervention in an unobserved context. The discussion characterizes existing virtual-cell models as good representations of their underlying data but limited at this kind of generalization. Answering new biological questions requires more than reproducing familiar observations.

Asked whether molecular dynamics and local interactions could bridge proteins and cells, the response distinguishes two complementary approaches: - Physical simulation: explain behavior through physical, chemical, and biochemical mechanisms. Whole-cell simulation from first principles is described as far beyond current computational capabilities. - Information modeling: learn patterns linking genomic information, gene transcription, and cellular outcomes, treating the cell’s underlying programs as a tractable abstraction. Protein structure prediction supplies the historical motivation: learned patterns proved productive where first-principles simulation had been expected to provide the answer.

Claude Shannon’s work on English-language entropy supplies another analogy: what would an ideal predictor look like for biology? The hypothesis is that sufficiently rich observations of cellular outputs could reveal underlying programs and support an information-based account of disease. Reaching that goal may require new measurement technologies, model architectures, and data that does not yet exist.

Build data for generalization deliberately

Protein models benefited from decades of genome sequencing, metagenomics, and structure determination. The next opportunity is to collect data explicitly for machine learning. BioHub’s Virtual Biology Initiative commits $400 million internally to data creation and measurement technology, plus $100 million to catalyze external data-generation efforts. It is presented as an initial contribution to a larger effort whose priorities will be shaped with the scientific community.

The ambition is to accumulate useful data in a couple of years rather than decades. Generalization calls for many interventions across many biological contexts, with three measurement priorities: - Perturbation biology: observe responses to interventions through approaches including Perturb-seq, alongside transcriptional measurements, imaging, and other cellular information. - Spatial biology: observe how cells interact and form systems within the body. - Simultaneous measurements: connect cellular traits, transcription, protein-level information, the genome, and epigenetic state.

The Human Cell Atlas, Tabula Sapiens cell atlases, and Cell by Gene—a single-cell transcriptomics database—are cited as foundations for expansion. Practical priorities include lowering costs, improving sequencing and cell encapsulation, and increasing both the number of interventions and the biological layers measured together.

Deploy existing assays while improving measurement technology. The estimate offered is that current technology could support 10–100 times the data with reasonable investment; another tenfold increase or more would require substantial technology development. These are projections. Automation and flexible robotics are expected to help build the experimental feedback layer.

Small sequence differences may teach function

The closing discussion returns to protein-model limits. Data and compute must grow together. Public genomic data had not been exhausted, and the new atlas contained more sequences and structures than ESM-C had trained on.

When one participant calls much of the larger sequence pool redundant, the response challenges the definition. Different learning goals may require different kinds of variation: - Broad family diversity may teach structure: many kinds of proteins help develop structural representations. - Small sequence differences may teach function: fine genetic variation reveals consequential constraints; a single mutation can destroy a protein’s function. Current models are described as still lacking a deep understanding of these small but critical patterns. What counts as redundant therefore depends on what the representation needs to capture.

Where diminishing returns begin remains an empirical question. The team describes ongoing work on the next ESM and enough data to justify another model, without announcing a name or schedule. The recording closes by inviting scientists to use the forthcoming tools and collaborate on capabilities that would accelerate their research. The companion article reports ESM-C and ESMFold2 as MIT-licensed releases and links the protein-model preprint and resources.