Simulation Fidelity and GPU Efficiency — Part 3 of 3

Simulation accuracy limits experimental predictions

Models trained on computational materials data need to answer real experimental questions. The closing part of the conversation opens with a barrier to that goal: training on inadequate physics approximations cannot, by itself, close the predictive gap. One speaker describes this as a problem pursued for a decade in AI for materials. The missing information lies in the underlying simulations, so learning from their outputs does not supply it.

AlphaFold provides a contrast through its use of experimental training data. The speaker distinguishes that approach from the simulation-based route, pointing to tens to hundreds of millions of simulated training points attributed to Meta. In their assessment, even that volume lacks the fidelity needed for the materials problems they want to solve. The distinction is data volume versus experimental relevance: more simulated examples can leave the underlying accuracy problem unresolved.

Better GPU utilization could fund more laboratory work

Another speaker turns to MFU, a measure of training efficiency relative to a GPU’s advertised peak floating-point throughput—the number of floating-point operations it can perform per unit time under favorable conditions. The acronym’s precise expansion is uncertain here, but the practical concern is clear: how much of the capacity being purchased does training actually use?

The speaker estimates reinforcement-learning utilization at roughly 5–6% of theoretical peak, without specifying hardware, workload, or measurement conditions. This is an estimate in the discussion, not a universal benchmark. Their hypothetical wish for full utilization has two benefits: reaching answers faster and buying fewer GPUs, freeing money for laboratory work.

Parallel experts accommodate different training timescales

Laboratory capacity constrains experimental rollouts—the runs that produce trajectories for training. Why, then, prioritize GPU throughput? The response accepts that constraint and distinguishes two ways to organize the learning process:

- Collect trajectories, then update a central model. Let the model perform rollouts, wait for enough trajectories to accumulate, then use backpropagation to update it. - Train experts in parallel, then distill into a central model. Have multiple expert models generate data or learn independently, then use their outputs to train the central model.

The speaker considers the second approach more efficient because the activities operate on different timescales. Laboratory capacity still constrains experimental rollouts, while expert-model work and central-model training have computational demands of their own. A hypothetical phase involving ten trillion tokens illustrates the point: processing that volume efficiently, with reinforcement learning layered on top, still makes GPU throughput consequential. The token count is a scale example, not a disclosed training run.

A computational hiring push in San Francisco

The conversation closes with a hiring invitation. At recording time, the organization described roughly 20–30 people at its San Francisco office at 181 Fremont Street, with plans to expand aggressively. Priorities included post-training and domain AI for life sciences and materials science. The office handled computational work and had no wet lab at that time. Interested listeners were invited to contact the speaker or Rafa.