What ClusterMAX Measures When It Ranks GPU Clouds

The GPU is only the beginning

A lab renting accelerators is buying more than a chip. Its jobs need machines to stay healthy, a network to connect them, software to schedule work, and people to respond when something breaks. A fast benchmark on an isolated machine does not answer all of those questions.

In a Latent Space discussion of ClusterMAX 3.0, the SemiAnalysis team explains its ratings of GPU cloud services. The team describes ten criteria spanning performance, security, reliability, ease of use, price and availability. It also says the project has evolved from largely manual tests to a more automated collection of industry-standard and custom benchmarks, reliability checks and burn-in tests.

The review is primarily for a buyer of a managed cluster. A provider supplies not only hardware but services such as managed Slurm or Kubernetes and replacing failed nodes. A very large lab may rent bare metal and run its own scheduling and operations; a customer buying an inference endpoint has a different question again. Those products can come from the same company, but they are not interchangeable.

Why deployment and recovery matter

ClusterMAX 3.0 raised its bar to include deployment of newer systems such as GB300 NVL72. The team describes a rack-scale group of GPUs connected within a scale-up domain, alongside a new network, ARM CPUs and direct liquid cooling. A provider has to make the facility, operating system, drivers and network work together; buying the hardware alone does not demonstrate an operational service.

The practical test continues after deployment. The team says a provider can improve its standing by running a working cluster, responding to feedback and fixing issues quickly. Conversely, stalled deployment, unresolved failures and customer reports of poor reliability can lower its assessment. A ranking can therefore change because the standard changed as well as because a provider’s service changed.

Reliability is especially important for long training jobs. The interview points to health checks as a way to identify unhealthy infrastructure before failures repeatedly interrupt progress. Managed replacement also matters: a customer wants a failed node dealt with so the cluster can return to useful work, rather than treating every failure as a separate hardware procurement project.

Security checks reveal a different failure

The team reports that during an earlier ClusterMAX assessment it encountered network isolation failures that let testers see other customers’ jobs, data or storage. It says it escalated the issue and rated the affected service poorly.

Some of the checks are deliberately basic. The discussion asks whether software and drivers are current and whether known vulnerabilities have been left unpatched. It also raises configuration of network access keys, including InfiniBand keys. An up-to-date package is not a complete security program, but an old version with a known vulnerability gives an attacker a documented place to start.

SemiAnalysis says it released a free CMAX command-line tool to check cluster software versions and point users toward updates. The tool addresses part of the maintenance problem; the cross-customer example shows why isolation and operating practices also deserve separate scrutiny.

Read a ranking for the job it measures

The team uses Crusoe to illustrate the scope problem. Its managed-cluster offering was downgraded in this assessment, while the speakers praise its data-center building and bare-metal business. A rating of one service should not be turned into a verdict on every business a provider operates.

The same distinction applies to finance. The speakers say ClusterMAX is meant to help customers choose compute, while their separate financial research can reach different conclusions about a company. A firm could operate data centers or sign large bare-metal contracts even if its managed service scores poorly; that operating assessment alone does not determine whether its stock is attractive.

The discussion also addresses incentives. A participant discloses investments in providers, and the team says it does not take payment for rankings and that its technical assessors have latitude to rate services. Those are disclosures and claims from the interview; a reader can assess the methodology and the available evidence alongside them.

A tight market changes the purchase

At the time of the conversation, the speakers describe GPU availability as unusually tight. Their explanation is competition from both large research labs renting smaller clusters than before and inference companies able to use even small allocations. A new lab looking for capacity is therefore competing with customers that can put those machines to work immediately.

Service quality still matters after capacity is found. The team proposes a separate evaluation of inference endpoints because high throughput on a small, artificial test is only one input to the buying decision. It describes cost, reliability, security and monitoring as other considerations: an endpoint with impressive raw speed may still be a poor fit if its API is unstable.

When inference becomes part of training

The discussion also complicates a familiar hardware split. In reinforcement learning for a model, forward passes generate outputs used in the training loop. Those passes resemble inference computationally, even though the larger job is training. A team may therefore want different kinds of accelerator for generation and for updating the model.

The attraction is matching hardware to work; the cost is coordination. The speakers point to numerical differences, memory and batch-size differences between chips, as well as models being partitioned differently across machines. If generation runs across sites, weight updates can arrive at different times and data must move between them. The episode presents this as an infrastructure challenge and an emerging direction, not a demonstrated universal recipe.

ClusterMAX is consequently a starting point for a specific purchasing decision. Its own authors say they are expanding the questions they examine as endpoint serving, managed clusters and other infrastructure diverge. For a buyer, the next step is to match the assessment to the actual workload and then investigate the service details that the workload depends on.