NeurIPS 2023 Startups, Part 3 of 4: Sparse Compute and Inspectable Data

Ask what the poster’s author is working on next

A conference poster can describe a model whose second or third version is already available. One participant in the NeurIPS 2023 startup conversations sees that publication lag as a reason to prioritize meeting researchers and collaborators. Another offers a practical response: ask presenters about their next project. The displayed result may be familiar; the work its authors are most excited about may not yet be on arXiv.

Jeremy illustrates the value of those conversations with a reception encounter: he and Tanishq approached Albert Gu to discuss Mamba, asking why he used state space models, what opportunities and limitations they presented, and whether attention still had a role. Jeremy’s deliberately provocative conference advice is to read the papers, skip the sessions, and spend time in the hallways. Find interesting attendees through the conference app, assemble a small group, and go somewhere together.

Whova, the conference app discussed here, lets attendees organize and discover meetups. Examples range from technical conversations in a Brazilian Portuguese chat to a sushi outing with no machine-learning agenda. At an earlier conference in Hawaii, a morning swim became a research discussion in the ocean.

The exchange follows Jeremy’s invitation to exceptional candidates interested in joining a small team. It closes with his comic retraction of criticism of Laundry Buddy: sorting whites from colors, choosing powder or pods, and handling stains become grand contributions to beneficial artificial general intelligence. The inflated capability claims are part of the joke.

Cerebras: language-data availability shapes the model

Joel Hestness begins with the models Cerebras’ expanded systems made possible. Multi-box systems enabled cluster-level training of larger models; CerebrasGPT, an openly released model family, demonstrated what he describes as compute-optimal scaling. That work attracted the OpenTensor Foundation, which wanted a three-billion-parameter model with general language capabilities and long sequences, intended to be easy to deploy on a laptop. Cerebras trained BTLM for that request; laptop performance is a deployment goal here, without throughput or memory measurements.

A concurrent engagement with Group 42 in the United Arab Emirates grew from Arabic-only demonstrations into Arabic-English models with 13 billion and 30 billion parameters. The first 30-billion-parameter version had just been released at the time of the interview. Hestness says it outperformed other public models in Arabic and was competitive with Falcon 40B in English, without naming benchmarks or evaluation conditions.

Asked about the UAE’s separate TII effort, Hestness describes initial competition between differently funded groups alongside a national push toward coordination. Cerebras had examined datasets such as RefinedWeb, and some exchange had occurred; direct coordination remained a possibility. In an inserted conversation, an unnamed TII representative confirms a Severance reference in the dataset naming discussion: periodically inspecting the data resembled looking at the show’s ominous numbers.

Should every country have its own model? Hestness approaches the question through unequal language-data availability. He describes token volumes as following a power-law-like distribution, with more than 100 languages having too little data for straightforward model development. His suggested strategies depend on that constraint: - Low-resource languages: Start with an English base model, then co-train on English and the target language. The aim is to transfer learned knowledge representations while adapting style, grammar, and syntax. - Medium-resource languages: For Arabic, mix languages during multilingual training, including to support translation. - Higher-resource languages: For French, Spanish, and German, training from scratch and multilingual training are more feasible. These are training recommendations, without quantitative comparisons in the conversation.

Make sparse weights save actual work

Hestness then introduces his role at the time: principal research scientist and lead of Cerebras’ core machine-learning group. The group connects foundation-model and multimodal work to compilation, hardware performance, numerical correctness, and training stability. Cerebras uses a full wafer as its processor, avoiding much of the packaging, networking, and software needed to connect separate accelerator cards.

Hestness estimates that a CS-2 delivers performance roughly equivalent to 20 A100 GPUs while being programmable like a single GPU—a broad comparison without a specified workload or precision. He attributes manufacturing and reliability advantages to fewer components and failure points, while presenting GPU-comparable price-performance as a goal.

The concrete efficiency mechanism is unstructured weight sparsity: the hardware can operate on individually selected weights. In weight-streaming mode, an incoming weight is multiplied by a vector of activations. Sending only the weights needed for an operation enables sparse matrix multiplication without loading the entire weight matrix at once.

Could deciding which operations to skip cost more than performing the arithmetic? Hestness answers that the sparsity is embedded in the model, with its pattern preferably held stable for stretches of training. He distinguishes two approaches: - Static sparsity: Substantial training identifies a mask—the pattern of weights to retain. The model is then rewound and retrained with that same pattern throughout, in the lottery-ticket-style process he describes. - Changing sparsity: A then-recent software release added beta support for changing the pattern during training. Hestness cites RIGL and SET, saying their updates are infrequent enough to avoid harming performance on Cerebras hardware.

The work discussed as SparseIFT explores how to widen a layer at the same arithmetic budget. A wider layer provides more activations in which to represent information; sparse weights offset the additional floating-point operations, or FLOPs. The activations themselves remain dense. Hestness reports roughly 2–3× performance improvement at 75% sparsity, or sometimes a 3–5% better model at the same FLOP budget. He does not specify the task, performance measure, or quality metric, including whether the latter percentage is relative or in percentage points.

Reassess batch size; retrain after pruning

The next efficiency question is how much data to put into each training batch. Hestness describes research on approximate per-example gradient norms—measures of gradient magnitude. A full-batch norm helps characterize training dynamics; combining it with sub-batch norms provides gradient-noise statistics, which help assess suitable batch sizes and training efficiency.

The approximation combines statistics from forward activations and backward-propagated gradient values to estimate per-example norms more cheaply, then combines those estimates to estimate gradient noise. The explanation does not specify the operation or complexity clearly enough to reconstruct the algorithm. The operational advice is to track the statistics throughout training, especially across phase or data-distribution changes, to reassess batch size and learning rate.

Deployment introduces a separate workflow, described as related to SparseGPT: 1. Pretrain a dense model. 2. Prune weights to produce a sparse model. 3. Retrain to recover capabilities before deployment.

Discussing teammates’ work with which he acknowledges limited familiarity, the speaker reports fully recovering upstream losses at upwards of 50% sparsity for very large models that had not been trained on a huge number of tokens. No specific model, token budget, or downstream evaluation accompanies the result. Efficient deployment was becoming a central challenge for organizations already able to pretrain large models; Qualcomm and NeuralMagic are named as collaborators.

Recovery training can use a different data distribution, but learning-rate warmup and scale need care. A larger distribution shift may warrant a slightly higher learning rate so the model can adapt. Comparisons among Llama 2’s data, SlimPajama, and the Pile remain approximate: overlapping content does not make their distributions identical or establish Llama 2’s original training mixture.

Cerebras was expanding its applied machine-learning team at the time and invited applications. The interview closes with conference advice: pace yourself, wander through posters, talk to people, ask questions, and allow ideas time to sink in.

FiftyOne: find the subset behind the failure

Jason Corso introduces Voxel51 as a spinout of his University of Michigan lab, where he was a faculty member in engineering and robotics. As co-founder and chief scientist, he describes its toolkit as a model and dataset debugger above frameworks such as PyTorch and TensorFlow. Downloading a dataset and training once—or selecting a foundation model—does not automatically produce the behavior an application needs. Datasets and models have to develop together.

FiftyOne represents data alongside models so engineers can query and inspect their relationship. Corso gives examples of finding outdoor cases where model A outperforms model B, locating road intersections, and visualizing embeddings—vector representations of vision, point-cloud, or multimodal data. Users can select points with a lasso in a 3D embedding view. The illustrative debugging task is finding 1,000 troublesome samples within a 10-million-sample dataset; those numbers describe the problem, not a measured retrieval result.

The active-learning discussion focuses on selecting useful additional data, rather than assuming more samples will improve the model. A Voxel51 colleague had published a workflow built on FiftyOne. At the time, built-in support covered images, video, and point clouds. Custom data types were possible; audio, text, and temporal signals from connected devices were under consideration.

Voxel51 had pivoted from video understanding as a service. Its name retained the original references to space-time volumes of pixels and Area 51. The current product boundary followed deployment assumptions: - Open source: local data, one user, one machine. Within those assumptions, the speaker claims no restriction on the machine learning users can do. - Enterprise: collaboration and cloud use. The paid product is described as adding something like a Google Drive layer over the open-source tool.

The conference advice is to limit daily scope: choose five to ten papers in the morning, spend focused time with their presenters or talks, and write down the key points at day’s end. The conversation then returns to developing datasets and models together. The interviewer adds a secondhand Cerebras example: monitoring loss by dataset subdomain during training and increasing exposure where loss was improving less. No quantitative outcome accompanies the account, but it describes a concrete feedback loop between subset-level behavior and data selection.

Atlas: make vector data explorable

Brandon introduces himself as Nomic’s co-founder and CEO, with two main products: - GPT4All runs language models with limited compute. His example for the open-source ecosystem is Mistral 7B fine-tuned on OpenOrca running on a MacBook or less common GPU, without throughput or memory measurements. - Atlas explores unstructured datasets in the browser. He presents it to the NeurIPS audience as large-scale, interactive t-SNE: a way to turn data into clickable maps.

Brandon says Atlas optimizes the t-SNE objective and characterizes its relationship to UMAP in terms of the choice of low-dimensional kernel. He reports mapping Wikipedia in about four hours, compared with a couple of days for an off-the-shelf NVIDIA RAPIDS UMAP implementation on what he calls the largest AWS machine. Dataset size, embeddings, settings, and hardware are insufficiently specified for a controlled speedup comparison. The product argument is that map generation must fit a daily iteration cycle to become routine engineering work.

Atlas rests on the thesis that generative AI changes the working shape of data from tables to vectors plus metadata. Nomic wants that shift to reach beyond vector databases into analytics and visualization: something analogous to Tableau, Looker, or Power BI for unstructured data. Text, images, audio, and social-media content can enter this interface provided they can be represented as vectors.

Two ingestion paths support that idea: - Bring embeddings: Supply vectors and metadata through the Nomic Python package and atlas.map_embeddings, then receive a link to an interactive map. - Bring raw data: Use built-in embedders developed internally or with partners.

The browser rendering engine, Deep Scatter, is described as supporting billion-point scatterplots, without benchmark conditions. Its code was source-available, rather than fully open source. The speaker says he would like to open the entire system but cites commercial constraints.

Inspect anomalies—and lower the barrier to running models

Work with Hugging Face supplies a concrete Atlas debugging example. The team reportedly found low-quality training content and leftover token-related material that should have been cleaned out. The multimodal training material included interleaved webpage content such as images and poetry. The speaker says anomalies often appear as small isolated clusters around a map’s edge, where users can lasso them for inspection.

Atlas also supplies a topic model that lets users investigate whether loss concentrates in particular subjects. The speaker reports a high-loss mode within poetry in the Hugging Face analysis, without loss values or an evaluation protocol. He offers two hypotheses: poetry deliberately disrupts common linguistic patterns, or it captures human qualities the model has not grasped. He favors the first; neither explanation is established by the observation.

For consulting customers, the user may be a doctor or regulatory specialist handed an unfamiliar document collection. Some partners reportedly track time to Atlas: how quickly incoming data becomes a map an analyst can explore without code. The team also mentions Google Cloud collaborations on public datasets.

Returning to conference advice, the speaker encourages approaching admired researchers. He recalls asking Oriol Vinyals for advice at an earlier conference and receiving generous time for a conversation about modeling, life, and career choices.

The closing technical theme is access for the GPU-poor. Q-LoRA is singled out for its implications for work with limited resources, though its implementation is not explained here. The speaker argues that research reducing the resources needed for training and deployment would particularly benefit the open-source community.

The interviewer returns to GPT4All, describing it as an early success, the tool transcribed as “Llama CPP” as developing into a platform, and Ollama as an emerging local-model option. Part 3 ends with the positioning question: how should people think about GPT4All among those tools? The answer falls beyond this part.