Andrej Karpathy

Andrej Karpathy is an AI researcher and educator, a founding member of OpenAI, and a former Tesla Director of AI. His contributions include collaborative image-description research, Stanford's first deep-learning course, and readable implementations of language-model training. His writing and teaching emphasize understanding the computation, inspecting the data, and checking that a model behaves as intended.[1][3][6]

Karpathy's nanoGPT exposes a compact PyTorch GPT model and training loop, with a character-level Shakespeare example as an entry point. It rewrote his earlier minGPT with a stronger emphasis on practical training and finetuning. A November 2025 notice deprecated nanoGPT in favor of nanochat, while leaving the earlier implementation available.[7]

With llm.c, he exposes lower-level training operations in C and CUDA, alongside a PyTorch reference and a simpler CPU implementation. Tests compare logits, losses, and training losses against the reference. Karpathy sets education and practical speed as joint goals, while protecting readability in the default files: he says he may reject a hypothetical 2% speed improvement that costs 500 lines of complex C and an exotic dependency. The dev/ directory allows greater local complexity for experimental and educational kernels.[8]

nanochat extends the compact implementation to a complete pipeline: tokenization, pretraining, finetuning, evaluation, and inference on a single GPU node. Karpathy chose one principal model-size control, transformer depth, from which related hyperparameters are derived. Its author-maintained “time to GPT-2” benchmark measures how long a run takes to exceed the GPT-2 1.6B checkpoint's DCLM CORE score of 0.256525 on an eight-H100 node; that criterion measures a particular benchmark rather than equivalence across all tasks.[9]

In March 2026, Karpathy's autoresearch turned a simplified single-GPU nanochat training setup into a bounded agent experiment loop. The agent edits train.py, runs training, evaluates the result, keeps or discards the change, and repeats. Preparation, data loading, and evaluation remain fixed in prepare.py; the human supplies research instructions through program.md. This separates the experiment code an agent can change from the criterion used to judge it.[10]

Each autoresearch training run has a five-minute wall-clock budget, excluding startup and compilation, and is judged by validation bits per byte, where lower is better. The original implementation requires one NVIDIA GPU and reports testing on H100. Karpathy explicitly warns that results across different compute platforms are not comparable: the time budget makes the search specific to the available hardware.[10]

These language-model projects followed earlier work in vision-language research, classroom teaching, and industrial training and deployment. Karpathy's doctoral research at Stanford focused on convolutional and recurrent networks and the intersection of computer vision and natural language processing.[1]

Karpathy studied computer science and physics at the University of Toronto from 2005 to 2009, attending Geoffrey Hinton's class and reading groups. At the University of British Columbia from 2009 to 2011, he worked with Michiel van de Panne on learned controllers for physically simulated figures. At Stanford, his doctoral adviser was Fei-Fei Li.[1]

Karpathy and Fei-Fei Li's Deep Visual-Semantic Alignments for Generating Image Descriptions, submitted in December 2014 and revised in April 2015, addressed how to connect image regions with the words describing them. Their model mapped convolutional-network representations of image regions and bidirectional recurrent representations of words into a shared embedding space. It inferred correspondences from whole-image captions, then used those correspondences to train a recurrent model that generated descriptions of individual regions.[3]

The collaborators reported abandoning an earlier approach based on off-the-shelf language parsers and explicit relationships between detected objects. Parsing errors, slow inference, and weak spatial signals made that approach difficult to use. They found that the bidirectional recurrent model improved performance and reduced code complexity. Their final caption generator prioritized simplicity and speed, while acknowledging a performance cost against some more complex contemporary models.[3]

Teaching developed alongside the research. Karpathy designed and was the primary instructor of Stanford's inaugural Winter 2015 CS231n course, with Fei-Fei Li as a fellow instructor and teaching assistants including Justin Johnson. Students learned to implement, train, and debug neural networks through practical assignments and a final project. The course also planned publicly available notes, slides, and assignments for learners outside Stanford.[1][2]

His May 2015 essay The Unreasonable Effectiveness of Recurrent Neural Networks made sequence modeling tangible through generated prose, markup, and code. Karpathy recalls that he and labmate Justin Johnson trained a multilayer LSTM on 16 megabytes of algebraic-geometry LaTeX. The output resembled mathematical exposition, but required manual syntax repairs to compile, and he described the mathematics as hallucinated. One sample began a proof environment and ended a lemma environment—an example of the model losing a longer-range dependency.[4]

Karpathy was a research scientist and founding member of OpenAI from 2015 to 2017. As Tesla's Director of AI from 2017 to 2022, he led the computer-vision team for Autopilot and, briefly, Optimus. His team handled in-house data labeling, neural-network training, and deployment on Tesla's custom inference chip. He returned to OpenAI in 2023–2024 and built a team working on midtraining and synthetic-data generation.[1]

During his Tesla period, Karpathy published Software 2.0 in November 2017. He framed neural-network weights as programs found through optimization: engineers specify desired behavior and an architecture, then training searches for a program that works. In his account, much practical development therefore shifts toward curating, growing, and cleaning labeled datasets. He also identified the costs of learned behavior: poor interpretability, inherited data biases, silent failures, and adversarial vulnerabilities.[5][1]

His April 2019 A Recipe for Training Neural Networks translated engineering experience into a debugging process. Karpathy warned that a network can train without exceptions while containing logical errors—for example, an autoregressive model accidentally receiving the target it is supposed to predict. He recommends inspecting the dataset, establishing simple baselines, overfitting a tiny batch, visualizing inputs immediately before model execution, and tracing gradients to check dependencies. Complexity should increase only after explicit hypotheses have been tested.[6]

Karpathy founded Eureka Labs in July 2024. Its launch announcement proposed teacher-designed courses supported by AI teaching assistants, beginning with an undergraduate-level course, LLM101n, that would guide students through training their own AI. Online materials and digital and physical cohorts were announced plans. In October 2025, Karpathy described nanochat as the course's capstone project and said he was still building the intermediate material and needed to assemble a teaching-assistant team.[16][11][12]

Karpathy described education in his October 2025 interview with Dwarkesh Patel as the technical problem of “building ramps to knowledge.” For nanochat, that meant making a complete system simple enough to inspect, while also recognizing that a finished repository does not show learners how to construct it incrementally. TIME's September 2024 profile recognized his educational influence through Stanford teaching and explanations of building GPT models from scratch.[12][16]

In his April 2025 account of MenuGen, Karpathy described building an application that turned a menu photograph into generated pictures of its dishes. He reported writing no code directly, using Cursor and Claude for the implementation. The quick local prototype was followed by difficult integration work involving APIs, deployment, authentication, and payments. He caught a consequential design mistake: generated code matched Stripe purchases to Google accounts by email, although those addresses could differ. He directed a correction using unique user IDs passed through request metadata.[17]

Karpathy's October 2025 assessment of coding models was task-specific. He found them useful for boilerplate, report generation, and translating a tokenizer into Rust with a Python reference he understood and tests. For nanochat's unusually arranged training code, he said the models repeatedly imported assumptions from more conventional implementations: “It’s not boilerplate code.”[12]

The same interview connected his caution about demonstrations to industrial experience. Karpathy recalled a convincing Waymo ride around 2014 that made self-driving seem close, followed by years of work across the field. He described reliability as a “march of nines,” with successive reductions in failure requiring further engineering, rather than treating one successful demonstration as readiness for broad use. His concise assessment was: “I’m very unimpressed by demos.”[12]

Karpathy also proposed a “cognitive core” in October 2025: a model that retains problem-solving procedures and a basic knowledge curriculum while relying more on external lookup for esoteric facts. He presented this as a research hypothesis, questioning whether extensive memorization helps models reason beyond familiar internet examples.[12]

On May 19, 2026, Axios reported that Karpathy was joining Anthropic.[14]