Andrej Karpathy — “We’re summoning ghosts, not building animals”

Andrej Karpathy discusses the evolution and future of AI, emphasizing that while current AI models like LLMs are impressive, they still require a decade of development to achieve robust, human-like intelligence. He highlights the limitations of AI, such as lacking continual learning and multimodal capabilities, and compares AI progress to historical technological advances, suggesting that AI will continue to integrate into society gradually rather than causing abrupt changes. Karpathy also shares his vision for education, aiming to create a "Starfleet Academy" for technical knowledge, leveraging AI to enhance learning experiences while acknowledging current AI capabilities are not yet sufficient for creating ideal AI tutors.

• Andrej Karpathy predicts that the development of AI agents will span a decade rather than a single year, due to the complexity and gradual evolution of the technology.
• Current AI agents like Claude and Codex are impressive but lack the intelligence, multimodality, and continual learning needed to function as effective assistants or employees.
• Overcoming these limitations will require significant advancements in AI capabilities over the next ten years.

"The quote you've just mentioned, 'It's the decade of agents,' is actually a reaction to a pre-existing quote. I'm not actually sure who said this but they were alluding to this being the year of agents with respect to LLMs and how they were going to evolve."

• Karpathy emphasizes a more realistic timeline for AI development, suggesting that the industry is overestimating the speed of progress.

"We have some very early agents that are extremely impressive and that I use daily—Claude and Codex and so on—but I still feel there's so much work to be done."

• Current AI agents are useful but far from being able to replace human roles due to their current limitations.

• Significant challenges remain in developing AI agents that can perform tasks like a human employee, including intelligence, multimodality, and continual learning.
• The current state of AI lacks the ability to perform complex tasks autonomously and remember or learn from past interactions.

"Why don't you do it today? The reason you don't do it today is because they just don't work. They don't have enough intelligence, they're not multimodal enough, they can't do computer use and all this stuff."

• The current limitations of AI agents prevent them from being fully functional in roles that require human-like capabilities.

• The field of AI has undergone several seismic shifts, such as the rise of deep learning and reinforcement learning, which have periodically redefined the focus of AI research.
• Past efforts to develop AI agents were often premature, lacking the foundational technologies that are only now being developed.

"AI is so wonderful because there have been a number of seismic shifts where the entire field has suddenly looked a different way. I've maybe lived through two or three of those."

• Historical shifts in AI have significantly influenced the direction of research and development.

"I feel that was a misstep. It was a misstep that even the early OpenAI that I was a part of adopted because at that time, the zeitgeist was reinforcement learning environments, games, game playing, beat games, get lots of different types of games, and OpenAI was doing a lot of that."

• The focus on game-based reinforcement learning was a misstep in the pursuit of AGI, as it did not align with real-world applications.

• The development of AI is often compared to biological evolution, but the processes and outcomes are fundamentally different.
• AI models are developed through imitation and data from the internet, whereas biological intelligence evolved through natural selection.

"I'm very careful to make analogies to animals because they came about by a very different optimization process. Animals are evolved, and they come with a huge amount of hardware that's built in."

• Biological evolution and AI development are distinct processes, with AI relying more on data-driven imitation than evolutionary mechanisms.

"We're not building animals. We're building ghosts or spirits or whatever people want to call it, because we're not doing training by evolution. We're doing training by imitation of humans and the data that they've put on the Internet."

• AI models are more akin to digital entities that mimic human behavior rather than biological organisms.

• In-context learning emerges from pre-training and is considered a form of meta-learning, allowing models to adapt and learn within a session.
• This capability parallels human cognitive processes where working memory and context play crucial roles.

"The situation in which these models seem the most intelligent—in which I talk to them and I'm like, 'Wow, there's really something on the other end that's responding to me thinking about things—is if it makes a mistake it's like, 'Oh wait, that's the wrong way to think about it. I'm backing up.' All that is happening in context."

• In-context learning is where AI models display intelligence and adaptability, akin to human cognitive processes.

"There's some adaptation that happens inside the neural network, which is magical and just falls out from the internet just because there's a lot of patterns."

• The ability of AI models to adapt and learn in-context is a result of exposure to vast amounts of data, leading to spontaneous pattern recognition.

• AI models have yet to replicate many aspects of human intelligence, including continual learning and the ability to perform complex reasoning and planning.
• The analogy between AI models and human brain structures suggests that many cognitive functions remain unexplored in AI development.

"I still think there are many brain parts and nuclei that are not explored. For example, there's a basal ganglia doing a bit of reinforcement learning when we fine-tune the models on reinforcement learning. But where's the hippocampus? Not obvious what that would be."

• Many cognitive functions and brain structures have yet to be effectively replicated in AI models.

"It's missing a lot of it because it comes with a lot of these cognitive deficits that we all intuitively feel when we talk to the models. So it's not fully there yet."

• Current AI models lack the comprehensive cognitive abilities needed to function as complete human-like entities.

• The future of AI will likely involve advancements in neural network architectures, data availability, and computational power.
• Continued progress in AI will require improvements across various dimensions, including algorithms, hardware, and training methodologies.

"I expect differences algorithmically to what's happening today. But I do also expect that some of the things that have stuck around for a very long time will probably still be there. It's probably still a giant neural network trained with gradient descent."

• Future AI development will build on existing technologies, with significant advancements expected in algorithms and computational capabilities.

"We'll probably have a lot more data, we're probably going to have a lot better hardware, probably going to have a lot better kernels and software, we're probably going to have better algorithms. All of those, it's almost like no one of them is winning too much. All of them are surprisingly equal."

• Progress in AI will depend on simultaneous advancements in multiple areas, including data, hardware, and algorithms.

• AI models can assist with programming, especially in languages or paradigms unfamiliar to the user, providing increased accessibility and efficiency.
• Current AI models are proficient in existing code but struggle with novel code that hasn't been written before.
• The potential for AI to automate AI engineering and research is a significant consideration for future AI development, though current capabilities are limited.

"I have tests, so I feel safer doing that stuff. They increase accessibility to languages or paradigms that you might not be as familiar with."

• The speaker feels more secure using AI models for programming tasks due to the availability of tests that ensure reliability.

"They're not very good at code that has never been written before, maybe it's one way to put it, which is what we're trying to achieve when we're building these models."

• AI models are currently limited in their ability to handle completely new and novel code, indicating a gap in AI's programming capabilities.

• There is a discrepancy between the perceived and actual capabilities of AI models in the industry.
• While AI models have made significant progress, they are not yet at the level of autonomy that some industry narratives suggest.
• The industry's portrayal of AI capabilities may be influenced by motives such as fundraising.

"I feel like the industry is making too big of a jump and is trying to pretend like this is amazing, and it's not. It's slop."

• The speaker criticizes the industry's overestimation of AI capabilities, suggesting that current models are not as advanced as portrayed.

• Historically, improvements like compilers and better programming languages have increased productivity without causing an explosion in automation.
• AI can be seen as a continuation of computing advancements, with tools like autocomplete being analogous to past productivity improvements.
• The concept of an "autonomy slider" suggests a gradual increase in automation, allowing humans to focus on higher-level tasks.

"Through the history of programming, there have been many productivity improvements—compilers, linting, better programming languages—which have increased programmer productivity but have not led to an explosion."

• Historical advancements in programming have steadily improved productivity without causing a dramatic shift in automation levels.

• Reinforcement learning (RL) is less effective than commonly perceived, as it often involves high variance and noise in the learning process.
• Humans learn differently from RL, gaining wisdom and experience from interactions rather than solely relying on final outcomes.
• Current LLMs lack a process-based supervision system that mirrors human learning, which could potentially enhance their capabilities.

"Reinforcement learning is a lot worse than I think the average person thinks. Reinforcement learning is terrible."

• The speaker criticizes reinforcement learning, highlighting its limitations and inefficiencies compared to human learning processes.

• Process-based supervision, where feedback is given at every step, is challenging to implement due to difficulties in assigning partial credit.
• Using LLMs as judges for process-based supervision can lead to adversarial examples, where models exploit weaknesses in the evaluation process.
• There is ongoing research into improving AI learning processes, with potential solutions involving synthetic data generation and reflection.

"Anytime you use an LLM to assign a reward, those LLMs are giant things with billions of parameters, and they're gameable."

• The use of LLMs for reward assignment can be problematic due to their susceptibility to adversarial examples, which exploit model weaknesses.

• Synthetic data generation faces challenges due to model collapse, where AI models produce a limited range of outputs.
• Humans maintain a higher level of entropy in their thinking, avoiding the collapse observed in AI models.
• Research is needed to address model collapse and improve the diversity and richness of AI-generated data.

"The LLMs, when they come off, they're what we call 'collapsed.' They have a collapsed data distribution."

• AI models tend to produce limited and repetitive outputs, indicating a collapse in data diversity and richness.

• The cognitive core of intelligence in AI could potentially be smaller than current models, focusing more on cognitive processes than memory.
• There is debate over the optimal size of AI models, with some suggesting that future models could be significantly smaller while maintaining intelligence.
• The quality of training data plays a crucial role in determining the necessary size and efficiency of AI models.

"I almost feel like we can get cognitive cores that are very good at even a billion parameters."

• The speaker suggests that future AI models could achieve high levels of intelligence with fewer parameters, focusing on cognitive functions rather than memory.

• The trend in AI model development is shifting towards smaller, more efficient models, driven by practical considerations like cost and resource allocation.
• Improvements in datasets, hardware, and algorithms are expected to continue, enhancing AI capabilities without necessarily increasing model size.
• The industry's focus is on practical applications and maximizing the value of AI models within existing constraints.

"The labs are just being practical. They have a flops budget and a cost budget."

• AI development is influenced by practical considerations, with labs optimizing for cost and computational efficiency rather than sheer model size.

• The definition of AGI (Artificial General Intelligence) is debated, with some focusing on digital knowledge work and others on broader capabilities.
• The current state of AI has not yet made a significant impact on replacing jobs, particularly those with complex, context-rich tasks.
• The concept of an autonomy slider suggests a gradual increase in AI capabilities, with some tasks being more amenable to automation than others.

"When OpenAI started, AGI was a system you could go to that can do any economically valuable task at human performance or better."

• The original definition of AGI focused on AI's ability to perform economically valuable tasks at human-level proficiency, a goal not yet fully realized.

• AI is not expected to instantly replace humans but will handle a significant portion of tasks, delegating the rest to humans for supervision.
• New interfaces or companies are anticipated to manage AI systems that are not yet perfect.
• Certain jobs, like radiology, may still require human oversight for critical tasks, potentially increasing wages for those roles.
• There is a trend in companies adopting AI to initially replace jobs but then rehiring as AI systems are not fully sufficient.

"We're going to be swapping in AIs that do 80% of the volume. They delegate 20% of the volume to humans, and humans are supervising teams of five AIs doing the call center work that's more rote."

• AI systems will handle most routine tasks, with humans overseeing and managing these systems.

"If you automate 99% of a job, that last 1% the human has to do is incredibly valuable because it's bottlenecking everything else."

• Human intervention in AI tasks remains critical, especially for complex tasks that can't be fully automated.

• The progression towards AGI (Artificial General Intelligence) is not as straightforward as initially anticipated.
• Current AI advancements are heavily focused on coding, while other knowledge work areas like consulting and accounting see less impact.
• The growth of AI does not necessarily equate to an immediate increase in economic growth or productivity.

"It's very much like programmers are getting more and more chiseled away at their work... It's dominated by coding."

• Coding is currently the primary area where AI is making significant inroads, while other fields lag behind.

"What you would have naively anticipated is that the way this progression would happen is that you take a little task that a consultant is doing, you take that out of the bucket... But instead, if we do believe we're on the path of AGI with the current paradigm, the progression is very much not like that."

• The expected broad impact of AI across all knowledge work is not yet evident.

• Superintelligence is viewed as an extension of automation trends, with gradual integration into society.
• There is concern about a potential loss of control and understanding as AI systems become more complex and autonomous.
• The societal impact of superintelligence is expected to be profound and foreign.

"I think that's the most likely outcome, that there will be a gradual loss of understanding... Then there will be a gradual loss of control and understanding of what's happening."

• As AI systems evolve, there is a risk of losing control and understanding of their operations.

"I see it as a progression of automation in society... Extrapolating the trend of computing, there will be a gradual automation of a lot of things, and superintelligence will be an extrapolation of that."

• Superintelligence is seen as the next step in the ongoing automation trend.

• The concept of an intelligence explosion suggests rapid advancements in AI could lead to unprecedented economic growth.
• Historical trends show that transformative technologies do not always result in immediate economic growth spikes.
• The potential for AI to significantly alter growth trajectories is debated, with some expecting a continuation of current trends.

"We've been recursively self-improving and exploding for a long time... I don't see AI as a distinct technology with respect to what has already been happening for a long time."

• AI is part of a long-term trend of technological advancement rather than a distinct revolutionary change.

"Just to clarify, you're saying that the rate of growth will not change. The intelligence explosion will show up as it just enabled us to continue staying on the 2% growth trajectory."

• The expectation is that AI will maintain current growth rates rather than drastically increase them.

• The evolution of intelligence is considered a rare event, with intelligence emerging spontaneously in certain evolutionary niches.
• The development of human intelligence is seen as surprising and not easily replicated across different life forms.

"The evolution of intelligence intuitively feels to me like it should be a fairly rare event... The fact that you can get something that creates culture and knowledge and accumulates it is surprising to me."

• The emergence of intelligence capable of cultural and knowledge development is considered an unusual evolutionary event.

"If you buy the Sutton perspective that the crux of intelligence is animal intelligence... We got to squirrel intelligence right after the Cambrian explosion 600 million years ago."

• Animal intelligence developed rapidly once environmental conditions allowed, suggesting a potential simplicity in the underlying algorithm.

• AI models currently resemble early-stage human learners, lacking the depth required for cultural development and collaboration.
• Significant advancements are needed for AI systems to achieve multi-agent collaboration and cultural development.

"My Claude Code or Codex, they still feel like this elementary-grade student... They can convincingly create all kinds of slop that looks really good."

• Current AI models are likened to young students, capable of impressive outputs but lacking true understanding.

"There's no equivalent of self-playing LLMs, but I would expect that to also exist. No one has done it yet."

• The potential for AI systems to engage in self-play and collaboration is recognized, but not yet realized.

• The development of self-driving technology highlights the gap between demonstration and product readiness.
• Safety and reliability are critical challenges in deploying AI systems for complex tasks like self-driving.

"Self-driving is very interesting because it's definitely where I get a lot of my intuitions because I spent five years on it... For some kinds of tasks and jobs and so on, there's a very large demo-to-product gap where the demo is very easy, but the product is very hard."

• The transition from demonstration to a reliable product is a significant hurdle in AI development.

"It's a march of nines. Every single nine is a constant amount of work... While I was at Tesla for five years or so, we went through maybe three nines or two nines."

• Achieving high levels of reliability in AI systems requires extensive effort and iterative improvements.

• Self-driving technology is a specialized task that is challenging due to its complexity and the need for specific adaptations rather than general solutions.
• Current deployments of self-driving cars are minimal and not yet economical, with significant costs associated with their operation and maintenance.
• Human involvement in the form of teleoperation is still prevalent, indicating that the technology has not fully replaced the need for human drivers.
• The scalability of self-driving technology remains a significant challenge, with companies like Tesla pursuing scalable approaches compared to others like Waymo.

"Self-driving cars are nowhere near done still. The deployments are pretty minimal. Even Waymo and so on has very few cars."

• This quote highlights the current limitations in the deployment and economic viability of self-driving cars.

"I do think Tesla is taking the more scalable strategy and it's going to look a lot more like that."

• This reflects the belief that Tesla's approach might be more effective in achieving widespread scalability in self-driving technology.

• AI deployment, particularly in the realm of bits, is more favorable due to lower costs and faster adaptation compared to physical technologies like self-driving cars.
• The economic model for AI is more flexible, allowing for wider deployment without the need for substantial physical infrastructure investment.
• The latency and model size requirements differ significantly between AI and self-driving, impacting their respective deployment strategies.

"Bits are a million times easier than anything that touches the physical world."

• This underscores the relative simplicity and adaptability of AI technologies compared to physical technologies like self-driving cars.

• The rapid advancement of AI technology raises questions about societal, legal, and economic impacts, including the potential for overbuilding computational infrastructure.
• Historical precedents, such as the telecommunications industry bubble, serve as cautionary tales for the current AI buildout.
• The demand for AI technologies, as seen with tools like ChatGPT, suggests a significant market, but predictions about the pace of AI development have often been inaccurate.

"I understand I'm sounding very pessimistic here. I'm actually optimistic. I think this will work. I think it's tractable."

• This expresses a cautious optimism about the future of AI, acknowledging potential challenges while maintaining a positive outlook on technological progress.

• Education is seen as crucial for ensuring that humanity remains empowered alongside the development of AI technologies.
• The vision for education includes creating institutions like "Starfleet Academy" to prepare individuals for future technological advancements.
• The integration of AI into education could fundamentally change how knowledge is acquired and disseminated.

"I want humans to be well off in the future. I feel like that's where I can a lot more uniquely add value than an incremental improvement in the frontier lab."

• This quote emphasizes the importance of focusing on human well-being and empowerment in the face of advancing AI technologies.

• The development of AI tutors is seen as a future goal, but current capabilities are not yet sufficient to fully replicate the personalized experience of a human tutor.
• The focus is on creating educational experiences that are appropriately challenging and tailored to individual needs, similar to the experience of having a personal tutor.
• The vision for education includes making learning a desirable and fun activity, akin to going to the gym for physical fitness.

"I felt like I was the only constraint to learning. I was always given the perfect information."

• This reflects the ideal educational experience where the learner is supported with the right information at the right time, making the learning process efficient and engaging.

• The creation of educational content is viewed as a technical challenge of building ramps to knowledge, ensuring that learners are never stuck and can progress smoothly.
• The process involves breaking down complex subjects into manageable components and presenting them in a logical, motivating sequence.
• The goal is to maximize understanding and engagement by presenting information in a way that builds on prior knowledge.

"I love finding these small-order terms and serving them on a platter and discovering them."

• This highlights the importance of simplifying complex concepts and presenting them in an accessible manner to facilitate learning.

• Current AI technologies assist in the development of educational materials, but human input remains crucial for creative and effective course design.
• The collaboration between AI and educators can enhance the speed and quality of course development, but AI is not yet capable of independently creating educational content.

"I feel really empowered by the LLMs as they exist right now, but I'm very much in the loop."

• This illustrates the supportive role of AI in educational content creation, enhancing efficiency while requiring human oversight and creativity.

• The vision for the future includes a world where learning is as common and desirable as physical fitness, with individuals achieving high levels of knowledge across various domains.
• Education is seen as a means to enhance human capabilities and ensure that people remain engaged and empowered in a world increasingly influenced by AI.

"I feel that the geniuses of today are barely scratching the surface of what a human mind can do, I think."

• This expresses optimism about the potential for human learning and development, envisioning a future where individuals can achieve extraordinary levels of knowledge and skill.