Unsupervised Learning with Jacob Effron

This episode with Lukasz Kaiser, co-author of the seminal "Attention Is All You Need" transformer paper and former researcher at both Google Brain and OpenAI, is a wide-ranging conversation about the fundamental limits of current AI architectures and whether transformers will continue to dominate or eventually give way to something new. Lukasz brings a rare dual perspective: deep belief in how far the current paradigm has taken us (he's an enthusiastic daily Codex user who's seen 10x productivity gains in his own research), while maintaining genuine intellectual humility about whether transformers can truly generalize the way humans do. The episode weaves together questions about data efficiency, the non-verifiable RL frontier, the coding agent revolution, the open vs. closed source gap, and what the next architectural leap might look like: all filtered through the lens of someone who helped build the foundation the entire field is standing on.

 

(0:00) Intro

(1:12) Transformers vs. Human Learning

(8:37) How Do We Get Physical World Generalization?

(10:52) What Comes After Transformers

(13:59) How Much Have Agents Improved Lukasz's AI Research Productivity?

(17:21) How Close Is an AI Research Intern?

(26:06) RL Beyond Verifiable Tasks

(35:38) App Companies: Build Models or Lean on Labs?

(46:21) Multimodal Is Still Missing Something

(49:46) OpenAI's Bet on Reasoning

(55:26) The AI Coding Wars

(59:26) Focus vs. Keeping Embers Burning

(1:02:09) Open Source vs. Closed Source Gap

(1:05:15) Quickfire


With your host:
@jacobeffron
- Managing Director at Redpoint

Sebastian Mallaby spent three years and 30+ hours interviewing Demis Hassabis in the back of a British pub to write The Infinity Machine, and the conversation uses that reporting to surface the most underexplored figure in AI. Demis founded the original AI lab in 2010, won a Nobel Prize, runs models that consistently top the leaderboards, and yet remains so unrecognized that Sebastian's own publisher worried no one would buy a book with his face on the cover. 

The throughline is a paradox: Demis tried to prevent the AI race we're now all living through, and now finds himself one of its central protagonists. He used to believe a single lab could carry the safety burden to AGI; he now sees safety as a collective action problem only governments can solve. He hedged DeepMind's research bets across every promising direction, and as a result missed the two most consumer-defining moments in modern AI — ChatGPT and Claude Code. He nearly spun DeepMind out of Google with a secret $1B Reid Hoffman pledge backing him, but never used the leverage and stayed — and won a Nobel Prize the next year.

The episode also zooms out to the structural forces shaping the race — why hyperscalers can't out-recruit concentrated-bet labs, why Sebastian gives OpenAI roughly 50/50 odds of being absorbed by next summer, why he thinks Anthropic should IPO right now, and what the personal histories between Demis, Elon, and Sam reveal about who actually trusts whom.

 

(0:00) Intro

(2:04) Was the AI Race Inevitable?

(4:03) The 2015 Safety Summit Backfire

(7:15) Can Governments Actually Fix This?

(9:26) How the World Misread DeepMind

(11:27) Why Google Never Makes the Concentrated Bet

(15:51) Project Mario: The Secret Spinout Plan

(19:43) What Demis Actually Regrets

(23:46) Venture Startups vs. Tech Behemoths

(27:50) Controlling the Narrative

(30:40) The Talent War and Hiring Brand

(34:08) David Silver and the RL True Believers

(38:21) Demis, Elon, and the Evil Genius Feud

(42:39) Great Man Theory vs. Inevitability

(45:00) What Demis Didn't Want Published


With your host:
@jacobeffron
- Managing Director at Redpoint

Oriol Vinyals, VP of Research at Google DeepMind and co-lead of the Gemini program, joins Jacob the day after Google I/O to unpack the research underpinning Google's latest announcements and where frontier AI is heading. The conversation moves from world models (why Google has uniquely bet on them as a path to AGI, what the "GPT moment" for video and images would look like, and how they connect to robotics and simulation) to agents (the Spark release, why the system and model need to be optimized jointly, and why scaffolding will eventually be written by models themselves). Oriol gets into the mechanics of memory in models, drawing on his cognitive neuroscience background to argue that file-system-style non-parametric memory is more practical than baking memory into weights at serving scale. He shares his views on the limits of RL today (LLMs are data-limited in a way that game-playing RL never was), why training on narrow domains like math and code generalizes surprisingly well, and what a true "Move 37" moment for science or ML research would look like. Throughout, he reflects on the unique advantages of being inside Google (TPU co-design, end-to-end revenue stability, the merger of Brain and DeepMind), the trade-offs between focus and exploration in research orgs, and why he believes AGI in some meaningful sense may already be here, even if the goalposts keep moving.

 

(0:00) Intro 

(1:36) Why World Models 

(4:21) The GPT Moment for Video 

(7:51) What Makes Omni a World Model 

(10:04) World Models & Robotics 

(12:37) Evaluating Physics in AI 

(14:51) Consumer Agents & Spark 

(18:39) Scaffolding & the Bitter Lesson 

(22:06) Memory & Continual Learning 

(26:54) Research Bets Inside Big Labs 

(32:30) Post-Training RL is Greenfield 

(35:57) What Real Intelligence Looks Like 

(39:11) RL Generalization 

(43:00) Advice for Founders 

(46:40) Can AI Truly Innovate? 

(49:48) Recursive Self-Improvement 

(52:14) Quickfire


With your host:
@jacobeffron
- Managing Director at Redpoint

Yann LeCun, Turing Award winner and former Chief AI Scientist at Meta, joins Jacob Effron. The conversation centers on Yann's contrarian thesis that LLMs are a dead-end on the path to human-level intelligence, despite being useful products — because they can't predict the consequences of their actions, can't plan, and fundamentally can't model the messy, high-dimensional real world. He unpacks his alternative architecture, JEPA (Joint Embedding Predictive Architecture), which learns abstract representations rather than generating pixel-level predictions, and explains why this approach is essential for robotics, industrial applications, and any system that needs to operate beyond the substrate of language. Yann also reveals the real story behind his departure from Meta (he had zero technical influence on Llama, contrary to public narrative), the genesis of his Tapestry project for sovereign open-source AI, why he believes LLMs are intrinsically unsafe, where he diverges from his fellow Turing laureates Hinton and Bengio, and why he predicts the industry will recognize the paradigm shift by early 2027. Throughout, he offers candid reflections on the tension between research and product at major labs, and why he intentionally headquartered AMI Labs in Paris with zero Silicon Valley VC money.

 

(0:00) Introduction 

(01:45) Why LLMs Aren't the Path to Intelligence 

(07:51) AMI and World Models 

(12:07) The JEPA Architecture Explained 

(15:55) Problems with Robotics Models Today 

(20:37) Silicon Valley Herd Behavior 

(28:18) Tapestry: Sovereign AI for the Rest of the World 

(35:49) OpenAI Is the Next Sun Microsystems 

(40:51) Why Yann's Views Diverged from Hinton & Bengio 

(44:32) LLMs Are Intrinsically Unsafe 

(58:00) Why Yann Left Meta 

(1:00:26) Reflections on FAIR 

(1:12:11) Advice for PhD Students

 

LeWorldModel Paper: https://arxiv.org/abs/2603.19312

 

With your host: 

@jacobeffron 

- Partner at Redpoint


With your host:
@jacobeffron
- Managing Director at Redpoint

This episode is a wide-ranging conversation between Jacob and Swyx (Shawn Wang), an AI engineer, podcaster, and now operator at Cognition, who sits at a uniquely informed intersection of builder, investor, and community organizer in the AI world. The two cover the current state of the AI engineering zeitgeist: from the stabilization of agent infrastructure and the surprising stickiness of Claude Code, to the competitive dynamics of the AI coding wars, the rise of open models, the threat to traditional SaaS, and the frontier questions around world models, memory, and what it actually means for AI to "understand" something. The episode is grounded in practitioner-level candor, with Swyx offering real takes from running AIE conferences, working inside Cognition, and thinking deeply about what the next wave of AI-native software development looks like.

 

(0:00) Intro

(1:17) What the Top AI Engineers Are Thinking About

(2:13) Has AI Infra Finally Stabilized?

(6:39) When Does Doing RL In-House Make Sense?

(11:26) Why Selling Dev Tools to Agents is Different

(17:18) AI Coding Wars

(29:04) Consumer AI Plateau

(30:22) Codex vs Claude Code

(44:52) Future of Open Models

 

With your co-hosts: 

@jacobeffron 

- Partner at Redpoint, Former PM Flatiron Health 

@patrickachase 

- Partner at Redpoint, Former ML Engineer LinkedIn 

@ericabrescia 

- Former COO Github, Founder Bitnami (acq’d by VMWare) 

@jordan_segall 

- Partner at Redpoint


With your host:
@jacobeffron
- Managing Director at Redpoint