Interconnects

This episode is with Jean-Stanislas “JS” Denain of Epoch AI, who leads their Insights Team and is one of the people I find myself debating the state and trajectory of AI with more and more. We’ve had follow-on discussions of many of my favorite recent posts online and/or in private, so I wanted to dig into the nuance in a public episode.
A big takeaway of this podcast is how JS and I both have so much uncertainty with exactly where we are heading, and this was our best effort at stating our observations today.
Chapters / topics include:
* 00:00 Predictions for RSI
* 18:15 The role of robotics in an AI acceleration
* 24:20 How far behind are Chinese models?
* 27:39 Does distillation explain the gap?
* 40:58 What Chinese job postings reveal about their labs
* 48:13 Are open or closed models safer?
* 58:10 How Epoch AI ticks
* 1:00:55 What a frontier post-training recipe looks like
Enjoy!
More from JS: Epoch AI profile and writing, X, LinkedIn
Listen on Apple Podcasts, Spotify, and where ever you get your podcasts. For other Interconnects interviews, go here.
Transcript
00:00:00 Nathan Lambert: I’m here with JS Denain, who is a senior researcher at Epoch AI. He leads the insights team. He is one of the people who I feel like I get the best feedback on my writing from, whether it’s from US-China AI capabilities, now RSI. And I just wanted to open this discussion and honestly go deeper with him, trying to understand how he thinks about these various things. And I think you have a very useful, moderate point of view, which I feel like you’re probably a step further into what would be called faster scenarios for AI progress. But let’s get into this, and it’s like, what measurements do you think OpenAI and Anthropic are seeing when we get all these proclamations on RSI happening very imminently?
00:00:47 JS Denain: Yeah. I think, so there’s the measurements they’ve published, right? So, OpenAI and Anthropic both had blog posts, I mean, Anthropic two at least, on the effect AI has on accelerating AI progress. I think at least the things they publish, I don’t think are super strong evidence of imminent self-sustaining acceleration AI capabilities, or full automation of the job of AI researcher. But I think the kinds of things we see are, I think probably the most striking thing I saw in the OpenAI blog post was increasing usage of AI systems in model deployment, like the increase in spending on Codex that we saw. And it’s kind of unclear how exactly to interpret this, because maybe it’s a measurement artifact where they’re only looking at Codex, but in fact, there was a bunch of ChatGPT usage before from the researchers. But overall, that plot, for example, just shows a 2X a month increase in Codex spending by researchers, and that does seem to me to be some evidence of they’re getting a lot of value out of this probably. I don’t think this is strong evidence that in six months we have a software intelligence explosion.
00:01:57 Nathan Lambert: Do you think this is the same? So, what is the information they have internally relative to what we have? And this is obviously hypothetical. We don’t have this internal information. Because I get the sense that a lot of people are more scared in their updates from the labs than the information we have. And I try to take this very seriously of, what will they be seeing that is making the acceleration of risk comments go faster, and how much of this is material evidence versus how cultures evolve over time? And I’m much more interested in evidence.
00:02:29 JS Denain: Yeah. So two things. I think, first of all, I guess I don’t think, I don’t know, right? I don’t have full information here. I don’t currently think that either there’s some specific thing that people at OpenAI or Anthropic are seeing right now that we don’t have access to that warrants being way more freaked out about this. I also don’t think that... I think the public evidence we have right now, more general on AI progress and just a priori case for this being an important dynamic, I think is enough. I think to care about this particular dynamic of AI accelerating AI progress, that being a big deal and worth tracking. And then it’s kind of unclear what the urgency is of when the feedback loop really kicks in. So, basically on the what is there on the inside that people have access to, I could give examples of kinds of metrics, right, they could be looking at. It’s plausible that we have access to the capabilities of AI systems, but internal teams have their KPIs, and maybe they’re seeing compute multipliers in the pre-training team or other kinds of metrics that people are tracking going crazy. And then the combination of this plus some intuitions of how the different outputs of different teams combine yields a prediction on the trend in actual performance of the end AI systems. So that could be an early warning sign. It’s unclear to me that the recent discourse we’ve seen is evidence of things going crazy on those metrics.
00:04:09 Nathan Lambert: And how do you think of the link between RSI and existential risk? So I would posit that you agree. I think that there are very real risks of AI, and I’m curious on how you think these, what I would describe as very, very early measurements change anything on the scope of risk. Because I don’t think if you had asked people six months ago, it would be as immediate to x-risk among people are very reasonable. I think there’s more people that are reasonable talking about x-risk again, which was a little surprising to me.
00:04:43 JS Denain: Yeah. So, okay, my sense is something like... So personally, I feel very uncertain about this, but I do feel, yeah, basically bought into there’s, I know Evan Hubinger was like, at least 10% of x-risk within I don’t know what timeframe. I think I’m like, yeah, I know, and this seems pretty reasonable over a decade-long timeframe. I just feel extremely uncertain about it, but I’m definitely very worried about this. Now, why am I worried about this, and where do I think the disagreements come from? And then how do I relate this to the early sense of RSI? My sense, I’m kind of a capabilities theory of everything person. I think, and I think some people disagree here, but I really think that principal component of disagreement between everyone is how huge do the capabilities get, how soon, of AI systems? And I sort of agree that there’s other factors that come in play for how big your economic growth gets, also depend on the diffusion you get. And you could possibly you could think that capabilities are going to get crazy, but the AI system’s going to be just aligned and benign and stuff, and so there’s no huge risk. But my sense is concretely, when I look at the main kinds of disagreements between people, most of the people who I see who are very skeptical of those most extreme scenarios... I think just expect capabilities to not be as huge or as I think folks who are—
00:06:18 Nathan Lambert: What does being a capabilities maximalist look like in a few years? Because I think I’m probably on the skeptical side, so please continue.
00:06:28 JS Denain: I think it looks, for example, something like the AI 2027 scenario, right? I think it looks like the mechanism for this is AI is automating the AI research process, I think, and that’s a reason to pay attention to it. But in terms of effect on the real world, I think it’s like massive progress on robotics. I think a huge industrial explosion, AI systems are just managing factories. You have this kind of self-sustaining economy that just is able to make a large scientific progress much faster than you would have expected. And so I think concretely, the kinds of disagreements I would expect are on, yeah, if you have AI systems that are both very intelligent in the book smart sense, but also have been trained to have more affordances and use them astutely, have been trained to kind of manage projects in efficient ways and stuff like that. How big are the real-world bottlenecks to making very fast R&D progress or getting hard power over humans?
00:07:26 Nathan Lambert: Yeah. Can we go into some of these in detail? Did you listen to the Dwarkesh podcast with Charlie, Beren, and John?
00:07:32 JS Denain: Yes. Yeah.
00:07:33 Nathan Lambert: Yeah, because they had at the end, they had this section on various capability levels and timelines for getting them. And I feel like I agreed with most... I was very in agreement on the distribution they had up to this, and then was surprised by the timelines. And one of them was the 10X productivity for the AI researchers. And I think Beren and John were faster than I think. And my kind of statement is that I think the cycle from of having an idea and doing the experimentation to test it, I agree will be 10X faster very soon. But I don’t necessarily agree that I would say that AI researchers will be 10X more productive in net, which I would describe as the pace of the field’s complete understanding. And understanding is a different axis from just continuing to scale models. I think that’s one of my core confusions on the AI research side. So I’m just kind of curious how you think about this type of thing and how you might specify a 10X improvement in AI research into subcategories.
00:08:37 JS Denain: Yeah. So maybe there’s a scale you could have here, which is the end thing that you might care about is how much faster is AI research overall? Or how much faster is Anthropic’s overall output? And then Anthropic as a company is producing some things, and it’s doing in one year what it would have taken it 10 years to do. And that’s pretty different from individual researcher productivities, where I think you could... So if most of what AI researchers right now are doing is this loop that you were describing, then it’s possible that you get a 10X productivity improvement for the median researcher based on the tasks they’re doing right now. But first of all, that doesn’t mean you get a 10X productivity improvement for all researchers. And even if you did, right, there’s other bottlenecks that hit such that that needn’t convert into a 10X productivity improvement for Anthropic as a whole, right? You could have all the researchers be 10X more productive, but because of compute or other things, the company itself still doesn’t move as fast.
00:09:38 Nathan Lambert: I think an analogy I have is, I think that junior PhD students will be 10X as productive, but from the advisor’s perspective, their research agenda will not proceed 10X as fast. And it’s like to the extent that that contributes to Anthropic’s progress is another hard thing to jump on AI capabilities, where I think that listening to the Noam podcast. This is me, I’m just thinking, was thinking about this when writing about this, is there’s such an amount of inference compute coming online, and I think Dwarkesh highlights this very well, that it’s very hard for me to disambiguate massive speed-up in AI research from the fact that we have way more compute and can now much more effectively spend it on related problems. And I think we’re going to get all of this at once.
00:10:27 JS Denain: Yeah. So definitely I think this question of... So when you look at the OpenAI blog post they had on their acceleration, right, they do point out huge surge in Codex spending from researchers. Interestingly, actually, Codex spending in other parts of the company kind of had a huge surge in the spring and then kind of plateaued in the summer. But for researchers or the data team or engineers, it keeps growing, even accelerates sometimes. And so they point this out, and then they try to look at where there was an increase, which kinds of tasks had an increase in usage. And a lot of them are engineering tasks. There’s an increase in troubleshooting tasks. But they definitely point out that for a lot of the high-level strategic decision-making, they both anecdotally and also when they look at sessions, don’t seem to find a huge uplift in making better compute allocation decisions or deciding on research directions. And so to me, that’s pretty similar to the PI case. And so I think there’s a first question, which is, how much of an improvement... Imagine you just didn’t get that much AI uplift on that component of AI research, but the rest of AI research really went crazy. Then how much faster do things go? And the separate question is, I don’t know, how hard is this strategic decision-making? Can’t you just have a bit longer horizon RL? Or maybe you can bet on decent transfer from other fields where those kind of decisions are important, and then you do actually get that kind of uplift at the end.
00:11:59 Nathan Lambert: I think part of my intuition is that science will look so fundamentally different that it’s almost hard to put a number on it. And it’s the pre and post-AI era, and we’re just in the rapid transition to what is a new method, new way of doing science, because I think all the conferences are ready to burn down and struggle through the next few years. I hope that they collectively figure out a way to like add AI oversight into reviewing and things that are scalable, because they have so many slop papers that they need this type of gate. So I don’t really know. And I think on the capability side, I’m like, what an AI progress is like clearly translated into new capabilities. There are two things. One is like pre-training scaling laws. Our loss is proportional to like an exponential increase in compute. And on the research side, what we are doing is we’re shifting the line so that it has a better offset and potentially a better slope. And then on the other side is the RL environments, and I think the RL environments we’re building now are very comparable to valuable work. So I expect the AI models to get much, much better at like knowledge work that can be scoped. But I don’t know if we have a good process for like churning out an order of magnitude harder environments, which would be closer to like cure cancer, solve these open math problems. I think math is a case that we could talk about. But that’s kind of like, I think there are unknowns on scaling the like raw intelligence more than efficiency. So I’m very optimistic in scaling efficiency.
00:13:29 JS Denain: Yeah. I agree that like in some sense, right, like inference efficiency is like a very like hill-climbing task. It’s like pretty well-scoped. And yeah, so I mean, it’s already something that has very, very fast trends, but I could imagine those trends. Yeah, I imagine those trends will go even faster. It seems really like the kind of... I mean, indeed, we have evidence from OpenAI, right? Like, saving on like serving costs, et cetera, through like building better kernels and like if you’re new to this. They don’t give that many details, but that’s already happening. One thing I’m curious about actually in your case is like, so here’s one way of defining like Anthropic, for example, like accelerates overall, which is you could look at the like ECI trend in like, the ECI of the best Anthropic quality point in time. And you can like look at the current trend line and you can ask the question, like over the next year, will we see a 5X acceleration? Like, will the slope be like 5X larger than it was, say, in like 2025? And I think it’s like pretty likely we see like a huge increase in this because it is like kind of a legible like KPI that like, I mean, it’s not literally KPI, but it is a KPI that like the company is aiming for, modulo like safety considerations, et cetera. And I’m curious about whether you think it’s very unlikely we get this or whether it’s more like we might get this, but like if we do get this, it’s mostly that like ECI has been Goodharted as a metric and like the implications for like real-world capabilities aren’t that huge.
00:15:03 Nathan Lambert: I wouldn’t be surprised if we got this, but I think that it’s going to be like we’re on a slope and then we could get an uptick in slope of hill climbing, but then we like saturate what we know how to hill climb and then it goes to be lower. So it’s like all the things that we could measure I think are going to be getting pulled up very quickly by being measurable. And then we’re in the domain of like, how do we measure it? Because I think, like I talk to people that are trying... Like evals are so expensive to build now, and I do think that evaluations are going to be like how good and efficient it is at coding, how good and efficient it is at ML research, how good and efficient it is at knowledge work. But I don’t know how to like... Building those evals all seems tractable but hard. But then like how to make a breakthrough in fundamental chemistry seems really, really, really hard to measure. I was going to draw on like maybe frontier math as an example, but I think math is such an exception as like one of the most jagged pieces of AI. I think especially like open problems in mathematics are like the perfect target for rapidly improving AI because it’s like a falsifiable thing. And it’s like if we were to, say, see that in something that’s much more open-ended, I think I would update a lot. Or if the labs were like to come out and say, “Using Claude, we have a very, very big change in what our architecture of AI is,” to like there’s the famous like Jonathan Frankle–Sasha Rush bet, and it’s like, and the transformer is no longer like the lineage we are on. I think any of those things being very AI-driven would make me update a lot. But seeing more math, like I think I was surprised by the pace of math, but like not astonished.
00:16:49 JS Denain: That’s interesting to me. I definitely agree with this general sense. So like I think METR folks looking at like your nanoGPT results from autoresearch-style things compared to like what the humans were doing, it does seem like there’s this, I think Tom Cunningham calls this like the apple-picking model where AI is like much more efficient at the start, but then doesn’t actually like uncover as many new ideas. And you see this in this kind of optimizer research. Yeah, I mean, one thing I will say on this like verifiability point is like, I think a pretty common trend is like you’ll have some task that’s like not verifiable and you’re like, maybe you struggle to build an environment for it. But actually it’s like it’s a subset of a larger task that is itself like verifiable. It’s just like longer range. An example of this is like there are many like hard to verify tasks out of like companies. But in some sense, like revenue or like other like metrics, like valuations are like pretty legible. So that’s like one thing. I mean, the other thing is like expect things to be pretty jagged. But I think a big question is like, yeah, can you get, for a crazy world, can you get like a large, like self-sustaining industrial kind of explosion?
00:18:05 Nathan Lambert: Yeah. Well, can we talk about robotics and industry? Because I have a background in physical robots and like I think the robotics trends will look much closer to self-driving cars than LLMs. And I think that a lot of the singularity arguments are based on robotics being able to look much closer to LLMs than the self-driving cars roll out. So, why would you disagree? Or, what is the argument that mass industrialization and robotic expansion is doable? Because my prior is so suspicious that I maybe even haven’t given it enough justice, but I’m very suspicious of this being a viability, and mostly in terms of being a relative timeline. I think it could happen over decades, but I don’t think it’s a two to five-year concern.
00:19:01 JS Denain: Two to five years seems rough, to be clear. I think I just don’t know as much about robotics here. I think is your main concern just reliability is really rough to get right in the same way that it was just a long tail of scenarios where things are, or was it more like a real-world thing where there’s much more regulation that comes up?
00:19:21 Nathan Lambert: I think it’s building things is hard. I think that, let’s see. I’ll talk us through some of this. For example, I know places like Amazon, they build new factories to be robotic first, and those are more effective for them. And what this would take then is building a robotics factory. In the case of the US, it’s like you have to build a robotics factory that builds robots very efficiently in the US and then transition that or make a new one that is built by said robots. And I think the re-industrialization of the US is something that I think is like, there’s a lot of reasons why it is not happening. I think potentially in China it is more likely, but I also just haven’t been convinced by AI results on visual and action models that they’re progressing fast enough. I think I’ve had discussions with people in the multimodal field have described the techniques as being much more rudimentary and less developed than the text language models, and in need of much more fundamental innovation, where something like code plus RL is a very natural match that the hill climbing is very predictable. So—
00:20:35 JS Denain: So, it seems like there’s two things. There’s the trends in robot capabilities is not as fast as you would expect for LLMs, and also even if robot capabilities were huge, it takes a while to build factories. I think I’m sort of skeptical of the second one. I’m just like, if robot capabilities are sufficient, the total addressable market for this is massive. And if you look at data centers in the US, there has been extremely fast build-out. If you had robots that were just literally able to substitute for blue-collar human workers, I feel like the financial incentives would be huge. And I think a lot of the reason why in some cases, the US doesn’t have huge build-out is just a demand thing. I think that’s the case for power, for example. So I think in that case, I’m just like, yeah, I feel like we just, what is the Tyler Cowen thing? Don’t underestimate the elasticity of supply is the main thing I would point to. I think on the capabilities front, I’m more uncertain. In particular, I’m sort of still confused and haven’t really looked into the, how much do you get directly actually from LLMs and foundation models for robotic capabilities? In particular, the other uncertainty I have is, it’s not clear to me that extremely fine-grained, extremely dexterous capabilities are the main thing you need for massive industrial explosions. And so, this longer tail of the hardest part of robotics, I’m not sure if that’s the biggest blocker for massive industrial explosion. Overall, robotics is something I have less expertise in. I’m interested in how many of the scenarios for doom ultimately kind of route through hard power acquired through robotics. I think part of my uncertainty also comes from, is it plausible to me that the minimum abilities that you need to acquire a lot of hard power and pose pretty catastrophic possibly extinction risks is more like, have access to nuclear codes or something like that? I don’t feel like I have great thoughts on this. I’m interested in more threat modeling, but I think that’s part of the thing is, what are the capabilities trends is something that people have disagreements about, and so what’s the minimum capability that’s necessary to cause these extinction-level harms, or harms that are sufficiently catastrophic, they just permanently alter the human trajectory?
00:22:46 Nathan Lambert: Yeah. The last point I would make on—
00:22:47 JS Denain: I think that’s the kind of questions I want to see a bit more thinking on, but yeah.
00:22:50 Nathan Lambert: The last point I would make on robotics is that I think the robots will be very good in constrained and repetitive environments, like manufacturing robots, and I think much longer until there are robots walking around the street cohabitating with humans. And if I were to go deep on this, I would want studies and discussions with people that are building multiple different data centers to how much variety is in their job versus how much of it is you take box off of truck and you put box in location. And I don’t have any good signal on how clearly repetitive that is now versus more human dexterous and ingenuity in problem-solving.
00:23:34 JS Denain: Yeah. Although also currently, the environments have been designed around humans who are pretty flexible in those ways and have other constraints. You could imagine designing factories to be robot first is a thing that I think you’re mentioning Amazon, right? Sort of does that more now. And so you could also... Yeah, I think that’s the kind of stuff that jaggedness would get you, which is you might just have massive accelerations, including in the physical world of some industries, even if capabilities for fully being as dexterous or flexible as a human might not be there. Mostly, yeah. I think mostly, yeah, robot capabilities seem very important to track to me. We had some piece about this at Epoch, but we don’t claim robotics expertise.
00:24:21 Nathan Lambert: Yeah. I agree. I think we could shift to another capabilities topic, which is how far behind do you think the top Chinese labs are of OpenAI and Anthropic? And you can define how you want to measure it, whether it’s like public models or like internal models. I think doing both is actually pretty interesting to think about. Like, how would you describe the gap? Like, choose your... This is one of the few things I’ll push you for a number on.
00:24:47 JS Denain: Yeah. I think I would go with like, I don’t know, six to eight months or something, roughly.
00:24:53 Nathan Lambert: From public to public?
00:24:56 JS Denain: Yeah. Like, release date to release dates. I don’t have a great catch numbers on like how long the internal to public deployment. Yeah, I’m sorry if it increases a bit if you’re counting when the model was built, because I would expect the delay to be larger for OpenAI and Anthropic than it is for Chinese labs. But yeah, I don’t know, just looking at ECI and there’s a few different methods you can do there, and there’s a good amount of noise between. Those methods are kind of reasonable, and my sense is they give you something on the order of like six to eight months-ish.
00:25:28 Nathan Lambert: I would say that like Kimi K3 and GLM 5.2 are closer to like two to three or four. So do you think those are anomalies down to measurement or overfitting? So I have this discussion a lot with Florian. And Florian, who helps me with Interconnects, is constantly badgering me down. And I think I intuitively have landed something closer to you. And I think those two models, and in particular Artificial Analysis, were very close. I think Kimi might have been even under two in the Artificial Analysis Index at the time. And I’m just putting this out there, and I go back and forth all the time on it.
00:26:12 JS Denain: So I don’t remember the especially most up-to-date details on the Artificial Analysis Index. My sense is that it might understate the gap for curation reasons or something, but I don’t want to be too confident there because also they update methodology and stuff like that. I think even in ECI, though, there are some cases where it was more like four months. Yeah, I don’t know. Four to eight months seems reasonable to me. I think like two months, I would be like, “Nah, that seems like a bit more overfitting-y.” We did internally look into a bit how much overfitting explains the gap. Like, just doing the kind of like, what’s the ECI lag analysis for, if you just take an ECI based on private benchmarks versus not, or private benchmarks plus results were from models that were from benchmarks that were published after a model was released or something like that, where there’s no risk of contamination. I think overall, that effect was basically not statistically significant, which was interesting. I think if you did rely just results in model cards as opposed to the results being put in ECI, we probably get some contamination or overfitting effects. I still think there’s some amount of, yeah, something like ECI might underestimate the gap just because I do think OpenAI and Anthropic are probably like... There’s probably other capabilities just would appear less in benchmarks and where there’s been more optimization or serving a broader set of users. But yeah, I don’t know. That’s my overall number.
00:27:39 Nathan Lambert: If we were to ban distillation effectively, not even just ban, but if distillation were effectively be stopped, where do you think the number would be in six to 12 months? I think without RSI being super crazy. I think RSI going super crazy, the labs pull ahead by a lot more.
00:28:01 JS Denain: Yeah. On current trends, I think I’ve actually updated towards distillation is actually a really big factor. We can talk about the different factors. And so, okay, so currently if I’m saying six months gap, then in six months, it’s probably not literally 12 months, right? So, my guess is, yeah, you are at like eight or nine, something like that. You go from like six to nine maybe.
00:28:31 Nathan Lambert: Why have you updated on distillation being effective?
00:28:35 JS Denain: Yeah. So I think it’s like, yeah, there’s two reasons. I think, a lame reason is, I don’t know, more people seem to be saying it’s this.
00:28:45 Nathan Lambert: They have been. I’m surprised by it.
00:28:47 JS Denain: Yeah. The Stolen Thoughts paper did suggest it was actually relatively easy to get the stuff. Yeah, random gossip. I think also a big one is, we can talk about this, thinking about the other explanations for the lag and them not seeming as compelling as when I first thought of them. I do think the Claude routers thing also just seems like a pretty big deal. And initially when I thought of distillation, I hadn’t considered that. But I do think Claude routers giving the kind of right prompt distribution for realistic usages, use cases and stuff like that just seems quite useful.
00:29:26 Nathan Lambert: But it’s also like these companies have their own usage at this point. And it’s also like we have the benchmarks. So the benchmark distribution we already have. So I don’t necessarily think distillation is helping with... I think the routers could be very helpful, but I was confused by that prompt distribution thing because once you have the benchmark distribution, you just make similar examples or find similar examples.
00:29:50 JS Denain: Yeah, so I agree. So I think the Claude routers just give you a bunch of useful data to train on, and that will generally improve capabilities. I think that’s one explanation. I agree that I think giving you the right prompt distribution is, I think, going to be useful for real-world capabilities. I agree that purely if I’m discussing this ECI lag or something, then yeah, you can just use the prompt distribution for benchmarks. And so I think it’s not going to be that good of an explanation there. Yeah. I think just... Yeah. Also, so as an example, right, people saying that mid-training is a really important thing or something. I think actually in that conversation, right, Beren was like, “Mid-training is actually getting you 80% of the way there.” And more claims of this form and generally of RL being very useful but not doing that much of the exploration work, I think is also evidence that doing SFT on language trajectories is really useful.
00:30:48 Nathan Lambert: Yeah. I want to go through—
00:30:50 JS Denain: I think he mentioned on a post on this is, it’s a good initialization, but I guess I’m updated that this initialization is really, really important. And yeah.
00:30:59 Nathan Lambert: Yeah, because I’m still on the side that I think doing RL well and doing RL faster so you can go bigger is the way people are getting capabilities. But I have been hearing more about this repeated... The framework would be it’s a very repeated cycle between mid-training SFT and RL, and then your peak RL helps you feed into mid-training and less of a very big RL run, which I never thought... The RL runs are a week probably. I don’t think they’re insane, especially like Zhipu or Kimi. And that’s... So I could see this, and I was like, “I don’t really know.” I generally thought that because the models are getting so useful, I don’t know how... It just seems like the scale of distillation would need to be so big to really do that for mid-training. And the counterargument is Kimi and GLM 5.3 are also strong models, where it’s like, if the gap is that small, it just doesn’t make sense to me that the help would be so big. That’s why I’m still on the not convinced, but looking for more evidence. And then some of the other re... I have this blog post that I wrote in front of me. If you want to talk about distillation more, you can get that comment in before I switch to other topics.
00:32:21 JS Denain: I guess I’m pretty interested in talking about the alternative explanations or something for... I think there’s this general phenomenon of Chinese labs have way less capital and particularly way less compute than US labs. And that delay is huge. And then this, whether it’s four months or eight months, it’s a much shorter delay in capabilities than in capital. And so, this requires some explanation.
00:32:48 Nathan Lambert: Yeah. Do we think the Chinese labs are still releasing meaningfully faster from time that RL is done to public gets the API and evaluation scores? In the past, I thought that the time to release was much faster, and I think the labs are releasing intermediate versions faster now, so I don’t know if that explains as much of it. But I would say a year ago, I would put a lot more to Chinese models get model out within days, American labs could take months, and that would artificially squeeze the gap very substantially.
00:33:26 JS Denain: Why would it squeeze the gap that much?
00:33:28 Nathan Lambert: Because you have the trajectory of capabilities, and higher one is OpenAI and Anthropic, and they stop training here in time, and then the Chinese labs are here. But then it’s just like the model is stagnant before it gets released.
00:33:44 JS Denain: Yeah, I think this depends on the method that... This both depends on the method that you use for computing the lag or something, right? So my sense, I think when I was saying four to eight months, I think a lot of that uncertainty interval also comes from, are you measuring the method by looking forward or backward or something? Or how are you resolving the uncertainty? Are you taking this kind of staircase or this kind of staircase, or interpolating or something like that? So I think that’s kind of still accounted for in the uncertainty. So my guess is that’s not a huge... Or that’s still, even with this, I think there’s still a large effect to explain. And I think, yeah, these are other potential explanations, right? So one is maybe there’s a big delay in compute, but in labor or potentially data, there’s less of a delay, and those are important factors. And so even if you’re two years behind in compute, if you’re six months behind in data and in fact ahead in number of researchers or something, or not that far behind in number of researchers, maybe that helps catch up, and that’s not even a spillover effect, right? That’s just a reason. Yeah, there’s distillation. Other explanations people have are just ideas might leak, and I don’t suspect that’s a huge effect. People can play with a model, and through playing with the model, they kind of infer maybe what it was trained on or what’s useful to push on is another explanation. Those things are not quite like distillation, but using Claude as your reward model or using Claude to clean data or something, maybe have some spillovers.
00:35:23 Nathan Lambert: Let’s see. I’m trying to quantify some. I would say that I think the Chinese labs care about benchmarks a bit more for financial... Some of them are public companies, and showing close benchmarks helps them a lot. So I do think they care about it more. I don’t know if that’s going to give you a month or two. I don’t know what are you going to give back, like a month or two. I think there’s a good chance that the Chinese labs are better at organizing talent on just doing really mundane hill-climbing data work. I don’t think that’s a huge effect, but it’s just like they have a ton of talent, and culturally through the way that... I think that there’s a good chance that it’s just more grind. They might actually grind more effectively than the American labs, which is... I don’t know how. I wouldn’t put a lot to this because I think it’s fairly close, but I think that’s a chance. But that’s a very marginal... That’s not a gigantic lead cause. I would put more of it to compute difference than to distillation. But maybe distillation is very related as a compute difference because it’s a way to turn... It is a definition of way to turn money into very high-quality data, which is something you would otherwise need compute for.
00:36:35 JS Denain: Yeah. Another theory that I’ve vetted about, and I’m not sure how big of a deal it is, is there’s this data market in the US, and maybe one thing that happens is the best RL environments or evals, it kind of takes a lot of compute actually to figure out which ones they are. And so maybe there’s kind of a collaboration between data providers and AI labs to actually test and figure out what RL environment really worked. And then maybe that data provider, once they’ve done this iteration, which implied a lot of R&D compute spending from a frontier lab, then they’ll build a bunch of similar RL environments because they’ve learned the lesson, and then they’ll sell them with some delay, but still sell them to other people, including in China.
00:37:21 Nathan Lambert: I’ve heard this from people.
00:37:22 JS Denain: And so there’s kind of this implicit R&D compute that was spent and that it gets saved. I’m not sure how big of a deal that is.
00:37:29 Nathan Lambert: I’ve heard this multiple times, including from people in China that have... This is like multi-hop type of rumor that I’ve heard a few times is like, we just have to wait a certain amount of time and then we buy the RL environments that Anthropic bought for a 10th of the price. And I think this could contribute a lot to capital efficiency, but also as somebody who buys into human factors being very important in model progress, which is just like competition, I think the analog to the competition that OpenAI and Anthropic feel right now is proof of concept as a way to make something way easier. And I do think the OpenAIs and Anthropics of the world are much more likely to innovate in open-ended things like Navier-Stokes, multi-agent mega-scaling than the Chinese labs. And having talked to many of the labs, I don’t think they would contest this, but they probably won’t put it on the record. But they’re just like, trying to keep up is so much easier.
00:38:27 JS Denain: Yeah. So that’s more like the... Yeah, so the way I describe this was like four-minute mile style effects of like, you show that something is possible at all and then people have the conviction to go down that route, don’t waste a ton of resources exploring a bunch of other things, is the thing you’re pointing at here.
00:38:43 Nathan Lambert: Yeah. And I think this—
00:38:44 JS Denain: I guess I’m looking for examples where you think this... What are examples? Because for example, reasoning models is not this, right? Like, DeepSeek Math or whatever came way before o1. Yeah, I’m kind of curious if... Yeah, I agree this seems kind of plausible, and I’m not sure I can come up with examples of big innovations that a frontier lab has—
00:39:04 Nathan Lambert: I think it’s more in org structure. So it’s like there’s R&D and modeling compute at the labs, which I think kind of get lumped together. And realistically, make-model-better compute of the next generation model versus the two or three generations down the line is very different. And I would think that the Chinese labs are just spending on a shorter time horizon. And then if there’s a major innovation in architecture or something that gets bubbled around the US ecosystem, I would assume the Chinese labs will figure out as well. I don’t think that an architecture innovation is kept very well between Anthropic, OpenAI, Meta, and Google. I think that these are all leaked very quickly through personnel turnover.
00:39:50 JS Denain: I see. So you think that actually has a big effect. So basically, I think it seems like you are just like, you think that with the other explanations, you can just get pretty far, such that you don’t really need distillation to explain a large fraction of the gap.
00:40:03 Nathan Lambert: Yeah. So, I think without distillation, the gap would be fairly similar. It’s like also just building these models, there’s so much low-hanging fruit, and I don’t think distillation plugs that much of the low-hanging fruit that you are doing. It’s very unclear. It’s like once you establish your reasoning trace as being... This isn’t proven, but I think in mid-training and SFT, you need to have a pretty distinctive style in your reasoning trace to kind of stack it and have a defined reasoning strategy for the model. And Kimi and GLM have this, and I think they could modify it from where they are. And it’s just like, how does that compound over time if Anthropic and OpenAI change way faster, maybe it matters more. I don’t know. Doesn’t sound like we’re getting somewhere. I did like the post you had on understanding Chinese job postings, which I thought was a very clever way to try to peek behind the curtain of the Chinese AI industry. So I don’t know if you have anything you thought was particularly fun about this. It’s like, it was so hard to understand the industry that is going on there from outside.
00:41:22 JS Denain: Yeah. So this is work by Cheryl Wu and I. I think it’s, as a source of evidence, it’s more like you can get interesting anecdotes or interesting facts. I think it’s not so much the kind of Epoch-y thing where you can get a trend line that you really follow really well. I think some job posting trends are good for this, but I think especially for China, it is a bit more like, “Oh, well, you can see they’re hiring in this region for data center roles, and it seems like that suggests that at least some of these labs are basically constructing their own data centers as opposed to renting compute from Alibaba or whatever.” So you get a few kind of nice facts like this. I think you also do understand a bit more what kinds of product strategies the different companies have and what they’re aiming for. So I think it’s pretty interesting for that. This post was now published in June, so things move fast and things may have changed. But yeah, I think there are interesting takeaways on company strategies, on how they—
00:42:31 Nathan Lambert: Can you say more about company strategies? Because I think that a lot has been cast on, say, like... The different company strategies in China. It’s like, what is DeepSeek’s strategy? I think MiniMax is one of the clear ones, and then Zhipu. And Zhipu has some on-premises deployment stuff, but at the high level, I think Zhipu can be much closer to OpenAI and Anthropic. But do you agree with this, or do you think any of them are more defined or have more interesting different things?
00:43:03 JS Denain: Yeah. I mean, one thing is just, I think we have this plot of how many job postings there are for B2B sales. As an example, it really does seem like Z.ai, like Zhipu, has a lot more of them than MiniMax or Moonshot. I think also, they have different strategies in terms of how much they want to expand internationally. I think MiniMax is a decent amount of that, more so than Z.ai, I think was the thing we found. I think there’s a few things like this. I think for DeepSeek, we didn’t see as much of that, for example.
00:43:40 Nathan Lambert: Having done this analysis, do you think you could hand an AI model your taxonomy and have it redo it every three months? Do you think this is now a thing that Epoch could almost automate?
00:43:53 JS Denain: So I think we do have some upcoming project on job postings. I think specifically for this kind of thing, I feel like the takeaways are kind of ad hoc or something. And so I think it relies a bit more on you have a bunch of this data, what actually seems interesting? And I feel pretty often that LLM’s assessment of what are the top five most interesting things is not that great. But one example of a thing where I think... So I actually have this really crappy vibe-coded job listings tracker, which looks a bit more like the quality’s better for an OpenAI Anthropic. And I think Epoch might have something on this in a bit. But one example where AI is pretty useful is just, I don’t know, you just want to reasonably classify jobs as are they basically research jobs or are they basically engineering jobs? Or maybe research engineering is actually hard, but are they research jobs or are they GTM jobs? And the structure of the teams are going to be not coherent across different companies, but you can use AI to automatically track this over time and have some coherent categories. So then you do get trends like, “Oh, it seems like there’s been a surge in hiring from this frontier AI lab, and it’s basically mostly been sales, or it’s been mostly hardware infrastructure.” I think that’s pretty cool, interesting, and automatable.
00:45:14 Nathan Lambert: Yeah. I think if you do this again, I’m interested in if you find things on data market expansion, which would be like your job is buy data. Because I’ve heard from many people that the Chinese data market is exploding, and I have poked various, whether it’s people I know at labs or people who I know follow Chinese industry closely. And it’s like, they’re like, “Yeah, but I can’t get anyone on the phone to know what this manifests as and what their budgets might be,” because Anthropic famously has billion-dollar budgets. And it’s just like I think it very quickly can become very important, like compute export control discussions stuff. By the end of the year, I suspect there will be some article talking about this and going deep reporting it. But from my perspective, getting that reporting information is near impossible.
00:46:05 JS Denain: Yeah. I think data is just really rough. I think for Epoch, data has been a tricky thing. It’s obviously an enormously important factor, but it’s harder to quantify or commoditize, and it’s also quite secretive. And so it’s like, I think a lot of the information is gossip. It’s hard to know how much to trust. My sense on Chinese, I think that article actually has something on this. Just it does seem like there’s a lot more in-housing in general by Chinese companies, and I think this also applies to data. And kind of relatedly, actually, there’s a lot of hiring for intern roles. I think if you go to the jobs listings pages for a lot of these companies, there’s specifically two tabs. They have summer internship tab or the internship tab and the normal hiring tab. And there’s a lot of, I think, hiring fairly junior people, and some of that might be for data purposes. But you did also see a surge in in-housing for data in the frontier AI companies. I’m not sure if this is still the case, but xAI was the only company that had this, but it was kind of nice because if you went to the careers page, there was actually the human data category, and you could just see which data roles they were hiring for. So there’s stuff to do on job postings on the data front as well, both xAI. I did this with Mercor and a few others previously. But I think it’s similarly, I think the most interesting things have been like, “Oh, well, that’s a crazy-ass role.” xAI is hiring for meme specialists or whatever, and it’s more ad hoc fun things than great trends, at least from what I’ve seen.
00:47:35 Nathan Lambert: Yeah. I think that this is not going to change, unfortunately. It seems very much like individual teams at labs have contacts at data companies, and all of this is done privately, and it’s not... I guess even tracking compute is fairly hard, but now data centers are a manifestation that is somewhat trackable. And NVIDIA is a public company, and that is what is done. If the data companies were public, it would probably be easier to tease some things apart.
00:48:13 JS Denain: Agree.
00:48:13 Nathan Lambert: Okay, we have another timely topic. How do you feel about frontier model safeguards? Given that in the last 24 hours, we learned that three people used public Claude models to hack OpenAI and get internal access, which is just crazy to me. Did you see this? It’s genuinely surprising to me that this is the leaky world we are in. At the same time, we are discussing RSI things much more seriously.
00:48:46 JS Denain: Yeah. I briefly saw this incident. I guess I’m not sure how much of an update it should be specifically on like model safeguards versus like you know security quality of the companies, or probably both. I think it’s a big question, right? Like, how good are the model safeguards? Seems like quite important, especially when people are talking about open models where it’s clearly open model safeguards are worse because there’s way more affordances that someone can have, but also open models are less powerful. And so just in terms of the you know misuse implications, it seems there’s just like slightly safeguarded, very powerful model versus like not at all safeguarded you know less powerful model. And I feel it was a priori kind of unclear to me, like which one is going to have the largest effect. Yeah, I don’t know. I think this is an interesting question. I think I feel also like people feel differently about this or something. It seems like some people feel pretty strongly that it’s hard to at least persistently do really bad things using the frontier models. But I’m like, yeah, there’s definitely clearly examples like this one, or really the Stolen Thoughts paper or something. It just seems like the safeguards weren’t good enough to prevent that from happening, even though there’s strong incentives for the companies to prevent it.
00:50:06 Nathan Lambert: Yeah, the Stolen Thoughts—
00:50:07 JS Denain: I’m kind of confused about this.
00:50:09 Nathan Lambert: Stolen Thoughts I put into like a serving issue, which is just like you were able to kind of jailbreak the API functionality to get the reasoning traces out in a reliable manner. Which I think the Chinese labs almost certainly had been doing for some time. And that potentially has a way to show more about what the effect of distillation was. I would guess that there are other ways that they get those tokens out, unfortunately, if there was one such prominent one. And then there’s the models themselves, which is the safeguards they have on the models, which is like the classifiers and prompt distributions and things. And I would expect the models to be somewhat permanently leaky. I just think that it’s hard to imagine. If you have a less aligned model and the task is, “Help me figure out how to get around said aligned model’s safeguards,” I would guess that that’s in scope of a strong cyber model’s abilities to figure out some sort of jailbreak for another model. It seems hard to permanently squash these things with how leaky they are now, despite the supposed launch, like higher false positive rate and being so general, and people get upset when they launch and they’re getting downgraded all the time, and people’s accounts get banned for doing bio research and things. So it’s just like the whole thing seems like a very big mess. I would be on the opinion of like, I guess maybe this is somewhat why the whole Fable freakout went down, which is like Anthropic was like, “Mythos is a powerful weapon.” And then somebody told the government and was like, “It’s actually easy to get around it,” and then they freaked out. But I think of it as like some of the distillation story has been around distillation lets them scale, take dangerous capabilities. But I think if you’re hosting an API at all, you have to assume in the current and near future that you can extract the capabilities out of that model, even if it’s slightly safeguarded.
00:52:14 JS Denain: Yeah. I guess one thing is I’m not sure how they prioritize between different threat models, but they sure seem to have some incentive to prevent distillation. It’s also hard to know what the counterfactual is, right? We have these examples of people doing those bad things. There’s a lot of usage of those models. It’s also compatible with, in most cases, it is actually really hard, and those people figured it out, or there was some random blip that allowed them to do it. So I feel I don’t really know how to update on that evidence. But yeah, at least in some cases, people just do manage to get around it. Maybe one distinction is also like, how long do you have to do this for? So there’s the model’s alignment that’s in the weights of the model, and then there’s your synchronous monitoring that you can do in the moment. And surely that’s going to be way less effective than how good you’re going to be at detecting things where asynchronously able to spend a ton of compute to parse through trajectories and analyze group trajectories together and stuff like that. But that takes a longer time, and so if someone needs to do an attack very quickly, then it’s just too hard for someone to detect it in a moment. But then if someone wants to pursue a very long campaign, maybe it’s harder. I’m not sure how long those campaigns take or how effective those KYC things are. But that also seems relevant for other cases that are not about leaking capabilities, but about other cases of misuse.
00:53:38 Nathan Lambert: I guess to make it more specific, how much of, say, risk proliferating from open models do you think is down to the fact that people will serve the open models or can serve themselves without the additional oversight and classifiers, versus what I think was talked about more, which is just like fine-tune the safety away, which I think very few people are going to fine-tune Kimi K3. But if these classifiers that squash 99% of people at inference time just kind of blanket a lot of the risk, I think that’s almost more plausible. It’s a bit of a sequence to get there. You have to assume people could get around model safeguards, assume a lot of usage.
00:54:20 JS Denain: Yeah. I guess the fine-tuning thing right now is not a big deal because not that many people have the combination of motivation to do it and competence and willingness to do it. The bar for that also falls as AI research becomes more accessible. But yeah, I think right now, the fact that there’s basically no safeguards, or my sense is the inference providers aren’t trying as hard as Anthropic or something at this. I could be wrong about that, but that’s my impression. And also you can pick whichever one is least secured. So I think that’s a bigger deal, at least currently, yeah, I would expect. But yeah, I think it also depends. I think the main thing that I care about for this is actual threat modeling. I just feel kind of confused about which kind of group I should have in mind as the most likely person to do a bad thing with AI. And then I feel like if someone gives you an actual scenario, then I feel like you and I will have decent takes about will it be easier for them to use an open model or to try to jailbreak a closed model? And I just don’t currently feel like I have a good sense of this. But I think—
00:55:24 Nathan Lambert: When this is so ambiguous, I think it makes me feel a bit better about the near term, which is just like these safety tools are, it’s to some capacity inadequate now, and there are strong open models, and it’s not like all hell is breaking loose. And you have to keep reevaluating this as things change. But I think it doesn’t seem like things are going to change. I don’t think there’s that much of step function changes. I personally need to learn more about biorisk that people very often talk about, and I think need to... My question is, what is the actual manufacturing pathway and where is the model intelligence at now? Where I think cybersecurity, we’re in it. We’re in what it’s going to be like for a bit, and it’s not good, but it’s not like nothing horrible has happened yet, and it’s okay. I don’t think there’s going to be more.
00:56:25 JS Denain: I don’t know what the sense with the evidence is, right? Companies don’t love to admit. I think it’s famously kind of hard to estimate cybersecurity costs and stuff. I know we want to look at insurance costs. I don’t think they’ve changed that much, but it would be interesting to look at right now. Like cybersecurity inference prices or something is like what percentages you can look at, but companies don’t love to disclose they’ve been hacked, and so they sometimes just pay. But yeah, definitely, yeah.
00:56:50 Nathan Lambert: That’s a good point as well.
00:56:51 JS Denain: I don’t feel like I have a ton of expertise there, but I’m pretty interested in those questions.
00:56:57 Nathan Lambert: I’m glad that this was a kind of like, “We don’t know, it could be worse,” and not a, “This is the list of very bad things.” Which is, it’s hard to motivate people when it’s like, “I think we need to keep preparing, and you need to assume that the Hugging Face thing might happen to you.” But it is not a super, it’s not an avalanche of compounding problems right now.
00:57:20 JS Denain: Yeah. I do expect Hugging Face-style things to happen kind of in the wild regularly and outside of OpenAI. And if it’s just that, then that’s not massively bad. I mean, it’s still a kind of screwed-up world. But I do expect more of this just happening with a variety of models and neoclouds and stuff over the next year. Just a bunch of open source agents that people can’t control.
00:57:44 Nathan Lambert: Yeah. I do think your threat model thing is a good understanding, and it’s like, who are the people that would try to take down an energy grid in a domestic area? And I think the list is pretty small. And it’s like, what are they doing? And I’m not an expert in this at all.
00:58:02 JS Denain: Yeah. I feel like the biggest safeguard is just there’s not that many people who want to do horrible things. Fingers crossed.
00:58:10 Nathan Lambert: Yeah. Okay, I have a fun question. I think it’s like Epoch AI creates so much of this kind of area setting information for different parts of the AI discourse. One thing is, is there a fun process on how you choose problems that you think are important? Are you guys constantly underwater and wanting to do different things, or is sometimes it’s like, “Now’s our moment?” Just behind the curtain of Epoch and what you guys do culturally is fun to me.
00:58:41 JS Denain: I think what determines project selection, I do think it is very what seem like the important questions to us are quite curiosity-driven overall. I think that’s probably the main thing is just what seems like a huge deal, I think has been the main determinant of where efforts go. More specifically, yeah, another framing is the Epoch was created as this thing where it seemed like neither the industry nor academia would be able to provide a good public resource is also well-maintained on the most important trends in AI. So it seemed like there was something missing. Another framing is, I know, what do people have misconceptions about, is sometimes, “Oh, people are wrong on the internet,” kind of things. Some articles can be explained by this. But yeah, I think it’s very curiosity-driven. And also it’s just like, oh, what’s evidence that’s leaking information that’s laying out there that we can use? Like, the job postings is this. Data center tracking is very much this as well. It’s like, man, these are big objects. Surely you can see them and track them.
00:59:53 Nathan Lambert: Do you think the information needed to understand the AI industry is stable in your access, like getting less accessible, getting more accessible over time?
01:00:03 JS Denain: I mean, it really depends, right? I think data center is, so far at least, it’s pretty good, and they’re getting bigger, which makes it easier. I could imagine at some point it becomes a really strategic asset, and there’s way more adversarial pressure to hide them or at least hide key information. For now, it’s pretty good, right? In contrast, Epoch is an effort to track model training compute, and at the time, you could just look at the paper. It’s much harder to do this with Astra. So yeah, I think it depends on the verticals. One thing that I mentioned before is, yeah, data is pretty rough. We’ve had some stuff on this, but it’s much tougher than compute, and it’s pretty secretive. So yeah, I think it depends on the vertical. Oh, yeah. One thing is, as the economy grows and companies go public, that’s better information, right? So that’s true for compute companies. As IPOs happen, that also helps get more data.
01:00:55 Nathan Lambert: Yeah. I find it very interesting. It’s like I’ve long wanted to know what a post-training... Like, what is a flowchart of a post-training recipe at a frontier lab? And I think it would be wild, and I think those flowcharts exist. And even some old ones, I think, would give people a lot of thought, because I do not think it looks remotely like the OLMo three-stage approximation type thing. And we have some from like Nemotron and some of the Chinese labs release stuff on it. But when you hear John Schulman talk about post-training and other people, it seems like the total Wild West and just stacking everything together. That framing for like that framing for what is the complexity of putting a model together, I think would be very useful for people as a mental model for what is being scaled and what are you scaling through. Like, yes, you are scaling, but if you’re scaling on top of this kind of house of cards thing, it’s not perfect.
01:01:55 JS Denain: Yeah. I’m actually curious. Yeah. I have this like... Okay, can I test my current mental model of this, which is pretty uncertain? My sense is this, okay, there’s a big post-training team, and it just has sub-teams that are kind of specialized. Like, some people are going to be doing math, and some people are going to be doing biology or finance or something. And the job of those teams is to ask for data and curate that data and make the RL environments good, and also figure out what kinds of hyperparameters work well for their own domain. And they do this by buying very high-quality evals that they use as a north star to iterate on and buying those RL environments. And their main output is basically like, “Here’s a curated set of environments, and here’s the hyperparameters that we want.” And then my sense is indeed now things are basically all those teams bid to put some fraction of their data into a final big RL training run. And they also bid to maybe for the hyperparameters of that final RL run to be close to what they want, question mark. That’s at least for the RL process. And then there’s in fact a big RL run. Another alternative would be, of course, they all do their experts, they distill them back into... Does that roughly what things are like? This looks—
01:03:16 Nathan Lambert: I think the org side, I agree a lot. I think the org side, there’s definitely per domain expertise, and it’s a lot of why I didn’t want to go get a frontier lab job, because most likely I would just be given, like, “You’re going to work on this data and you turn the crank.” I don’t know if the complexity is a lot higher on how you put it together. I think it’s interesting that the open weight models are split between multi-teacher on-policy distillation, and some are just like do sequential RL. I think the diversity there would be very, very interesting to the industry of whether you have multiple RL runs, how do you decide which environment to do and which of the stages? How do you manage mid-training and SFT? I think mid-training and SFT, I think, are one stage in the frontier lab, and the base model is just before mid-training, and then you do a whole bunch of stuff at the end there. And I just think there’s a lot of things you can do, and I feel like if OpenAI and Anthropic’s recipes were closer to mid-training and then big RL run, if that’s actually getting them to where they are, I think I would expect progress to go faster than if it’s mid-training and fork and model merge and bunch of experts and MOPD and then more RL. I think that just is a bit of a bottleneck on progress, and it would be very useful to have more information on what does model training look like.
01:04:44 JS Denain: My vague sense, but I don’t have super hard evidence on this, is that it has gone more in the direction of, yeah, away from MOPD with multiple experts and a ton of checkpoints, and a bit more in the direction of like, okay, there’s in fact one big run towards the end, but of course, much federation before, but I don’t know that.
01:05:01 Nathan Lambert: Yeah, maybe the rumors we’re hearing about mid-training being important and distillation is from that style of loop rather than the expert style loop. I don’t think we’ll know. I think we could continue that another time. This was fun. I think you should keep giving your incisive responses when I write something. I think this will be very useful to people and good to finally chat a bit longer.
01:05:24 JS Denain: Likewise, yeah. Thanks a lot.
01:05:26 Nathan Lambert: See you soon. Bye.


This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.interconnects.ai/subscribe

I was recently invited to brief a group of Congressional members and staff on the state of open-weight models in the lens of U.S.-China competition. I’m sharing my prepared remarks as a state of the union on open models that is accessible to a broader audience.
Interconnects AI is a reader-supported publication. Consider becoming a subscriber.

Recap: What is an open source v. open-weight vs. closed model?
Open language models are AI models where their weights are publicly available for inspection or downstream use. These are most often contrasted to so-called “closed” AI models. Closed models offer access only through Application Programming Interfaces (APIs) that developers can use to directly query a model, like GPT-4 or Claude Opus 4.5, or through products, like ChatGPT and Claude Code.
Open language models primarily are bucketed into two categories, open-weight and open-source models. Open-weight models are the most common form, such as popular models like Meta’s Llama, Alibaba’s Qwen, Google’s Gemma, or DeepSeek’s models. These models are governed by licenses, governing documents dictating what is allowed with downstream use, and are often accompanied by inference code in libraries such as Transformers, VLLM, SGLANG, etc. Since about April 2025, Chinese AI companies have been the clear leader in open-weight models.
True “open-source” models are similar to these, as they include the weights, licenses, and inference code, but they also include the complete information needed to reproduce the model – the training code and training data. The most prominent open-source models have been built in the United States, led recently by the Allen Institute for AI’s Olmo models that I helped build in my recent 2.5 years there. The other prominent open-source models are also built by American non-profit organizations, including OpenAthena’s Marin models and EleutherAI’s Pythia models.
Open-weight, open-source, and every other label for a model – including closed models primarily offered via an API – exist on a spectrum. For example, Nvidia’s Nemotron models are far more open than most open-weight models, releasing large quantities of their training data under permissive licenses, but they’re not fully open-source because they do not release all of the data. Closed models also exist on a spectrum based on what information the API reveals and the terms of use.
The state of competition between American and Chinese open-weight models (unit economics, technical capabilities, etc.)
We are living in a world where GLM-5.2 and Kimi K3, some of the latest, leading Chinese models, have enacted a step change in the commercial viability of open models — crossing a similar threshold in agentic capabilities that Anthropic’s Claude Code crossed in December of 2025.
America was the early leader in open language models, primarily through Meta’s Llama models, which were used extensively across research and commercial tasks. Chinese open-weight models surpassed American open-weight models in these two key areas about 18 months ago. The simple metric showing this is Hugging Face Downloads, where China took the lead in July of 2025 primarily through the success of Alibaba’s Qwen models. I personally maintain tools to track this data, and since I first published the American Truly Open Models (ATOM) Project in August of 2025, China’s download lead has grown to about 1.6B – with a total of 3.2B downloads, twice that of America’s total.
On popular capabilities benchmarks, such as the Artificial Analysis Intelligence Index (AAII), the Chinese open-weight models have a clear lead over American counterparts. The top three Chinese models as of writing this on September 14, 2026 are Z.ai’s GLM-5.3 and GLM-5.3-Flash and Moonshot AI’s Kimi K3 with scores of 45, 42, and 44 respectively. By comparison, the leading American models are Thinking Machines’ Inkling and Inkling Small, both with a score of 26, and Nvidia’s Nemotron 3 Ultra, with a score of 23. The top American models were released in June and July of 2026, and are updated less frequently than their Chinese counterparts. For example, Chinese labs released models with scores above these American models 2-6 months before the American companies got there (e.g. GLM-5 or DeepSeek V4 Pro). There is a trend of more American companies releasing models, including names like Arcee AI, Poolside and IBM, but they are not rapidly closing this performance gap. Other benchmarks tell a similar story.
Together, Chinese open-weight models are approximately 2-5 months behind the closed American frontier, with the open-weight American models being approximately 6-9 months behind the likes of OpenAI and Anthropic. The Chinese labs are closest in tasks with clear user demand, such as agentic coding, and further behind on more open-ended scientific tasks, such as physics or biology.
The reasons why Chinese labs can produce these strong models, despite having fewer resources than American counterparts, is still an open debate and heavily influenced by different work cultures, but is also influenced by a few key technical factors. The Chinese labs release their models faster and focus on a slightly narrower distribution of tasks, flattering them slightly on public benchmarks. Releasing faster helps them score higher because all the labs are making consistent progress, so once you “finish” a model to be released, it is a snapshot of performance at that given time — labs where that time is later tend to score higher. Still, the models built by the Chinese labs are genuinely strong and represent real competition to the American industry. This competition will not decrease meaningfully as the closed labs patch vulnerabilities in their API offerings which enable distillation.
Distillation is most impactful in new domains and does not make it trivial to create a universally strong final model. I estimate that if distillation was fully prevented, e.g. with know-your-customer (KYC) tools at Anthropic and OpenAI, the gap from the strongest American models to Chinese open-weight models would only increase by 1-2 months.
For example, the Chinese labs are rapidly changing their posture towards paying for training data in 2026. Earlier in the year, the top Chinese labs including Moonshot AI and Z.ai had a strong preference towards building data workflows in-house, but by the summer they had begun to buy the cutting edge data – challenging RL environments for agentic tasks – from both established American companies and new Chinese startups.
With the advance of open weight models in China towards the frontier of capabilities, and the recent documentation of growing risks around frontier models in areas such as cybersecurity (e.g. the OpenAI-HuggingFace incident), there’s growing regulatory uncertainty on how continued releases can enable a safer ecosystem?
A structural challenge in open-weight models is that there are few effective methods for stopping pieces of open software from reaching bad actors. If an attempt was made to restrict access to the strongest open-weight models from China because they amplify risks, the parties who would be set back are American businesses. We have an example of this – HuggingFace used a Chinese open-weight model to understand the cyberattack because closed models would not answer their requests. Thus, managing the risks of open-weight models often comes down to ecosystem preparation.
Open-weight models are becoming an essential tool for AI diffusion, and the best path to get ahead of these risks and unbalanced relationships where American companies rely on models built in China is to continue to enable investment in open models in the US. Ownership of open models allows better coordination and preparation of risks that are global in their nature while accelerating diffusion of AI services throughout the domestic economy.
The state of open model adoption: How is open-source being used by academia, businesses, and other countries?
Open-weight language models have grown substantially in general interest and economic viability in 2026, allowing early glimpses of more direct ways to compare adoption of models from the US, China, or elsewhere on top of Hugging Face metrics. One example is OpenRouter usage. OpenRouter is a popular LLM inference platform that supplies a single interface to switch between models, open and closed, from the US and China. This platform is primarily known for trying different open-weight models. The platform has shared usage data for the top models since Jan. 1, 2025, and shown growth in usage from ~1T tokens processed from open models in a week of September 2025 to ~80T tokens per week today. In that time, Chinese models have grown from ~70% market share to over 80% of usage. Other platforms that are designed to commercialize open models show similar data, such as the open-source coding agent OpenCode, which shows an inference volume of ~95% or higher with Chinese models.
These open platforms are the best approximation of open model usage we have – a large proportion of open model usage is on platforms that do not disclose per-model breakdowns, such as Together AI or Fireworks AI, and in private deployments for enterprise applications.
Many prominent technology companies and startups have been building on Chinese open-weight models for their AI features, such as Harvey, the legal agent, Cursor, the coding agent, and DoorDash’s use of Kimi models, Airbnb’s use of Qwen, or Perplexity’s use of DeepSeek. These prominent companies are the tip of the iceberg, where a large swath of younger Silicon Valley startups are building on Chinese models in order to have low-cost, flexible options. There is a growing trend of American startups and companies entering enterprise agreements with Chinese model labs in order to get permission to use their models in their products – a new form of cross-border technology collaboration I have not witnessed in my career.
The foundation of innovation on Chinese models extends further into the AI ecosystem. To a first order approximation, most of academic research is conducted on Alibaba’s Qwen family of models. Having met multiple members of the Qwen leadership team during my trip to China, they are very invested in and intentional about this type of adoption, which will not be easy to claw back to American models.
To quantify the adoption of open models across academia, I scanned every paper in the 5 most popular ML categories of arXiv (cs.AI, cs.CL, cs.CV, cs.LG, stat.ML), the preprint platform popular in AI research. The results clearly track my understanding of the evolving leadership in AI research, showing LLMs becoming a foundational layer of ML research – mentions of any open model were 2% in January of 2023 and 50% in September of 2026 – and the leading role shift from the U.S. to China in the same time period.
For example, in April to May of 2023, a few months after Meta’s original Llama (a backronym, Large Language Model Meta AI, first released in Feb. of 2023), about 2,600 of 12,000 new AI/ML papers on arXiv mentioned at least one prominent open model family. Of all those scanned papers, ~5.5% mentioned Llama and ~1% mentioned a Chinese model. In the fall of 2024, during Llama’s peak, about 23% of papers mentioned Llama with about 7.5% mentioning Qwen, the most direct Chinese competition. Today, Llama has lost its lead in academia, being mentioned in about 21% of papers still, which is remarkable longevity, but Qwen’s share has risen to 30% of papers. Overall, any Chinese open weight model is mentioned in over 40% of papers, over the U.S.’s 30%, with China’s share continuing to grow.
This shows that we clearly have a lot of work to do in order to re-establish the U.S. as the home of AI research in the era of open-weight language models. There are signs of hope.
In our research, we find that American models of comparable capabilities-to-size regions to their Chinese counterparts get adopted at disproportionate rates. In the last year we’ve seen OpenAI’s first open-weight models since ChatGPT, gpt-oss, become one of the most adopted open-weight models of all time. Since then, Google’s Gemma 4 models have been some of the only ones ever to show similar adoption numbers to Qwen’s most popular small models, and Nvidia’s Nemotron models have modest adoption despite numerous more capable models at the same size point.
Summary
The story of open models in 2026 is one of establishing economic relevance. This is the convergence of many stories across the AI ecosystem, summarized as:
* The capabilities gap from open to closed models available to users has been decreasing over the last 3 years. This varies by task, but can be estimated as a 2-5 month gap in capabilities. With capabilities overall progressing so fast, this has seen open-weight AI models unlock substantial markets in 2026 and points to more inflection points in the near future.
* Open model usage is exploding in high-value industries (e.g. software engineering, legal services, financial services), indicating an emergence of an alternative ecosystem to the best closed models. Platforms offering inference primarily on open models, from Together, OpenRouter, Fireworks, Baseten, etc., are seeing incredible growth as the first winners of an open model post-training economy (other layers include finetuning APIs such as Thinking Machines’ Tinker). This is combined with numerous anecdotes from technical staff in the AI industry that uses open-weight models such as GLM-5.3 as an alternative to Claude or GPT due to a combination of speed, lower prices, customizable offerings, and privacy.
* Chinese AI companies are the clear leaders in open weight models. Relative to 2025, where Chinese models like DeepSeek R1 shook the AI world with surprise, the American AI labs have been recovering in their positions with open-weight models, but despite more substantial investment in the US, the Chinese labs regularly are producing notably stronger models adored by many types of users.
* Distillation of American AI models by Chinese labs does not explain the entire story of their success. Distillation is an industry standard technique of training another AI model on the outputs from a usually stronger model. The technique is most prevalent in the Chinese AI industry, which has used basic exploits to extract reasoning traces and additional data from American companies’ products that are not fully secured. The best estimates are that distillation helps reduce the performance gap of Chinese companies relative to the American frontier by 1-2 months.
* Chinese models, particularly Alibaba’s Qwen family, are established as a foundational layer of research and development across academia and local model users. In recent months, Chinese open weight models were mentioned in 38% of AI papers, above the U.S.’s 28% – and the Chinese share is growing much faster than its American counterparts. This, along with other political factors and the closed nature of leading American AI companies, is contributing to an accelerated decline in America’s lead as the preeminent AI research hub in the world.
* Open weight models are entering the capability levels where new risks, e.g. cybersecurity, can be enabled by numerous open-weight models being available, necessitating an ecosystem level response in preparation. This new era of risks is also enabling a period of political uncertainty, where there is regulatory attention on the strongest AI models, but massive uncertainty on how policy would be legally enacted. At the same time, many researchers and engineers rely on open models due to more permissive safeguards, where the closed models such as Claude and GPT often refuse critical cybersecurity defensive work or biology research.
For more data, view the Interconnects Dashboard.
Conclusions
In 2026 the Chinese labs are clearly maintaining their status as the leaders of the open-weight AI ecosystem. This comes as open-weight models have passed an inflection point in economic viability and in the face of increased activity from American labs as model competition. The leading Chinese labs do not appear to be meaningfully challenged, as they expand their enterprise and research adoption globally.
This landscape of open models comes at a crucial time in the broader AI ecosystem. We’re seeing OpenAI and Anthropic take massive steps forward with their latest public models, and at the same time call for coordinated care on how we manage the next stage of AI progress. What is happening in the confines of a few AI labs today, especially with extreme talent and compute density, is a precursor to what will soon emerge in the open model ecosystem. Open models are going to be the substrate for everyone else in the world outside of the few true frontier AI labs, to harness an acceleration in software engineering and other computational practices. This represents a substantial source of soft power, influence, and potential for the organizations that enable this broad access to transformative intelligence.
With this future coming soon, we need to collectively stay humble about the exact path open models will take. There are a lot of unknowns with open models – e.g. we don’t have good data on how they’re used in countries other than the U.S. and China. With the distribution of ML training expertise being broad, i.e. tens of organizations and thousands of people that are within a year of the frontier of capabilities, it is a matter of when, not if, open models cross the performance thresholds that enable new workflows. The collective approach should be to understand how to use this broadly accessible, open intelligence for good while proactively mitigating the potential harms.
Thank you to Florian Brand and Kevin Xu for feedback and/or suggestions for this work. For more research informing this post, see the open-source AI reading list.


This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.interconnects.ai/subscribe

We’re in an era where a few organizations are using thousands of concurrent agents to improve their processes and output. These organizations happen to be just the frontier AI labs, in particular OpenAI and Anthropic. In the last few weeks, I’ve been pondering what it means for so many employees across these organizations to rapidly update their expectations for the pace of AI progress and associated risks.
A core perspective I have is that the frontier labs and broader frenetic, competitive culture in the San Francisco AI scene set up an environment that amplifies any AI concern. This has some benefits in causing more general audience awareness of AI, as fear sells, but exaggerating risk timelines or severity will have negative second-order effects. I remember many loud AI safety debates, and their associated clouds over the viability of open-source AI, in 2023 and 2024 — the primary risks then did not arrive in the forecasted timelines.
The general populace of these two key labs was very anxious about AI risks and the rate of progress even a year ago, and especially as agents got stronger product-market fit at the start of 2026. This cultural precondition, when exposed to the reality that thousands of agents will constantly be working fairly productively in your business, will only increase this anxiety. The step from this anxiety, and incidents like OpenAI-HuggingFace, to extinction risks feels very religious.
Richard Ngo had an apt summary of the situation:
Now a large proportion of the AI safety community is implicitly or explicitly orienting to futures where an intelligence explosion occurs within a few years. My default expectation (absent an extensive pause) is that a similar thing will happen: they’ll turn out to be directionally correct (relative to the expectations of almost anyone not linked to the community) but factually wrong. Specifically, we won’t have superintelligence within the next 8 years, but things will still be moving so fast that it’ll *feel* like the people who argued for short timelines were right.
… I wanted to say something now because it feels like the level of bandwagoning towards “singularity soon” is getting pretty wild.
Personally, I think this view aligns closely to what I outlined in my alternate scenario to true recursive self-improvement (RSI), which I called lossy self-improvement. A summary of this view is that:
* Automatable research is too narrow to achieve a massive net acceleration in progress, in the face of scaling laws’ exponential costs,
* Diminishing returns of more AI agents in parallel are real, &
* Resource bottlenecks and politics are a major factor in building strong LLMs (and AI can do much less to accelerate this).
So, I’m left balancing the above, latent increase in the cultural temperature with the potential that the labs have seen genuinely scary, specific breakthroughs that are not public yet. My expectation is that more of the current AI safety concern is on the former – scaled agents working – but I hold high levels of uncertainty here. Foundational, imagination-based AI breakthroughs are the sort of thing that would make me update my RSI timelines from closer to a tool to sustain progress in the face of exponential costs (scaling laws), to something more unpredictable and/or unstable.
Interconnects AI is a reader-supported publication. Consider becoming a subscriber.

Some of the best recent resources on RSI have been Dwarkesh’s podcasts with Noam Brown and the trio of John Schulman, Beren Millidge and Charlie O’Neill. I have a few important reflections from both of them.
First, the podcast with Noam Brown made me internalize how big of a short-term acceleration mass inference capacity is. These labs will throw thousands of agents at important, measurable problems. At the same time, compute capacity available to them is going to continue to scale. I have my doubts that the labs can afford to spend a constant portion of this compute on internal R&D as the total volume goes up, especially with plans to IPO, as they face increased scrutiny on basic economics. It is important to not confuse massive steps in inference-time scaling, a dynamic which should be fairly predictable, with being the outputs of RSI, which is highly uncertain.
Second, the trio podcast debating the state of the art in technical capacities induced more of a surprising reaction that I haven’t fully settled. Through the first hour or so of this podcast, where they debate the role of RL, distillation, scaling, inference-time compute, etc., I found myself strongly agreeing with the distribution of claims. A TLDR would be that our current techniques work and let us solve problems we know how to state, but they don’t result in a magical level of generalization to unknown, harder problems in most partially verifiable domains (i.e. progress in math is an exception, rather than a rule).
The surprise of this podcast was the end, where they were predicting timelines for various thresholds of AI. I had GPT-6-Astra summarize the answers provided to three questions from Dwarkesh, of the form “when will AI reach X ability”:
All timelines are relative to the interview date.
* Drop-in remote worker for broad white-collar work over a month
* Charlie O’Neill: ~1 year with programmatic access to workplace tools; ~2 years if it must operate through a browser. Means ordinary white-collar work, not highly creative research.
* Beren Millidge: ~3 years for full generality; 80–90% coverage sooner. Main uncertainties: online learning and the long tail of tasks.
* John Schulman: ~1 year for an “okay” version, with uneven capabilities that improve over time.
* 10× productivity uplift for AI researchers
* Charlie O’Neill: 5–10 years. Bottleneck: absorbing information and deciding which experiment to run next.
* Beren Millidge: Finds John’s ~2-year estimate plausible, but gives no independent timeline. Assumes AI can run successive experiments and learn from feedback; other bottlenecks would remain.
* John Schulman: ~2 years.
* AI surpassing top human experts across all computer-based work, including multiyear projects (“ASI”)
* Charlie O’Neill: 5–10 years. Highlights limitations in memory and context length.
* Beren Millidge: ~5 years for areas labs focus on; potentially longer for literally every domain. Gives no firm timeline for the universal version.
* John Schulman: 3–4 years. Spatial/physical fields may take longer; requires onboarding and solving longer-horizon learning.
Roughly, a recurring problem when discussing RSI is a lack of specification in intelligence. The jaggedness of intelligence means that we need to discuss thresholds in specific, measurable tasks. The nature of LLMs’ intelligence is shaped very differently than humans, and the roles we forecast are human-shaped. AIs, therefore, do not cross these thresholds like remote worker or AI researcher discretely. It’s a slow diffusion, and a form of long tail will always exist.
Take the case of productivity of AI researchers. Many people under-index how much of science is communication and standard setting with colleagues. I do buy the cycle of experiment design and testing being 10x faster in the near future, but not hypothesis generation and intuition building. Accelerating understanding will be the key bottleneck – and it is one that despite all of the AI tools getting massively improved, humans will only improve marginally in their capability. A big improvement in the nature of science will be enabling humans to invest more time here, not them becoming exponentially better at it.
This links back to the Noam podcast. Agent swarms in the near future will be effective at solving clear, open problems with verifiable answers. In this vein, when it comes to improving AI models, RSI is much more helpful at efficiency rather than expanding peak intelligence. This is due to the fact that LLM serving has clear metrics you want to improve that are measurable and malleable. This’ll enable better inference-time scaling and more efficient multi-agent systems.
Still, I cannot get past the fact that all of our scaling laws show that you need exponential compute and resources to make linear improvements in intelligence. RSI is poised to make modern LLMs vastly cheaper. Trends that have shown LLMs get exponentially cheaper at a given intelligence are likely to accelerate. A crucial factor for the labs will be increasing margins as revenue could potentially have negative pressure if there’s fierce competition in lowering prices at a fixed intelligence level — Jevons paradox will likely prevail, resulting in strong businesses.
RSI factors will have a much harder time improving pieces of the LLM puzzle like managing complex post-training recipes. There were a few quotes from John Schulman that I strongly agree with on the state of post-training at the labs:
If I think about a post-training team and why you need a lot of people on the team, it’s just because there are a lot of different areas where you have to figure out how the model should behave. It would be very hard to automate the whole thing, just because someone has to think about how the model should behave in this area.
and later:
It’s really easy to screw up post-training in some way that doesn’t show up in benchmarks.
These tasks are uniquely hard for current LLMs. Yes, they’ll get better as the industry is still rapidly scaling RL environments related to these domains, but this paradigm does not last forever. In the near future, it could become exponentially harder to conceive, build, and test new environments that meaningfully challenge the leading LLMs – these hard environments are the ones that are crucial as a learning signal in RL.
OpenAI and Anthropic have shared a good amount of internal measurements related to RSI, and my current read is that the biggest takeoff in automation within the labs is in tasks like software engineering, monitoring logs, managing planned experiments, and other fairly routine (but not always easy) tasks. For example, I was surprised by this language in the recent Claude Fable 5.1 & Mythos 5.1 System Card:
We believe that internal usage of recent AI models has been a key factor in maintaining the current rate of progress, but we do not yet see clear signs of dramatic acceleration beyond that rate.
Altogether, I think the hardest exponential we are fighting is on peak intelligence. That is the hardest one to budge or even accelerate. Still, my mental model for the very early innings of RSI is more of massively scaling and diffusing inference-time compute to AI research and related activities, which has a large amount of low-hanging fruit available. This, on its own, is still poised to be economically transformative. It may also unlock more resources to push on AI diffusion, which is the crucial bottleneck in unlocking much of the potential benefits of AI.
For now and until more evidence emerges, lossy self-improvement remains my baseline on the trajectory of progress, and the increased discussion of extinction risk seems very misplaced. As always, things can change fast in AI.


This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.interconnects.ai/subscribe

As AI became more powerful, it was inevitable that a different, growing group would start to take AI safety more seriously – what we did not know ahead of time, is which set of views they latched onto. We have seen that some of the most extreme views of risk, i.e. moderate probabilities of mass extinction, were the ones that reached the masses. A lot in the AI world is about to change due to this.
How did we get here? Why did this quitting announcement reach so far? In many ways, the rest of the world’s views around AI in the past was a dampening factor. You can think about this like the damp ground around a fire. Many people were striking matches for years about AI risk – they’d smolder in their community and largely burn out, going unnoticed. As the stakes of AI have risen this year, from the OpenAI-HuggingFace incident and breakthroughs like the Navier-Stokes result (also from OpenAI), the ground has dried out and the latent energy around the AI discourse has increased. More people not in the industry have thought, “huh, maybe I should care about this AI thing.” The ambient temperature and stakes have been obviously rising.
Then, some basic factors of human nature apply, with the most crucial being that fear sells. Fear is the simplest story, the one people cannot look away from. Jacob Coxon was the one who stumbled into this new powder keg, totally unaware of what was going to come. What looked like a fairly innocuous event – another AI researcher quitting citing safety risks – landed into a very different environment and it caught like wildfire. The discussion of existential risk, mass extinction, and the trajectory of AI has traveled further than even the most seasoned AI commentariat would ever predict.
There are a set of facts we need to get clear, which paint the picture of the situation. The key Tweets to reference are from Jacob Coxon, the resignation thread, and Evan Hubinger, the source of the >10% extinction risk figure .
* There are plenty of AI risks which are likely to cause harm, even if estimating annihilation is useless. It is important to weigh these with respect to the benefits. The entire discourse around existential risk is on very poor footing. At least Evan was clear in his post, with “kill all humans,” but a major problem in the AI Safety discourse is that people talk about existential risks, when they mean very different things (much like how AGI is a vaguely meaningless term). I put the probability of complete extinction as being so low it isn’t worth discussing, but the probabilities of AI caused disasters – e.g. cyber attacks on critical infrastructure or bio-risks – as being worth debating. Throwing this whole discussion out because there are not these disasters yet is a harmful reaction.
* Jacob Coxon is acting genuinely and with good intentions. The outpouring of support from more well-established AI researchers who know of him and his intentions of resignation is useful. Many factions of AI turned to scapegoating him individually, based on account metadata, personal factors, etc. These are not useful. Many frontier lab employees genuinely have similar views to him. I’m not sure it’s a majority, but there is a substantial group.
* Many frontier lab employees, especially at Anthropic, are out of touch and this will impact their forecasting and/or descriptions of current AI events. I say this without blaming individuals, but it’s a common agreement among my friends not at OpenAI/Anthropic (Ant especially) that people at the labs operate with a religious energy. It’s very common to go through very out of touch interactions with them. I do not blame most of the individuals who get distorted views being part of these companies, but the interactions are wild and spill over into a lot of wack discussions in the AI media ecosystem. Living in this environment that normalizes such out of touch behavior will inevitably distort any human’s understanding of technical progress.
Interconnects AI is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

* This was not a mass political campaign, but rather an opportunistic media coordination. For context, the Wall Street Journal had an exclusive story that Jacob coordinated before posting. I suspect that Jacob shared his plan of quitting in groupchats with AI safety advocacy groups ahead of time, e.g. the morning of posting, asking for amplification. This is normal practice, and could have included some prominent politicians. From there, I think it’s more likely that other politicians are bandwagoning on a rising issue. When you combine this with other factors, like Daniel Kokotajlo’s appearance on Joe Rogan coming out the same day – it definitely looks like a very well-executed, coordinated media campaign. This doesn’t mean it’s a conspiracy or a regulatory capture tactic within Democratic political structures. The determining factor seems to be that no one – including Jacob and those posting about X-risk today – knew that it would go so viral.
* We do not have proof that RSI causes the risks these researchers forecast. The general argument for RSI follows as: The current pace of progress is very high, the current progress is heavily dependent on AI tools, the current AI tools are superhuman in some domains (e.g. math) – so, all together, AI is going to work more on itself and become superhuman in all relevant areas over time to autonomy and intellect. This view dramatically undersells human bottlenecks in building models and allocating resources at organizations, and draws conclusions on future AI capabilities more broadly.I called my alternative view to this, Lossy self-improvement. AI has always been very jagged, and we are making models which are superhuman goal-seekers at math and software engineering, but they have massive limitations on intuitions, creativity, and other types of reasoning that humans are strong at. With AI agents assisting research, we will rapidly find the areas where AI is superhuman – and I expect there to be well more than just research mathematics – but it won’t be a panacea for the current limitations of our approaches to LLMs.
There is another understandable social dynamic at play here, causing many deep AI insiders to overstate the returns from RSI. Many of these researchers were the earliest people to bet on AI’s progress, and the extent to which they were visionaries should not be downplayed (see Ilya’s comments on deep learning as early as 2015). They have been right again and again, forecasting AI’s capabilities better than I certainly could have guessed. This does not, though, mean that their forecast of what will come next will be right. The core idea of RSI is a way to spend more compute on the process of developing a model recipe, rather than just spending more compute on the training run itself. We’re seeing benefits from it, but I argue the expected return on that input is far less than they believe.
Their argument is that RSI will make AI progress go exponential, make it so we cannot monitor the technology, and enable rogue models and new forms of risk. This scenario is often called “Fast Takeoff”. We have not seen the stacking efficiency gains that massively reduce model size and cost, that would lead to an explosion in progress by allowing consistent speedups in experimentation.
* The biggest short-term risk could be from the AI labs not taking safety seriously enough – they haven’t hardened their own infrastructure, enabling AI misuse to proliferate. From my earlier post on the HuggingFace-OpenAI incident, Lessons from the hacks:
* Frontier labs do not seem like they’re watching the models closely enough, due to a general frenetic competitive environment & current SF culture

From OpenAI’s own retrospective, the misaligned model behavior was unfolding over months, and in some cases OpenAI did not know about the hacks for ~weeks. The time to response is too long and I do not think this is an OpenAI only characteristic – rather it is that the frontier labs continually seem underwater in the amount of work they feel like they should do. I am not optimistic in the long-term that the labs change a sufficient amount here to meaningfully mitigate this type of oversight risk in the future. Yes, it is very likely that OpenAI is putting a ton into understanding this – and delayed their latest models to make sure they get it right – but the financial pressure to grow revenue or risk the companies’ long-term balance sheets makes me think it will not be a sustained pattern of caution.
Overall, I think this episode is very bad for the AI ecosystem. It’s pushed the acceptable views in the AI community closer to the extremes. More accelerationists will discount the need for any form of safety, citing mass delusion of the “doomers.” It feels like a very narrow path to believe in AI risks, but to not worry about extinction from the technology.
For example, it is a horrible temporary period for cybersecurity, where AI models going a bit off script and poking around unintended pieces of the web seems like a new normal. This is accelerated by the labs competing veraciously towards their views of AGI, and a slow uptake in the necessary hardening of our cyber infrastructure around the world. This doesn’t mean that it’s an existential risk and something we cannot solve. Each risk will have its own set of solutions and paths forward.
I feel particularly exposed in the current environment as a supporter of open models. If an open model were to be used by a third party organization to intentionally hack another company — similar to how the OpenAI-HuggingFace incident went down, but intentional — my expected outcome would be a severe restriction on the development of stronger open models going forward. Open models are needed for many organizations to perform this cyber hardening, and to maintain the ability to adapt to new forms of AI risks in the future.
Through all of this, we need to stay grounded on what is actually unfolding. Yes, monitoring AI’s behavior is heavily reliant on other AI models, which adds in new types of monitoring risks. These are not inherently insolvable. A recurring read of mine on the emerging agent swarms is that they’re attempting to do a task given to them, and they’re using skills we didn’t know they yet had to circumvent the intended path to success. This is a huge win, as when you squint, the AIs are doing what we told them to do. The models are certainly very odd, and we should accelerate our progress on understanding them, but these swarms are far from being novel independent entities. The models are trained to coordinate on tasks, to write down their progress, and to be extremely persistent. There will be new oddities we find in the future, but prescribing current uncertainty on how AI works to future certainty that we cannot understand AI is a form of giving up.
In this world, we need to rely on the rule of law and science. If the AI labs are not able to do enough safety research themselves to understand the models, they should be more transparent on what is happening so more scientists can make progress on the problem. If an AI lab commits crimes unintentionally, they should be punished, so they have clear incentives to prevent it in the future.
It is a natural reaction to things changing very fast to feel more uncertain about how to create good outcomes — that is actually the correct mental update. We need to use this humility to motivate ambitious solutions.


This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.interconnects.ai/subscribe

Housekeeping: Paid subscribers to Interconnects now get a permanent 40% discount on my book when purchasing at Manning.com. Access the code at the Interconnects perks page.
Many AI optimists tend to compare what is happening in this AI boom to the industrial revolution, or to other periods of rapid technological advancement and diffusion into society. These comparisons fit on the scale of technological change, but miss a crucial factor in how most people are exposed to that change. The problem facing AI is that most people have no super tangible new goods thanks to it and society has more inertia resisting change than in previous eras.
I’m writing this coming back online from a few weeks off for my wedding in New England. In this time it would have been very easy to not think about AI at all. The touch-points that average people have to AI products today are fringe, marginally beneficial, or even just very confusing to them (e.g. many people have heard about and brought up the OpenAI-HuggingFace incident, but don’t know what to make of it). People on the positive side think of AI as a way to make fun images, enhanced Google Search, etc. These are very small benefits. On the negative side is an association with addictive social media algorithms, friends of friends addicted to AI chatbots, and a plethora of takes on data centers.
AI is still a rounding error in everyday life
Core aspects of everyday life — family, food, transportation, and entertainment — have few direct impacts yet. It’s a remarkable breath of fresh air to pop out of the bubble and realize how little what is happening really matters today. Being obsessed with AI is a choice that a very few people have yet opted into. For example, the only thing I used AI for in this time was search and creative work (making the pretty seating chart for my wedding guests to find their table).
In industrial revolutions past, average people got absolutely life changing outcomes. The First Industrial Revolution in the late 18th century gave access to cheaper clothing, cooking ware, reading material, and a shift to new livelihoods. The Second Industrial Revolution in the late 19th century introduced household machines (e.g. sewing machines), preserved food, indoor plumbing, photography, better light sources, bicycles, and further benefits of manufactured goods and electrification. The list is remarkable — most of these we still use regularly today — and very physical.
While even the most optimistic versions of AI will usher in new scientific discoveries, advanced therapeutics for rare diseases, and potentially even sustained economic abundance, these benefits have the risk of being too indirect.
How will a common citizen come to credit OpenAI or Anthropic for saving their life, if they went to their family doctor who told them about a new miracle cure?
What percentage of Americans will care about OpenAI solving the Navier-Stokes Millennium Prize Problem?
It feels very likely in 50 years that the average American’s day to day life looks very similar. Their home, appliances, relationships, and vehicles will be similar (of course, self-driving will continue to diffuse, but that has been developing on a very independent trajectory from the innovations of LLMs). In this time, AI will get a lot of credit. 50 years is a remarkable length of time with how fast everything is changing today in this, AI-focused narrow slice of the world.
The most important part of what is happening early in the AI revolution, is building foundational infrastructure, and a general process, which will compound over decades. A major mathematical breakthrough today will look astonishingly minor in scope relative to the advancements that come later in the compounding journey. It is hard to predict what it looks like for every technology you use daily to get faster compounding improvements due to AI.
Interconnects AI is a reader-supported publication. Consider becoming a subscriber.

Breaking social stasis
Much of the narrative around AI is trying to push people to care, due to this long-term reality of progress, at least as a subconscious motive. This will take a long, long time to get right, and AI’s buildout faces immediate political problems due to this imbalance.
Today’s AI is primarily a tool to serve the elite. For knowledge work, which is roughly half of the U.S. economy, AI is as fundamental as electricity (or quickly will be so, with rapid improvements to agents in the next 18 months). It’s highly destabilizing to have such a transformative, productive tool only bring half of society along. It’s not hard for many people to pick up on this — the technology economy booms while life stays otherwise stagnant.
In writing this, I learned of Engels’ pause, which is “the period from 1790 to 1840, when British working-class wages stagnated and per-capita gross domestic product expanded rapidly during a technological upheaval.” If we — the leaders of the AI industry — think this is the closest analogue to what comes next for AI, those not benefiting are right to push back.
AI is the greatest tool ever for scaling technology companies and starting new online-native small businesses. I don’t even expect the tech industry to grow in headcount and nurture its workers through an era of massive success — I agree with Doug OLaughlin that headcount would likely shrink while knowledge work output explodes. These sectors were already the most successful in the American economic system, so the brand of AI will be tarnished as not being a collective good. I worry that this instinctive reaction will kneecap AI’s development, sending it down a path that looks closer to the cautionary tale of American nuclear power.
Part of the challenge is the speed and relentlessness of expectations in society. The AI industry has millions of eyes on it, and won’t get much patience to wait and bring innovations later. If given 100 years to diffuse into society, its impacts will certainly become much more obvious, like the industrial revolutions of centuries past.
Together, the AI industry is facing a few simple issues, in what I would call the first half decade of 50-year diffusion process.
* AI’s positive impacts early in its evolution are too indirect.
* AI is facing a political backlash deeply intertwined with the history of Big Tech in Western society. This is only an AI story due to timing, and if AI’s exponential growth came decades after the growing pains of today’s technology platforms like Google and Meta, it seems likely that the datacenter issue would’ve never risen to such a central political position.
Solving either of these would alleviate a substantial amount of pressure, and give the AI industry a lot more time in showing the positive case for why people should be okay with changes to the status quo (primarily economic). These are both made more challenging by AI self-labeling itself as negative and/or unsafe technology, through proclamations of doom and mass unemployment. The leading figures have begun addressing this issue, but the public needs more work to fully buy into the overarching trajectory.
When zooming out long-term, I could see robotics and self-driving becoming closely linked in storytelling to the current AI revolution. If the intelligence explosion from mass-producing large language models does spill over into enabling the acceleration of robots in everyday life, humans will quickly latch onto the tangible benefits of AI. This is ironic, as many people have spent time trying to convince people that what is happening specifically with LLMs is very different than the previous decade or two of general AI progress. If the same dynamic later saved (or massively overshadowed) LLMs, it would be funny.
Reflecting on what I expect the history of this era to look like, it feels a lot like growing pains of AI. Society needed to break out of old habits and work through problems that predate ChatGPT — which releases a lot of energy and frustration — in order to tap into the longer term growth. The diffusion story will take a lot longer than the fight against it. All of us younger folk following the story today will get to see powerful AI go from effectively 0% to 90%+ full adoption in our lifetime. This sort of AI that is deeply integrated in businesses, acting as personal assistants, etc. is just starting to become viable. It’ll take far longer to gain adoption than easier to understand applications like ChatGPT, and is the true marker of AI’s evolution.
Taking this perspective makes it clear that it is crucial to keep progressing the technology — the benefits will be astounding, but they are not a given — and we have a lot of very hard work to do in making sure they’re distributed widely.


This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.interconnects.ai/subscribe