Machine Intelligence Research Institute

MIRI, concluding that it is highly unlikely alignment will make progress rapidly enough otherwise, has shifted its strategy to largely advocate for major governments coming up with an international agreement to halt AI progress and to do communications, although research still looks to be a large portion of the budget, and they have dissolved its agent foundations team. Hence the book.

That is not a good sign for the world, but it does reflect their beliefs.

They have accomplished a lot. The book is at least a modest success on its own terms in moving things forward.

I strongly believe they should be funded to continue to fight for a better future however they think is best, even when I disagree with their approach.

This is very much a case of ‘do this if and only if this aligns with your model and preferences.’

Donate here, or reach out to harlan@intelligence.org.

The biggest news from MIRI in 2025 is that they published a book. The book was widely read and got some endorsements from important people, including people who I wouldn’t have expected to give endorsements. It remains to be seen what sort of lasting impact the book will have, but the launch went better than I would’ve predicted a year ago (perhaps in the 75th percentile).

MIRI’s 2026 plans include:

• growing the comms team and continuing to promote the book;
• talking to policy-makers, think tanks, etc. about AI x-risk;
• growing the Technical Governance team, which does policy research on how to implement a global ban on ASI.

I’m less enthusiastic about policy research than about advocacy, but I like MIRI’s approach to policy research better than any other org’s. Most AI policy orgs take an academia-style approach of “what are some novel things we can publish about AI policy?” MIRI takes a more motivated approach of “what policies are necessary to prevent extinction, and what needs to happen before those policies can be implemented?” Most policy research orgs spend too much time on streetlight-effect policies; MIRI is strongly oriented toward preventing extinction.

I also like MIRI better than I did a year ago because I realized they deserve a “stable preference bonus”.

In My Cause Selection (2015), MIRI was my #2 choice for where to donate. In 2024, MIRI again made my list of finalists. The fact that I’ve liked MIRI for 10 years is good evidence that I’ll continue to like it.

Maybe next year I will change my mind about my other top candidates, but—according to the Lindy effect—I bet I won’t change my mind about MIRI.

The Survival & Flourishing Fund is matching 2025 donations to MIRI up to $1.3 million.

MIRI used to do exclusively technical research. In 2024, it pivoted to focus on policy advocacy—specifically, advocating for shutting down frontier AI development. MIRI changed its mind at around the same time I changed my mind.

Some observations:

• MIRI gets considerable credit for being the first to recognize the AI alignment problem.
• I have a high opinion of the general competence of MIRI employees.
• Historically, I have agreed with MIRI’s criticisms of most technical alignment approaches, which suggests they have good reasoning processes. (With the caveat that I don’t really understand technical alignment research.)
• Eliezer Yudkowsky’s TIME article publicly argued for AI pause and brought some attention to the issue (both positive and negative). My vibe sense says the article was valuable but who knows.
• Eliezer personally has a strong track record of influencing (some subset of) people with the LessWrong sequences and Harry Potter and the Methods of Rationality.
• I know that MIRI is serious about existential risk and isn’t going to compromise its values.
• Eliezer believes animals are not moral patients, which is kind of insane but probably not directly relevant. (Rule thinkers in, not out.)
• MIRI (or at least Eliezer) says P(doom) > 95%. Some people say this is crazy high and it makes MIRI want to do dumb stuff like shutting down AI. I do think 95% is too high but I think most people are kind of crazy about probability—they treat probabilities less than 50% as essentially 0%. Like if your P(doom) is 40%, you should be doing the same thing that MIRI is doing. You should not be trying to develop AI as fast as possible while funding a little safety research on the side.

MIRI’s new communications strategy has produced few results so far. We know that MIRI is working on a new website that explains the case for x-risk; a book; and an online reference. It remains to be seen how useful these will be. They don’t seem like obviously good ideas to me, but I expect MIRI will correct course if a strategy isn’t working.

Until recently, MIRI was not seeking funding because it received some large cryptocurrency donations in 2021ish. Now it’s started fundraising again to pay for its new policy work.

I consider MIRI a top candidate. It only recently pivoted to advocacy so there’s not much to retrospectively evaluate, but I expect its work to be impactful.

#5: MIRI

MIRI plans to target a general audience, not policy-makers (update 2024-11-20: see correction below). That means they can reach more people but it’s also lower leverage. My guess is that targeting a general audience is worse on balance.

I put PauseAI US higher than the two lobbying orgs because it has such a small budget. Like PauseAI US, MIRI’s strategies are also neglected, but considerably less so. I expect policy-maker outreach to be more effective than MIRI’s approach (credence: 60%).

Lest I give the wrong impression, MIRI is still my #5 candidate out of 28 charities. I put it in the top five because I have a high opinion of MIRI leadership—I expect them to have reasonable prioritization and effective execution.

CORRECTION: MIRI’s technical governance team does research to inform policy, and MIRI has spoken to policy-makers in the US government. This bumps up my evaluation of the org but I’m keeping it at #5 because working with policy-makers is only one part of MIRI’s overall activities.

General Support

The Machine Intelligence Research Institute (MIRI) does computer science and math research on AI alignment. MIRI is a challenging organization to evaluate, both because their public work is hard to evaluate (because much of it does not fit neatly within an existing discipline, and because much of the case for why the research matters is based on hypotheses about how AI research will develop in the future) and because they have decided to make much of their research non-public, for strategic reasons.

Nonetheless, I believe there is sufficient evidence that MIRI is doing good work that it makes sense for the Fund to support them. This evidence includes the track record of their team, the quality of some recent hires, and some promising signs about their ability to produce work that well-established external reviewers consider to be very high-quality—most notably, the acceptance of one of their decision theory papers to a top philosophy journal, The Journal of Philosophy.

My thoughts and reasoning

• MIRI is a 20-year-old research organization that seeks to resolve the core difficulties in the way of AGI having a positive impact.
• Only in recent years have there been routes to working on alignment that have also given you funding, status, and a stable social life. Nowadays many others are helping out the work of solving alignment, but MIRI core staff worked on the problem while all the incentives pulled in other directions. For me this is a strong sign of their integrity, and makes me expect they will make good decisions in many contexts where the best action isn't the locally incentivized action. It is also evidence that if I can’t understand why their weird action is good, that they will often still be correct to do it, and this is an outside view in favor of funding them in cases where I don't have my own inside-view model of why the project they're working on is good.
• On that note, MIRI has also worked on a number of other projects that have attempted to teach the skills behind their general methodology for reasoning quantitatively and scientifically about the world and taking right action. I regret not having the time to detail all the impacts of these projects, but they include (and are not limited to): LessWrong, The Sequences, HPMOR, Inadequate Equilibria, Embedded Agency, and CFAR (an organization I discuss below). I view these as some of the main reasons the x-risk community exists.
• Another outside view to consider is the support of MIRI by so many others whom I trust. Their funders have included Open Phil, BERI, FLI, and Jaan Tallinn, plus a variety of smaller donors I trust, and they are advised by Stuart Russell and Nick Bostrom. They’ve also been supported by other people who I don’t necessarily trust directly, but who I do think have interesting and valuable perspectives on the world, like Peter Thiel and Vitalik Buterin .
• I also judge the staff to be exceptionally competent. Some examples:

In sum, I think MIRI is one of the most competent and skilled teams attempting to improve the long-term future, I have a lot of trust in their decision-making, and I’m strongly in favor of ensuring that they’re able to continue their work.

Thoughts on funding gaps

Despite all of this, I have not actually recommended a large grant to MIRI.

• This is due to MIRI’s funding situation being solid at its current level (I would be thinking very differently if I annually had tens of millions of dollars to give away). But MIRI’s marginal use of dollars at this point of funding seems lower-impact, so I only recommended $50k.
• I feel conflicted about whether it might be better to give MIRI more money. Historically, it has been common in the EA funding landscape to only give funding to organizations when they have demonstrated concrete room for more funding, or when funding is the main bottleneck for the organization. I think this has allowed us to start many small organizations that are working on a variety of different problems.

However, this is all complicated by a variety of countervailing considerations, such as the following three:

• Power law distributions of impact only really matter in this way if we can identify which interventions we expect to be in the right tail of impact, and I have a lot of trouble properly bounding my uncertainty here.
• If we are faced with significant uncertainty about cause areas, and we need organizations to have worked in an area for a long time before we can come to accurate estimates about its impact, then it’s a good idea to invest in a broad range of organizations in an attempt to get more information. This is related to common arguments around “explore/exploit tradeoffs”.
• Sometimes, making large amounts of funding available to one organization can have negative consequences for the broader ecosystem of a cause area. Also, giving an organization access to more funding than it can use productively may cause it to make too many hires or lose focus by trying to scale too quickly. Having more funding often also attracts adversarial actors and increases competitive stakes within an organization, making it a more likely target for attackers.

I can see arguments that we should expect additional funding for the best teams to be spent well, even accounting for diminishing margins, but on the other hand I can see many meta-level concerns that weigh against extra funding in such cases. Overall, I find myself confused about the marginal value of giving MIRI more money, and will think more about that between now and the next grant round.

MIRI is seeking funding to pursue the research directions outlined in its recent update. We believe that this research represents one promising approach to AI alignment research. According to their fundraiser post, MIRI believes it will be able to find productive uses for additional funding, and gives examples of ways additional funding was used to support their work this year.