Centre for the Governance of AI, Inc.

Centre for the Governance of AI does alignment research and policy research. It appears to focus primarily on the former, which, as I’ve discussed, I’m not as optimistic about. (And I don’t like policy research as much as policy advocacy.)

Its policy research seems mostly unrelated to x-risk, for example it has multiple reports on AI-driven unemployment (1, 2).

My favorite of its published reports is Lessons from the Development of the Atomic Bomb. It’s written by Toby Ord, who doesn’t work there.

Centre for the Governance of AI has received $6 million from Open Philanthropy.

The org appears reasonably well-funded. I don’t have major complaints about its work, but (1) the work does not look particularly strong and (2) it doesn’t cover the focus areas that I’m most optimistic about.

Setting up a nonprofit focused on longtermist AI governance field-building.

This grant was funding the setup of the Centre for the Governance of AI (GovAI) as an independent nonprofit, co-led by Allan Dafoe and Ben Garfinkel. GovAI has historically been a research centre within University of Oxford, but Allan Dafoe has recently taken a senior role at DeepMind which prevented him from heading the centre as part of the university. GovAI is now spinning off into a nonprofit, while the AI governance researchers at FHI are forming an AI governance research group led by Ben Garfinkel.

The application sought $257,000 of funding to cover 3 months of Markus Anderljung’s salary as he sets up the nonprofit, legal and miscellaneous expenses, and salaries for the nonprofit’s first two hires. $80,000 of their budget was covered by another funder, and we covered the rest.

From the application:

The nonprofit will occupy a unique role in the longtermist AI governance space, similar to that which The Forethought Foundation has played in the global priorities space. The organisation will be in a particularly good position to e.g. build and maintain a community of scholars and practitioners interested in longtermist AI governance, to engage with established academics in fields such as political science and economics, to convene conferences, and to provide scholarships and prizes. It may also be in a particularly good position to do grantmaking and house AI governance projects.

Looking back on GovAI’s history, I’d guess that it’s had most of its impact through training longtermist researchers and practitioners who have gone on to do good work at GovAI and elsewhere. GovAI has close ties to academia, which I suspect has allowed them to recruit talented people who would be less excited to work at a non-academically-affiliated institution.

GovAI has also produced several research outputs that I have found informative, including O’Keefe et al.’s report on the Windfall Clause, Ding’s report on Deciphering China’s AI dream along with his newsletter, and Zaidi and Dafoe’s report on lessons from the Baruch Plan for nuclear weapons. Some of their work has also informed Luke Muelhauser’s thinking.

I’m interested in supporting GovAI’s new nonprofit in recruiting more talented young people to work on AI governance and policy, as well as in facilitating additional research work. I think as a nonprofit, GovAI risks losing some of the academic prestige it used to attract people when it was part of the University of Oxford, but I’m hopeful that its existing reputation and connections have a comparable effect, especially given that there are still affiliated AI governance researchers working at FHI.