David Manheim

• We’ve given multiple grants to David in the past (example). In this case, David was planning to work with FHI, but FHI was unable to pay him for his time. To enable him to continue doing work on longtermist policy, we offered to cover his salary at ALTER, the new organization he has set up. I did not evaluate this grant in great depth, given that FHI would have been happy to pay for his time otherwise.
• so we offered to cover his salary. I did not evaluate this grant in great depth, given that FHI would have been happy to pay for this.

Building understanding of the structure of risks from AI to inform prioritization.

Recusal note: Daniel Eth and Ozzie Gooen did not participate in the voting or final discussion around this grant.

We made a grant to David Manheim in the August 2019 round. In that grant report, I wrote:

However, I am excited about this grant, because I have a good amount of trust in David’s judgment. To be more specific, he has a track record of identifying important ideas and institutions and then working on/with them. Some concrete examples include:

• Wrote up a paper on Goodhart’s Law with Scott Garrabrant (after seeing Scott’s very terse post on it)
• Works with the biorisk teams at FHI and OpenPhil
• Completed his PhD in public policy and decision theory at the RAND Corporation, which is an unusually innovative institution (e.g. this study);
• Writes interesting comments and blog posts on the internet (e.g. LessWrong)
• Has offered mentoring in his fields of expertise to other people working or preparing to work projects in the x-risk space; I’ve heard positive feedback from his mentees

Another major factor for me is the degree to which David shares his thinking openly and transparently on the internet, and participates in public discourse, so that other people interested in these topics can engage with his ideas. (He’s also a superforecaster, which I think is predictive of broadly good judgment.) If David didn’t have this track record of public discourse, I likely wouldn’t be recommending this grant, and if he suddenly stopped participating, I’d be fairly hesitant to recommend such a grant in the future.

As I said, I’m not excited about the specific project he is proposing, but have trust in his sense of which projects might be good to work on, and I have emphasized to him that I think he should feel comfortable working on the projects he thinks are best. I strongly prefer a world where David has the freedom to work on the projects he judges to be most valuable, compared to the world where he has to take unrelated jobs (e.g. teaching at university).

Since then, I haven’t received any evidence that contradicts this perspective, so I continue to think that it’s valuable for David to be able to work on projects he thinks are valuable, especially if he plans to write up his findings and thoughts publicly.

However, the scope of this project is broader than just David’s personal work, so it seems worthwhile to further explain my take on the broader project. From the application:

This grant will fund a group of researchers who are collaborating on a project working to improve understanding of the structure of risk from AI, with the aim of making results publicly available to AI safety researchers and academia. The project started as a collaboration between researchers from several institutions including FHI and APL, so while several members of the group already have funding, many do not. APL has declined funding for year 2, so we are going to take the project forward with a slightly different focus.

The project goals for year 2 (2020-2021) are first, to refine the model for uncertainties that drive the different risks which emerge from AGI and/or ASI and to write up our preliminary understanding in a series of posts on the Alignment Forum. Second, to continue developing and perform a series of elicitations to understand what AI safety experts think about both the specific uncertainties, and the structure of how these uncertainties interrelate to create risk, and then to publicise those results along with an interactive probabilistic model. In the process we will refine some inaccessible or unclear arguments so that they are suitable for academic treatment, and hopefully improve the quality of communication around AGI and ASI risk scenarios. The people who will receive the funding are being given time to think through and write more about AI safety, and we will continue to encourage them to write papers and LessWrong or Alignment Forum posts about their insights or questions.

I am co-leading the project with Daniel Eth, with assistance from Sammy Martin. The group of collaborators are involved in AI safety research already, but most have other work and/or do not have funding that allows them to focus on AI safety. The individuals are David Manheim (myself), Issa Rice, Ben Cottier, Ross Greutzmacher, Jeremy Perret, Alexis Carlier, and Sammy Martin. Daniel Eth and Aryeh Englander are also working extensively on the project, but they are funded at FHI as a Research Scholar and at APL respectively. We are also consulting with other groups, including AI Impacts and GCRI. The direct near-term impact of the project is expected to be several-fold. We believe, and have been told by others, that the direct impact of the work in clarifying and cataloguing which uncertainties exist and how they are related is of high value, and in general this issue is not being tackled by individual researchers outside our group. Our team members agree that there is a real need for a formalization and an organization of the arguments around AGI and ASI risk.

I’ve observed the work of a good number of the people involved in this project, and the group seems pretty promising to me, though I don’t know all of the people involved.

I do have some uncertainty about the value of this project. In particular, it feels quite high-level and vague to me — which isn’t necessarily bad, but I feel particularly hesitant about relatively unfocused proposals for teams as large as this. My current best guess as to the outcome of this grant is that a number of people who seem like promising researchers have more resources and time available to think in a relatively unconstrained way about AI risk.

Multi-model approach to corporate and state actors relevant to existential risk mitigation.

I feel a lot of skepticism toward much of the work done in the academic study of international relations. Judging from my models of political influence and its effects on the quality of intellectual contributions, and my models of research fields with little ability to perform experiments, I have high priors that work in international relations is of significantly lower quality than in most scientific fields. However, I have engaged relatively little with actual research on the topic of international relations (outside of unusual scholars like Nick Bostrom) and so am hesitant in my judgement here.

I also have a fair bit of worry around biorisk. I haven’t really had the opportunity to engage with a good case for it, and neither have many of the people I would trust most in this space, in large part due to secrecy concerns from people who work on it (more on that below). Due to this, I am worried about information cascades. (An information cascade is a situation where people primarily share what they believe but not why, and because people update on each others' beliefs you end up with a lot of people all believing the same thing precisely because everyone else does.)

I think is valuable to work on biorisk, but this view is mostly based on individual conversations that are hard to summarize, and I feel uncomfortable with my level of understanding of possible interventions, or even just conceptual frameworks I could use to approach the problem. I don’t know how most people who work in this space came to decide it was important, and those I’ve spoken to have usually been reluctant to share details in conversation (e.g. about specific discoveries they think created risk, or types of arguments that convinced them to focus on biorisk over other threats).

I’m broadly supportive of work done at places like FHI and by the people at OpenPhil who care about x-risks, so I am in favor of funding their work (e.g. Soren’s grant above). But I don’t feel as though I can defer to the people working in this domain on the object level when there is so much secrecy around their epistemic process, because I and others cannot evaluate their reasoning.

However, I am excited about this grant, because I have a good amount of trust in David’s judgment. To be more specific, he has a track record of identifying important ideas and institutions and then working on/with them. Some concrete examples include:

• Wrote up a paper on Goodhart’s Law with Scott Garrabrant (after seeing Scott’s very terse post on it)
• Works with the biorisk teams at FHI and OpenPhil
• Completed his PhD in public policy and decision theory at the RAND Corporation, which is an unusually innovative institution (e.g. this study);
• Writes interesting comments and blog posts on the internet (e.g. LessWrong)
• Has offered mentoring in his fields of expertise to other people working or preparing to work projects in the x-risk space; I’ve heard positive feedback from his mentees

Another major factor for me is the degree to which David is shares his thinking openly and transparently on the internet, and participates in public discourse, so that other people interested in these topics can engage with his ideas. (He’s also a superforecaster, which I think is predictive of broadly good judgment.) If David didn’t have this track record of public discourse, I likely wouldn’t be recommending this grant, and if he suddenly stopped participating, I’d be fairly hesitant to recommend such a grant in the future.

As I said, I’m not excited about the specific project he is proposing, but have trust in his sense of which projects might be good to work on, and I have emphasized to him that I think he should feel comfortable working on the projects he thinks are best. I strongly prefer a world where David has the freedom to work on the projects he judges to be most valuable, compared to the world where he has to take unrelated jobs (e.g. teaching at university).