Vanessa Kosoy

This is a grant to cover half of Vanessa’s stipend for two years (the other half being paid by MIRI). We also made another grant to Vanessa in Q4 2020 for a similar amount.

My model of the quality of Vanessa’s work is primarily indirect, having engaged relatively little with the central learning-theoretic agenda that Vanessa has worked on. The work is also quite technically dense, and I haven’t found anyone else who could explain the work to me in a relatively straightforward way (though I have heard that Daniel Filan’s AXRP podcast with Vanessa is a better way to get started than previous material, though it hadn’t been published when I was evaluating this grant).

I did receive a decent number of positive references for Vanessa’s work, and I have seen her make contributions to other conversations online that struck me as indicative of a pretty deep understanding of the AI Alignment problem.

If I had to guess at the effects of this kind of work, though I should clarify I am substantially deferring to other people here in a way that makes me not particularly trust my specific predictions, I expect that the primary effect would be that the kind of inquiry Vanessa is pursuing highlights important confusions and mistaken assumptions in how we expect machine intelligence to work, which when resolved, will make researchers better at navigating the very large space of potential alignment approaches. I would broadly put this in the category of “Deconfusion Research”.

Vanessa’s research resulted in various public blog posts, which can be found *here*.

Creating a mathematical theory of AGI and AGI alignment, based on *the Learning-Theoretic AI Alignment Research Agenda*.

This grant is to support Vanessa for independent research on her learning-theoretic research agenda for AI alignment. Vanessa has been a long-time contributor to AI alignment research, with dozens of posts and hundreds of comments on the AI Alignment Forum.

I sadly haven't had the opportunity to come to a full understanding of Vanessa's research agenda myself, so I primarily relied on external references in order to evaluate this grant. We received strongly positive references from researchers at MIRI and FHI, as well as a few other independent researchers in the field. I wouldn't say that all researchers think that her research is the single most promising open direction we have for AI alignment, but almost all of them agreed that it seems to be an angle worth pursuing, and that Vanessa should have the relevant funding to do more research in this space.

I am familiar with and have benefited from Vanessa's research on quantilization and IRL. She has also written many critiques and comments on the research of other contributors in AI alignment. Examples include this recent comment on Andrew Critch's overview of a number of research areas and their relevance to AI existential risk, these excellent reviews of a number of articles during the 2018 LessWrong Review, and many other critiques and comments.

Overall, of the grants we are making this round, this is the one I am most excited about: While I think that technical work on existential risks from AI is currently one of the most valuable interventions to support with additional funding, I've found it historically very hard to find researchers in this space that I am excited about. My sense is that most people find it very hard to find traction in this space, and to orient on the seemingly insurmountable problems we are facing. Vanessa's research seems like a trailhead that can open more avenues of research, and seems to strike right at the heart of the core problems in technical AI alignment research.