Adam Shimi

• Adam Shimi has been doing independent AI safety research under a previous grant from us, but has found that he is tight on funding and could improve his productivity by receiving an additional top-up grant. Given that we continue to be excited by Adam’s research, and he thinks that the extra funding would be helpful for his productivity, I think this is a very robustly good grant.

Independent research in AI alignment for a year, to help transition from theoretical CS to AI research.

Recusal note: Evan Hubinger did not participate in the voting or final discussion around this grant.

Adam applied to perform independent research on AI safety for 1 year in collaboration with Evan Hubinger at MIRI, whom he has been working with since mid-2020. Adam completed a PhD from IRIT (publications) in 2020 focused on distributed computing theory. Since then, he has written a number of posts on the Alignment Forum.

I particularly liked his Literature Review on Goal-Directedness, which I felt served as an excellent introduction to the area with a useful taxonomy of different motivations (or "intuitions") for the work. This seems like a good example of research distillation, a category of work that is generally under-incentivized both by academia and by the major corporate AI labs. I'm not sure AI safety needs much in the way of distillation at this point (it's still a fairly small field), but at the margin I think we'd likely benefit from more of it, and this need will only grow as the field scales. Given this, I'm generally excited to see good research distillation.

In addition to continuing his research distillation, Adam plans to work on formalizing goal-directedness. He seems well-prepared for this given his CS theory background and grasp of the existing space, as evidenced by his literature review above. I find this direction promising: slippery and imprecise language is common in AI safety, which often leads to miscommunication and confusion. I doubt that Adam or anyone else will find a universally agreed-upon formalism for goal-directedness, but attempts in this area can help to surface implicit assumptions, and I expect will eventually lead to a more coherent and precise taxonomy of different kinds of goal-directedness.

I tend to apply a fairly high bar to long-term independent research, since I think it is hard to be a productive researcher in isolation. There are several factors that make me more positive about this case:

• Adam already has significant research experience from his PhD, making him more likely to be able to make progress with limited oversight.
• Evan Hubinger is spending around 1 hour a week on informal mentorship, which I think helps to provide accountability and an independent perspective.
• Adam is interested in working at an organization long-term, but has found finding a position particularly challenging this year owing to the pandemic.

Given this, it seems reasonable to support Adam for a year to continue his research while he searches for positions.

The main concern I have regarding this grant is that Adam's previous work in distributed computing seems to have had a limited impact on the field, with only 6 citations as of the time of writing. While it's entirely possible that Adam has better personal fit in AI safety, his track record in that field is brief enough that I’m not yet confident about this. However, there are other positive signs: his previous publications were all in reasonable conferences, including one in PODC, which I understand to be one of the top-tier venues in distributed computing. Unfortunately, I am not familiar enough with this area of CS to render my own judgment on the papers, so have instead relied primarily on my impressions of his more recent AI-focused work.