David Krueger

Computing resources and researcher salaries at a new deep learning + AI alignment research group at Cambridge.

David, a newly appointed AI faculty member in Cambridge's CBL Lab, applied for funding to support PhD students and to purchase computational resources. David has a strong track record of working on AI safety research. He is a co-author on the ARCHES, Trustworthy AI Development, and Recursive Reward Modeling research agendas. He has also made several relevant technical contributions, including Hidden Incentives for Auto-Inducted Distributional Shift, Out-of-Distribution Generalization via REx, and Active Reinforcement Learning. David has also helped with field-building via co-organizing the AI Safety Unconference held concurrently with NeurIPS.

I am generally excited about supporting academic labs that want to focus on AI safety research. Right now CHAI is the only academic lab with a critical mass of people working on technical AI safety. While there are a handful of other early-stage labs, I believe we still need many more if we are to seriously expand the field of AI safety in academia.

The most direct path for impact I see for additional safety-focused academic labs is allowing PhD students with a pre-existing interest in safety to work on related research topics. Right now, many PhD students end up in labs focused on unrelated topics. Indeed, I've even recommended that prospective students "give more weight to program quality than the ability to pursue topics you consider important." While I stand by this advice, I hope that as safety-focused labs expand, students will no longer have to make this trade-off.

I also expect that students graduating from a safety-focused academic lab will be better prepared to perform safety research than if they had worked in another lab. I’d estimate that they will save somewhere between 6 months and a year of time that would otherwise need to be spent on literature review and clarifying their thinking on safety after graduation.

Perhaps the largest upside, though also one of the more uncertain ones, is an indirect field-building effect. Right now, a large fraction of people working on AI safety have been influenced by the philosophy of effective altruism (EA). While I am grateful for the support of the EA community on this topic, this also highlights a failure to tap into the much larger talent pool of (prospective) AI researchers who aren’t connected to that community. Academic labs can help with this by recruiting PhD students who are interested in, but have little prior exposure to, safety, and by shifting the thinking of other more senior researchers.

Despite this positivity, we do not want to indiscriminately fund academic labs. In particular, most of the best academic AI research happens in the top 20 or so universities. Outside of those universities, I would expect a significant drop-off in the caliber of PhD applicants, as well as the lab being in a worse position to influence the field. On this criterion, Cambridge seems like a good place for a new lab: it is one of the best universities in the world for Bayesian ML, and benefits from the presence of strong mathematics and CS departments. While I would be even more excited by a lab at the likes of Stanford or MIT, which have amongst the strongest AI groups right now, Cambridge is clearly still a good place to do AI research.

David intends to use our funding to support new PhD students and purchase computational resources. Both of these seem like reasonable uses of funding:

• PhD students: I reviewed a list of prospective PhD students and saw several applicants who seem strong, and I would be excited to see them join David's lab. While PhD students can often receive fellowships or scholarships from other sources, these opportunities are limited for international students in the UK. Furthermore, current applicants are not eligible for most of those funding sources this year, because David received his faculty offer after many funding deadlines had already passed.
• Computational resources: The CBL lab that Cambridge is joining has limited in-house computational resources, with a ratio of around 2 GPUs per student. Moreover, the existing resources are fragmented between many small servers and workstations. I spoke to a current student at the CBL who confirmed that there is a shortage of computational resources which both limits the scope of possible experiments and wastes time.

The main risk I see with this grant is that, since the funding is unrestricted, we might be negatively surprised by what it is spent on. In particular, it is conceivable that it could be used to fund a PhD student the LTFF would not have chosen to fund directly. However, unrestricted funding is also more valuable to David, allowing him to make timely offers to students who are faced with short deadlines from competing institutions. Moreover, David has a strong incentive to recruit good students, so it isn’t clear that the LTFF has a comparative advantage in evaluating his prospective students. Overall, these considerations seem to weigh heavily in favour of unrestricted funding, although this does make the outcome harder to predict.

It's also possible that mentorship will cease to be a major bottleneck on the growth of the AI safety field in the near future. Senior safety researchers are graduating in increasing numbers, and many of them are interested in faculty positions. Moreover, my (by no means rigorous) impression is that the growth in PhD candidates is beginning to slow (though is still expanding). If mentorship becomes more plentiful in the future, David’s lab may be significantly less influential than expected. However, I think this grant (barely) clears the bar for funding even on the basis of its expected short-term impact alone (before mentorship "catches up" with the influx of junior research talent).

We thought the Survival and Flourishing Fund would also be likely to want to fund David, and we thought it would be fair to share the costs with them, so we arranged with them that the first $120,000 of any money they decide to give to David Krueger will be given back to us instead.