Sören Mindermann

Doing a 3rd and 4th year of a PhD in machine learning, with a focus on AI forecasting.

We are providing financial support to Sören, a PhD student at Oxford advised by Yarin Gal, to improve his research productivity during his PhD. This is a renewal of our grant made in August 2019 at the start of his PhD. Sören anticipates spending money on items such as home office equipment, meal replacements (to spend less time cooking), a cleaning service, and other miscellaneous productivity-boosting costs.

We have made several grants providing supplementary funding to graduate students, such as Vincent Luczkow and an anonymous PhD candidate, so it's worth taking a moment to reflect on our general approach to evaluating these grants. In general, I find the basic case for providing supplementary funding to graduate students fairly strong. The stipend provided to graduate students is often very low, especially outside the US. In Soren's case, his stipend pays £15,000/year ($21,000/year equivalent). I therefore expect him, and other students in similar positions. to be able to productively use additional funding. Concretely, I would anticipate a grant of $20,000/year to be able to "buy" an extra 2 to 8 hours of additional productive work per week, giving a cost of around $50 to $200/hour.

As a rule of thumb, I'd be very excited to buy an additional hour of the median junior AI safety researcher’s time for $50. At $100, I'd still weakly favor this purchase; at $200, I would lean against but feel uncertain. This is very sensitive to personal fit and track record — I could easily go 4x higher or lower than this depending on the candidate.

In the case of existing researchers, I think we should actually be willing to pay slightly more than this. In other words, I would prefer to have 10 researchers who each produce "1.1 units" of research than 11 researchers who each produce "1 unit". My rationale for this is that growing the number of people in a field imposes additional communication costs. Additionally, the reputation of a field depends to some extent on the average reputation of each researcher, and this in turn affects who joins the field in the future. This makes it preferable to have a smaller field of unusually productive researchers than a larger field of less productive researchers.

In addition to increasing direct research output, I anticipate these grants also having positive indirect effects. In particular, the track record someone has in a PhD heavily influences their future job opportunities, and this can be very sticky, especially in faculty positions.

One countervailing consideration is that people already in PhD programs applying for funding are highly likely to continue in the field. The impact on someone's career trajectory might be greater providing funding pre-PhD, especially if that allows them to secure a position they otherwise wouldn't. For this reason, all else being equal I think we should be more excited about funding people for a PhD than topping up funding of someone already in a PhD. However, in practice it's quite rare for us to be able to truly counterfactually change someone's career trajectory in this way. Most talented researchers secure PhD funding from other sources, such as their advisor's grants. When we have made a grant to support someone in a PhD, I expect it has most often merely accelerated their entry into a PhD program, rather than altering their long-run trajectory.

While I do not believe this applies in Sören's case, I also think having a norm that additional funding is available may encourage people to pursue PhDs who would otherwise have been reluctant to do so due to the financial sacrifices required. This is particularly relevant for people later in their careers, who might have to take a significant pay cut to pursue a PhD, and are more likely to have dependents and other costs.

Turning back to the specifics of this grant, Sören's work to date has focused on causal inference. His publications include a more theoretical paper in NeurIPS (a top-tier ML conference), and a joint first-author paper in Science evaluating different government interventions against COVID-19. This seems on par with or exceeding the publication track record of other PhD students in similar programs and seniority. Having a Science paper in particular is very unusual (and impressive) in ML, however I do think this was facilitated by working on such a topical issue, and so may be a difficult feat to repeat.

My biggest concern is that none of the work to date has focused on directly improving the long-term future. There's a tentative case for the COVID-19 work helping to improve governments’ response to future pandemics or generally improve methodology in epidemiology, but I'm unconvinced (although I do think the work looks good from a short-term perspective). Similarly, while Soren's work on causal inference is in a relevant area, and will be useful for skill-building, I don't see a clear pathway for this to directly improve technical AI safety.

However, I think that it can be a reasonable choice to focus more on skill-building during your PhD. In particular, it is often useful to start by imitating the research direction of those around you, since that will be the area for which you can get the most detailed and valuable feedback. While this may limit your short-term impact, it can set you up for an outsized future impact. The catch is that you must eventually pivot to working on things that are directly important.

In Sören's case, I'm reasonably confident such a pivot will happen. Sören is already considering working on scaling laws, which have a clear connection to AI forecasting, for the remainder of his PhD. Additionally, we have clear evidence from his past work (e.g. internships CHAI and FHI) that he has an interest in working on safety-related topics.

Nonetheless, I think research directions can end up surprisingly sticky. For example, you tend to get hired by teams who are working on similar things to the work you've already done. So I do think there's around a 25% chance that Sören doesn't pivot within the next 3 years, in which case I'd regret making this grant, but I think the risk is worth taking.