Richard Ngo

Doing a PhD on understanding the analogy between the development of human intelligence and AGI.

This grant was made prior to our standard cycle.

This grant covers the tuition and living expenses for Richard Ngo's 3-year PhD in the philosophy of machine learning at Cambridge University. Richard applied to do this PhD after the standard funding deadline for PhD students had passed, so he was not able to acquire university funding. This grant enabled him to start the PhD now rather than wait another year for university funding to be available. Richard is also applying for funding from other sources, and will return some of this grant if his other applications are successful.

Richard previously worked as a research engineer at DeepMind, and his continued work there would have been primarily engineering-focused. I was excited to enable him to work on AI philosophy full-time instead. I've been extremely impressed with Richard's previous philosophical work -- in particular, his report on AGI safety from first principles is the best articulation I'm aware of about why AGI might pose an existential threat. He also has a strong background in philosophy and AI: a BA in Computer Science and Philosophy from Oxford, and an MPhil in machine learning from Cambridge. His proposed PhD topic is on the analogy between the development of human intelligence and AGI.

From his application:

I think this is one of the most crucial foundational issues in AI safety. Most of our current beliefs about future AI systems (e.g. that they'll be generally intelligent and pursue large-scale goals) are based on loose analogies to humans. However, early arguments for why AI might be dangerous have been critiqued for relying on formalisms like utility maximisation in inapplicable ways (e.g. by Rohin Shah, Eric Drexler, and myself). Instead, I believe that taking a perspective closer to cognitive science and evolutionary analysis will allow us to generate concepts which, while less precise, may be more useful for predicting and shaping AGI behaviour.

I'd classify Hubinger et al.'s paper on mesa-optimisation (https://arxiv.org/abs/1906.01820) as a good example of how we can get more clarity about AGI by drawing on the analogy to humans. I'd also note that the foundations of the field are still in flux - a significant number of safety researchers think that inner alignment is the hardest problem, yet we didn't even have a term for it until very recently. So I expect further work on clarifying the conceptual foundations of the field to have major benefits for the research that's being done elsewhere.

I agree with Richard's justification for this work.

Looks exciting. My personal view is that there's a lot of progress waiting to be made on theoretical/agent foundations research. The quality of the program will of course depend a lot on the quality of fellows; I'm curious if there are many people already on your radar, or if you think you have good leads there.

A few other thoughts:

• I think trying to persuade people that the alignment problem is hard is often counterproductive. The mindset of "I need to try to solve a extremely difficult problem" is not very conducive to thinking up promising research directions. More than anything else, I'd like people to come out of this with a sense of why the alignment problem is interesting. Happy to talk more about this in a call.

• Some of the selection criteria seem a bit counterproductive. a) "Decent team players, non-disruptive to the group cohesion" seems like a bad way to select for amazing scientists, and might rule out some of the most interesting candidates. And b) "would care about saving the world and all their friends if they thought human extinction was likely" seems likely to select mainly for EA-type motivations which IMO also make people worse at open-ended theoretical research. Meanwhile c) "highly technically skilled" is reasonable, but I care much more about clarity of thinking than literal technical skills.

If the organizers have good reasons to expect high-quality candidates I expect I'd pitch in 5-10k.

Building a culture of hands-on experimentation is probably the best way to do AI safety outreach, and Apart seems to have executed on it really well.