Orpheus Lummis

Upskilling in contemporary AI techniques, deep RL and AI safety, before pursuing a ML PhD

From the application :

Notable planned subprojects:

• Engaging with David Krueger’s AI safety reading group at Mila
• Starting & maintaining a public index of AI safety papers, to help future literature reviews and to complement [https://vkrakovna.wordpress.com/ai-safety-resources/](https://vkrakovna.wordpress.com/ai-safety-resources/), as a standalone wiki-page (eg at [http://aisafetyindex.net](http://aisafetyindex.net/))
• From-scratch implementation of seminal deep RL algorithms
• Going through textbooks: Goodfellow Bengio Courville 2016, Sutton Barto 2018, …
• Possibly doing the next AI Safety camp
• Building a prioritization tool for English Wikipedia using NLP, building on the literature of quality assessment ([https://paperpile.com/shared/BZ2jzQ](https://paperpile.com/shared/BZ2jzQ))
• Studying the AI Alignment literature

My thoughts and reasoning

We funded Orpheus in our last grant round to run an AI Safety Unconference just after NeurIPS. We’ve gotten positive testimonials from the event, and I am overall happy about that grant.

I do think that of the grants I recommended this round, this is probably the one I feel least confident about. I don’t know Orpheus very well, and while I have received generally positive reviews of their work, I haven’t yet had the time to look into any of those reviews in detail, and haven’t seen clear evidence about the quality of their judgment. However, what I have seen seems pretty good, and if I had even a tiny bit more time to spend on evaluating this round’s grants, I would probably have spent it reaching out to Orpheus and talking with them more in person.

In general, I think time for self-study and reflection can be exceptionally important for people starting to work in AI Alignment. This is particularly true if they are following a more conventional academic path which could easily cause them to try to immediately work on contemporary AI capabilities research, because I generally think this has negative value even for people concerned about safety (though I do have some uncertainty here). I think giving people working on more classical ML research the time and resources to explore the broader implications of their work on safety, if they are already interested in that, is a good use of resources.

I am also excited about building out the Montreal AI Alignment community, and having someone who both has the time and skills to organize events and can understand the technical safety work seems likely to have good effects.

This grant is also the smallest grant we are funding this round, making me more comfortable with a bit less due diligence than the other grants, especially since this grant seems unlikely to have any large negative consequences.