Lee Sharkey

Researching methods to continuously monitor and analyse artificial agents for the purpose of control.

Lee applied to spend the year before his PhD researching methods for monitoring the internal processes of artificial agents while they're running, which would allow us to detect and respond to undesirable internal behavior. His work will consist of doing a literature review of existing work before trying to make progress on methods that are potentially promising. This research direction has a lot in common with work on the interpretability of neural networks, though it is additionally focused on finding methods that can be used to understand agent internals in real time.

Lee was previously a Masters student at ETH Zurich studying Neural Systems and Computation. During his Masters, Lee worked with Valerio Mante, who was doing work on understanding recurrent neural network models of the brain using an interpretability method called dynamical systems analysis. I'm not sure that dynamical systems analysis in particular is likely to be a scalable solution to real-time monitoring, but I'm interested in supporting Lee doing more work in this area and think he will benefit from his connections to existing academic researchers. Lee also shared some preliminary work he did on a related research direction with the fund; I thought the work was interesting and showed that he was likely to make progress working as an independent reinforcement learning (RL) researcher.

I also shared Lee's proposal with a few AI safety researchers, including people working on interpretability; overall, they thought this was a potentially promising research direction.