David Reber

• This funding is to help David improve his productivity and free up time to read up on AI safety while pursuing his AI PhD at Columbia. I think that these are valuable things for David to be doing and I think it will increase his odds of being able to contribute meaningfully to AI safety in the future. That being said, we decided to only fund David’s productivity improvements and not fund a teaching buyout for David during his PhD, as we decided that teaching was likely to be somewhat valuable to David at this point in his career and we were unsure enough about his own work at this point in his career for us to decide that a full teaching buyout didn’t make sense.

Researching empirical and theoretical extensions of Cohen & Hutter’s pessimistic/conservative RL agent.

David applied for funding for technical AI safety research. He would like to work with Michael Cohen to build an empirical demonstration of the conservative agent detailed in Cohen et al.’s “Pessimism About Unknown Unknowns Inspires Conservatism.” David is planning on working on this project at the AI Safety Camp.

On my inside view, I have mixed feelings about creating an empirical demonstration of Cohen et al.’s paper. I suspect that the guarantees surrounding the agent described in that paper are likely to break in a fundamental way when applied to deep learning, due to our inability to really constrain what sorts of agents will be produced by a deep learning setup just by modifying the training setup, environment/dataset, and loss function—see “Risks from Learned Optimization in Advanced Machine Learning Systems.” That being said, I think Cohen et al.’s work does have real value to the extent that it gives us a better theoretical understanding of the space of possible agent designs, which can hopefully eventually help us figure out how to construct training processes to be able to train such agents.

On the whole, I see this as a pretty speculative grant. That being said, there are a number of reasons that I think it is still worth funding.

First, Michael Cohen has a clear and demonstrated track record of producing useful AI safety research, and I think it’s important to give researchers with a strong prior track record a sort of tenure where we are willing to support their work even if we don’t find it inside-view compelling, so that researchers feel comfortable working on whatever new ideas are most exciting to them. Of course, this grant is to support David rather than Michael, but given that David is going to be working directly with Michael—and, having talked with Michael, he seemed quite excited about this—I think the same reasoning still applies.

Second, having talked with Michael about David’s work, he seemed to indicate that David was more excited about the theoretical aspects of Michael’s work, and would be likely to do more theoretical work in the future. Thus, I expect that this project will have significant educational value for David and hopefully will enable him to do more AI safety work in the future—such as theoretical work with Michael—that I think is more exciting.

Third, though David initially applied for more funding from us, he lowered his requested amount after he received funding from another source, which meant that the overall quantity of money being requested was quite small, and as such our bar for funding this grant was overall lower than for other similar but larger grants. This was not a very large factor in my thinking, however, as I don’t believe that the AI safety space is very funding-constrained; if we can find good opportunities for funding, it’s likely we’ll be able to raise the necessary money.