Center for Human-Compatible AI

Center for Human-Compatible AI does mostly technical research, and some advocacy. To my knowledge, the advocacy essentially consists of Stuart Russell using his influential position to advocate for regulation. While that’s good, I don’t think Stuart Russell is personally funding-constrained, so I don’t think marginal donations to the org will help advocacy efforts.

Hiring research engineers to support CHAI’s technical research projects.

Recusal note: Adam Gleave did not participate in the voting or final discussion around this grant.

This grant is to support Cody Wild and Steven Wang in their work assisting CHAI as research engineers, funded through BERI.

Overall, I have a very high opinion of CHAI’s ability to produce good alignment researchers—Rohin Shah, Adam Gleave, Daniel Filan, Michael Dennis, etc.—and I think it would be very unfortunate if those researchers had to spend a lot of their time doing non-alignment-relevant engineering work. Thus, I think there is a very strong case for making high-quality research engineers available to help CHAI students run ML experiments.

However, I have found that in many software engineering projects, there is a real risk that a bad engineer can often be worse than no engineer. That being said, I think this is significantly less true when research engineers work mostly independently, as they would here, since in those cases there’s less risk of bad engineers creating code debt on a central codebase. Furthermore, both Cody and Steven have already been working with CHAI doing exactly this sort of work; when we spoke to Adam Gleave early in the evaluation process, he seems to have found their work to be positive and quite helpful. Thus, the risk of this grant hurting rather than helping CHAI researchers seems very minimal, and the case for it seems quite strong overall, given our general excitement about CHAI.

Hiring a research engineer to support CHAI’s technical research projects.

Over the last few years, CHAI has hosted a number of people who I think have contributed at a very high quality level to the AI alignment problem, most prominently Rohin Shah, who has been writing and updating the AI Alignment Newsletter and has also produced a substantial number of other high-quality articles, like this summary of AI alignment progress in 2018-2019.

Rohin is leaving CHAI soon, and I'm unsure about CHAI's future impact, since Rohin made up a large fraction of the impact of CHAI in my mind.

I have read a number of papers and articles from other CHAI grad students, and I think that the overall approach I see most of them taking has substantial value, but I also maintain a relatively high level of skepticism about research that tries to embed itself too closely within the existing ML research paradigm. That paradigm, at least in the past, hasn't really provided any space for what I consider the most valuable safety work (though I think most other members of the Fund don't share my skepticism). I don't think I have the space in this report to fully explain where that skepticism is coming from, so the below should only be seen as a very cursory exploration of my thoughts here.

A concrete example of the problems I have seen (chosen for its simplicity more than its importance) is that, on several occasions, I've spoken to authors who, during the publication and peer-review process, wound up having to remove some of their papers' most important contributions to AI alignment. Often, they also had to add material that seemed likely to confuse readers about the paper's purpose. One concrete class of examples: adding empirical simulations of scenarios whose outcome is trivially predictable, where the specification of the scenario adds a substantial volume of unnecessary complexity to the paper, while distracting from the generality of the overall arguments.

Another concern: Most of the impact that Rohin contributed seemed to be driven more by distillation and field-building work than by novel research. As I have expressed in the past (and elsewhere in this report), I believe distillation and field-building to be particularly neglected and valuable at the margin. I don't currently see the rest of CHAI engaging in that work in the same way.

On the other hand, since it appears that CHAI has probably been quite impactful on Rohin's ability to produce work, I am somewhat optimistic that there are more people whose work is amplified by the existence of CHAI, even if I am less familiar with their work, and I am also reasonably optimistic that CHAI will be able to find other contributors as good as Rohin. I've also found engaging with Andrew Critch's thinking on AI alignment quite valuable, and I am hopeful about more work from Stuart Russell, who obviously has a very strong track record in terms of general research output, though my sense is that marginal funding to CHAI is unlikely to increase Stuart's output in particular (and might in fact decrease it, since managing an organization takes time away from research).

While I evaluated this funding request primarily as unrestricted funding to CHAI, the specific project that CHAI is requesting money for seems also quite reasonable to me. Given the prosaic nature of a lot of CHAI's AI alignment works, it seems quite important for them to be able to run engineering-heavy machine learning projects, for which it makes sense to hire research engineers to assist with the associated programming tasks. The reports we've received from students at CHAI also suggest that past engineer hiring has been valuable and has enabled students at CHAI to do substantially better work.

Having thought more recently about CHAI as an organization and its place in the ecosystem of AI alignment,I am currently uncertain about its long-term impact and where it is going, and I eventually plan to spend more time thinking about the future of CHAI. So I think it's not that unlikely (~20%) that I might change my mind on the level of positive impact I'd expect from future grants like this. However, I think this holds less for the other Fund members who were also in favor of this grant, so I don't think my uncertainty is much evidence about how LTFF will think about future grants to CHAI.

(Recusal note: Due to being a grad student at CHAI, Adam Gleave recused himself from the discussion and voting surrounding this grant.)