Joe Collman

Research on amplification approaches to AI alignment.

We are funding Joe to continue his independent research into improved understanding of amplification approaches to AI alignment. We previously funded Joe in September 2020 and November 2019. I found his work over the past year to have produced several valuable insights. Most notably, he identified a novel problem in the popular AI safety via debate paradigm. While I think there are more pressing challenges for debate, I am glad this problem is now in the literature.

Joe has also developed a new conceptual approach for debate, based on splitting the debater into separate agents with one behind a veil of ignorance, which he argues results in better equilibria. This approach seems promising to me, but I have been unable to conduct a full assessment, as at the time of evaluation there was no detailed write-up.

After interviewing Joe, I think his proposal has significant potential, and so think his work merits continued funding. However, I do have some hesitation here due to the limited tangible research output produced so far. While Joe has made clear progress, I would usually like to see a clear track record of successful research projects after several years of funding. However, Joe has been focused on high-risk, high-reward research problems, which by their nature may not produce immediate positive results. Given this, I am inclined to give the benefit of the doubt here, but I intend to place greater weight on concrete output in evaluations of any future renewals.

Developing algorithms, environments and tests for AI safety via debate.

Joe was previously awarded $10,000 for independent research into extensions to AI safety via debate. We have received positive feedback regarding his work and are pleased to see he has formed a collaboration with Beth Barnes at OpenAI. In this round, we have awarded $25,000 to support Joe's continued work and collaboration in this area.

Joe intends to continue collaborating with Beth to facilitate her work in testing debate in human subject studies. He also intends to develop simplified environments for debate, and to develop and evaluate ML algorithms in this environment.

In general, I apply a fairly high bar to funding independent research, as I believe most people are more productive working for a research organization. In this case, however, Joe has demonstrated an ability to make progress independently and forge collaborations with established researchers. I hope this grant will enable Joe to further develop his skills in the area, and to produce research output that can demonstrate his abilities to potential employers and/or funders.

Investigation of AI Safety Via Debate and ML training.

From the application:

I think that David Krueger is doing good work in the space of AI alignment, and that funding Joe to work on things David thinks are important seems worth the small amount of requested funding. We recommended this grant mostly on the basis of referrals and testimonials. David has been collaborating with many good people I trust quite a bit over the past few years (FHI, Deepmind, CHAI and 80k), so that’s where a lot of my trust comes from.