Caleb Parikh

Please note this grant has been approved but at the time of writing it has not been paid out.

• We funded Robert Long to run a workshop on the science of consciousness for current and near-term AI systems. Robert and his FHI colleague, Patrick Butlin, began the project on consciousness in near-term AI systems during their time at FHI, where they both worked in the digital minds research group. Since January of this year, Rob has been continuing the project while a philosophy fellow at CAIS. There are surprisingly few people investigating the consciousness of near-term AI systems, which I find pretty worrying given the rapid pace of progress in ML. I think that it’s plausible we end up creating many copies of AI systems and use them in ways that we’d consider immoral given enough reflection , in part due to ignorance about their preferences. The workshop aimed to produce a report applying current theories of consciousness (like integrated information theory and global workspace theory) to current ml systems.
• I think that Rob is an excellent fit for this kind of work; he is one of the few people working in this area and has written quite a lot about AI consciousness on his blog. He has a PhD in philosophy from NYU, where he was advised by David Chalmers, and has experience running workshops (e.g. in 2020, he ran a workshop on philosophy and large language models with Amanda Askell).

Please note this grant has been approved but at the time of writing it has not been paid out.

• Jeffrey Ladish applied for funding to set up an organisation to do AI risk communications, with a focus on cybersecurity and alignment risks. His organisation, Palisade Research Inc., plans to conduct risk assessments and communicate those risks to the public, labs and the government. The theory of change is that communicating catastrophic risks to the public and key decision makers could increase political support for slowing down AI and other measures that might reduce AI risk. I am particularly excited about Jeffrey’s organisation demonstrating offensive AI cyber capabilities and other demos that help to communicate current risks from advanced AI systems.
• I am pretty excited about Jeffrey’s organisation. He has worked on information security in various organisations (including Anthropic), he seems well-networked amongst people working in think tanks and AI labs, and I like his public writing on AI risk. I am generally sceptical of people doing work related to policy without having first worked in lower stakes positions in similar areas first, but I thought that Jeffrey was orienting to the downsides very reasonably and doing the sensible things, like developing plans with more experienced policy professionals.

• SERI MATS is a program that helps established AI safety researchers find mentees. The program has grown substantially since we first provided funding, and now supports 15 mentors, but at the time, the mentors were Alex Gray, Beth Barnes, Evan Hubinger, John Wentworth, Leo Gao, Mark Xu, and Stuart Armstrong. Mentors took part in the program in Berkeley in a shared office space.
• When SERI MATS was founded, there were very few opportunities for junior researchers to try out doing alignment research. Many opportunities were informal mentorship positions, sometimes set up through cold emails or after connecting at conferences. The program has generally received many more qualified applicants than they have places for, and the vast majority of fellows report a positive experience of the program. I also believe the program has substantially increased the number of alignment research mentorship positions available.
• I think that SERI MATS is performing a vital role in building the talent pipeline for alignment research. I am a bit confused about why more organisations don’t offer larger internship programs so that the mentors can run their programs ‘in-house’. My best guess is that MATS is much better than most organisations running small internship programs for the first time, particularly in supporting their fellows holistically (often providing accommodation and putting significant effort into the MATS fellows community). One downside of the program relative to an internship at an organisation is that there are fewer natural routes to enter a managed position, though many fellows have gone on to receive LTFF grants for independent projects or continued their mentorship under the same mentor.

• Conjecture applied for funding to host a two day AI safety workshop in Japan in collaboration with Araya (a Japanese AI company). They planned to invite around 40 people, with half of the attendees being AI researchers, and half being alignments researchers from the US and UK. Japanese researchers were generally senior, leading labs, holding postdoc positions in academia, or holding senior technical positions at tech companies.
• To my knowledge, there has been very little AI safety outreach conducted amongst strong academic communities in Asia (e.g. in Japan, Singapore, South Korea …). On the current margin, I am excited about more outreach being done in these countries within ultra-high talent groups. The theory of change for the grant seemed fairly straightforward: encourage talented researchers who are currently working in some area of AI to work on AI safety, and foster collaborations between them and the existing alignment community.
• Conjecture shared the invite list with me ahead of the event, and I felt good about the set of alignment researchers invited from the UK and US. I looked into the Japanese researchers briefly, but I found it harder to gauge the quality of invites given my lack of familiarity with the Japanese AI scene. I also trust Conjecture to execute operationally competently on events of this type, having assisted other AI safety organisations (such as SERI MATS) in the past.
• On the other hand, I have had some concerns about Conjecture, and I felt confused about whether this conference gave Conjecture more influence in ways that I would feel concerned about given the questionable integrity and judgement of their CEO, - see this and this section of a critique of their organisation (though note that I don’t necessarily endorse the rest of the post). It was also unclear to me how counterfactual the grant was, and how this traded off against activities that I would be less excited to see Conjecture run. I think this is a general issue with funding projects at organisations with flexible funding, as organisations are incentivised to present their most fundable projects (which they are also the most excited about), and then in cases where the funding request is successful, move funding that they would have spent on this projects to other lower impact projects. Overall, I modelled making this grant as being about a quarter as cost-effective as it might have been without these considerations (though I don’t claim this discount factor to be particularly reliable).
• Overall, I thought this grant was pretty interesting, and I think that the ex-ante case for it was pretty solid. I haven’t reviewed the outcomes of this grant yet, but I look forward to reviewing and potentially making more grants in this area.
• Update: Conjecture kindly directed me towards this retrospective and have informed me that some Japanese attendees of their conference are thinking of creating an alignment org.