AI Safety Camp

By all accounts they are the gold standard for this type of thing. Everyone says they are great, I am generally a fan of the format, I buy that this can punch way above its weight or cost. If I was going to back something in this section, I’d start here.

Donors can reach out to Remmelt at remmelt@aisafety.camp, or leave a matched donation to support next projects.

Running a virtual and physical camp where selected applicants test their fit for AI safety research.

We’ve made multiple grants to the AI Safety Camp in the past. From the April 2019 grant report:

I’ve talked with various participants of past AI Safety camps and heard broadly good things across the board. I also generally have a positive impression of the people involved, though I don’t know any of the organizers very well.

The material and testimonials that I’ve seen so far suggest that the camp successfully points participants towards a technical approach to AI Alignment, focusing on rigorous reasoning and clear explanations, which seems good to me.

I am not really sure whether I’ve observed significant positive outcomes of camps in past years, though this might just be because I am less connected to the European community these days.

I also have a sense that there is a lack of opportunities for people in Europe to productively work on AI Alignment related problems, and so I am particularly interested in investing in infrastructure and events there. This does however make this a higher-risk grant, since I think this means this event and the people surrounding it might become the main location for AI Alignment in Europe, and if the quality of the event and the people surrounding it isn’t high enough, this might cause long-term problems for the AI Alignment community in Europe.

Concerns

• I think organizing long in-person events is hard, and conflict can easily have outsized negative effects. The reviews that I read from past years suggest that interpersonal conflict negatively affected many participants. Learning how to deal with conflict like this is difficult. The organizers seem to have considered this and thought a lot about it, but the most likely way I expect this grant to have large negative consequences is still if there is some kind of conflict at the camp that results in more serious problems.
• I think it’s inevitable that some people won’t get along with organizers or other participants at the camp for cultural reasons. If that happens, I think it’s important for these people to have some other way of getting connected to people working on AI Alignment. I don’t know the best way to arrange this, but I would want the organizers to think about ways to achieve it.

[...]

I would want to engage with the organizers a fair bit more before recommending a renewal of this grant, but I am happy about the project as a space for Europeans to get engaged with alignment ideas and work on them for a week together with other technical and engaged people.

Broadly, the effects of the camp seem very likely to be positive, while the (financial) cost of the camp seems small compared to the expected size of the impact. This makes me relatively confident that this grant is a good bet.

When we next funded them, I said:

This grant is for the AI Safety Camp, to which we made a grant in the last round. Of the grants I recommended this round, I am most uncertain about this one. The primary reason is that I have not received much evidence about the performance of either of the last two camps [1], and I assign at least some probability that the camps are not facilitating very much good work. (This is mostly because I have low expectations for the quality of most work of this kind and haven’t looked closely enough at the camp to override these — not because I have positive evidence that they produce low-quality work.)

My biggest concern is that the camps do not provide a sufficient level of feedback and mentorship for the attendees. When I try to predict how well I’d expect a research retreat like the AI Safety Camp to go, much of the impact hinges on putting attendees into contact with more experienced researchers and having a good mentoring setup. Some of the problems I have with the output from the AI Safety Camp seem like they could be explained by a lack of mentorship.

From the evidence I observe on their website, I see that the attendees of the second camp all produced an artifact of their research (e.g. an academic writeup or code repository). I think this is a very positive sign. That said, it doesn’t look like any alignment researchers have commented on any of this work (this may in part have been because most of it was presented in formats that require a lot of time to engage with, such as GitHub repositories), so I’m not sure the output actually lead to the participants to get any feedback on their research directions, which is one of the most important things for people new to the field.

During this grant round, I spent additional time reviewing and evaluating the AI Safety Camp application, which seemed important given that we are the camp’s most central and reliable funder.

To evaluate the camp, I sent out a follow-up survey to a subset of past participants of the AI Safety camp, asking them some questions about how they benefited from the camp. I also spent some time talking to alumni of the camp who have since done promising work.

Overall, my concern above about mentorship still seems well-placed, and I continue to be concerned about the lack of mentorship infrastructure at the event, which, as far as I can tell, doesn’t seem to have improved very much.

However, some alumni of the camp reported very substantial positive benefits from attending the camp, while none of them reported noticing any substantial harmful consequences. And as far as I can tell, all alumni I reached out to thought that the camp was at worst, only a slightly less valuable use of their time than what they would have done instead, so the downside risk seems relatively limited.

In addition to that, I also came to believe that the need for social events and workshops like this is greater than I previously thought, and that they are in high demand among people new to the AI Alignment field. I think there is enough demand for multiple programs like this one, which reduces the grant’s downside risk, since it means that AI Safety Camp is not substantially crowding out other similar camps. There also don’t seem to be many similar events to AI Safety Camp right now, which suggests that a better camp would not happen naturally, and makes it seem like a bad idea to further reduce the supply by not funding the camp.

A research experience program for prospective AI safety researchers.

https://aisafetycamp.com/

http://bit.ly/2FFFcK1

http://bit.ly/2KKjPLB

This grant is for the AI Safety Camp, to which we made a grant in the last round. Of the grants I recommended this round, I am most uncertain about this one. The primary reason is that I have not received much evidence about the performance of either of the last two camps [1], and I assign at least some probability that the camps are not facilitating very much good work. (This is mostly because I have low expectations for the quality of most work of this kind and haven’t looked closely enough at the camp to override these — not because I have positive evidence that they produce low-quality work.)

My biggest concern is that the camps do not provide a sufficient level of feedback and mentorship for the attendees. When I try to predict how well I’d expect a research retreat like the AI Safety Camp to go, much of the impact hinges on putting attendees into contact with more experienced researchers and having a good mentoring setup. Some of the problems I have with the output from the AI Safety Camp seem like they could be explained by a lack of mentorship.

From the evidence I observe on their website, I see that the attendees of the second camp all produced an artifact of their research (e.g. an academic writeup or code repository). I think this is a very positive sign. That said, it doesn’t look like any alignment researchers have commented on any of this work (this may in part have been because most of it was presented in formats that require a lot of time to engage with, such as GitHub repositories), so I’m not sure the output actually lead to the participants to get any feedback on their research directions, which is one of the most important things for people new to the field.

After some followup discussion with the organizers, I heard about changes to the upcoming camp (the target of this grant) that address some of the above concerns (independent of my feedback). In particular, the camp is being renamed to “AI Safety Research Program”, and is now split into two parts — a topic selection workshop and a research retreat, with experienced AI Alignment researchers attending the workshop. The format change seems likely to be a good idea, and makes me more optimistic about this grant.

I generally think hackathons and retreats for researchers can be very valuable, allowing for focused thinking in a new environment. I think the AI Safety Camp is held at a relatively low cost, in a part of the world (Europe) where there exist few other opportunities for potential new researchers to spend time thinking about these topics, and some promising people have attended. I hope that the camps are going well, but I will not fund another one without spending significantly more time investigating the program.

Footnotes

[1] After signing off on this grant, I found out that, due to overlap between the organizers of the events, some feedback I got about this camp was actually feedback about the Human Aligned AI Summer School, which means that I had even less information than I thought. In April I said I wanted to talk with the organizers before renewing this grant, and I expected to have at least six months between applications from them, but we received another application this round and I ended up not having time for that conversation.