Apart Research

I’m (still) confident in their execution of the hackathon idea, which was the central pitch at SFF although they inform me generally they’re more centrally into the MATS-style programs. My doubt for the hackathons is on the level of ‘is AI safety something that benefits from hackathons.’ Is this something one can, as it were, hack together usefully? Are the hackathons doing good counterfactual work? Or is this a way to flood the zone with more variations on the same ideas?

As with many orgs on the list, this one makes sense if and only if you buy the plan, and is one of those ‘I’m not excited but can see it being a good fit for someone else.’

To donate, click here. They can be reached at hello@apartresearch.com.

Building a culture of hands-on experimentation is probably the best way to do AI safety outreach, and Apart seems to have executed on it really well.

I like Apart; I like their results.

The top-of-funnel reference class they're in includes Bluedot, Nontrivial, AISC, aisafety.info. Three of those groups also struggle for institutional funding. Two of those groups are also trying to put out real research as legible output. (MATS isn't top-of-funnel in this sense.)

What do we want from this kind of effort?

• leaders who know what they're talking about technically (check)
• friendly (check)
• otherwise good vibes and epistemics (check)
• actually doing stuff; empirical; e.g. seeking (proxy) feedback for ideas from conference reviews (check)
• throughput (check)
• buy-in from ML or safety or governance or whatnot (mixed?)

I find it difficult to estimate the value of openness and friendliness in field-building but it's not small, and this is not their only selling point. Good luck!

"resulting in three publications accepted at top-tier academic ML venues (NeurIPS, ACL, ICLR),"

To add context in case people get misled by this line, the NeurIPS and ICLR papers (N2G here) were workshop papers, as far as I can tell, not main conference papers. For people not in ML, a conference like NeurIPS or ICLR has both conference papers (one of the highest status ways to publish in ML) and workshop papers (lower prestige and less selective, I'd roughly say a workshop paper is 1/3-1/2 as impressive as a conference paper).

To me, the prior is that most hackathon projects are a total flop and don't go anywhere, so helping someone convert it to a workshop paper is still impressive! (But main conference would have been very impressive). And the ACL paper was a main conference paper, which is impressive!

- TLDR: I’m contributing to this project based on a) my experience running one of their hackathons in April 2023, which I thought was high quality, and b) my excitement to see this model scaled, as I think it has an outsized contribution to the talent search pipeline for AI safety. I’d be interested in seeing see someone with a more technical background evaluating the quality of the outputs, but I don’t think that’s the core of their impact here. - My experience with Apart - I ran an AI governance hackathon as part of my work at Condor Camp, an organization that does AI safety talent search focused on Brazilian university students. - We were experimenting with models different than the camp that could potentially be more cost-effective. We considered doing a hackathon, but the overhead of coming up with good questions, finding speakers/mentors, and overall putting together the intellectual infrastructure would have been too much for us. Apart Research solved that bottleneck. - We ran a hackathon (technically, an ideathon since it was on governance) with ~30 participants. This helped some Condor Camp alumni keep engaged with AI safety beyond the camp, and ultimately “reactivated” one of them who wasn’t working with AI governance. That participant led a team, received an honorary prize, and ended up using the hackathon project to get into a CHERI fellowship. - Why I’m excited about scaling this model - I think Apart’s approach solves a bottleneck for community builders/talent searchers, making a somewhat robust model (hackathon-type competitions) much more easily accessible. - It also reduces the cost of intellectual work for everyone: mentors and evaluators work on the best projects from all over the world, instead of having to repeat the work for every single iteration. - As a result, I expect them to help the community broaden our surface area considerably, finding talent more cost-effectively. - Challenges and concerns - While I’m excited about the talent search and engagement aspect, and the AI governance questions seemed relevant and on point, I’m less sure the quality of the output itself. - Perhaps focusing too much on publishing the output is a suboptimal move, considering it might not be that high quality. On the other hand, they’re getting external feedback from conferences and peer reviewed journals, so this might easily be an upside rather than a concern. Also, it’s a great incentive for participants to have the expectation of seeing their work published. - I’d be interested in takes from folks with more technical expertise about the quality the outputs. So far, my impression is that they at the very least have pretty good evaluators like Neel Nanda (it’d be useful to have evaluators listed on their website, if possible) - I didn’t donate more than 5k because that’s 10% of my pot and I’m expecting to donate to other opportunities where I’ll have more counterfactual impact. But I wanted to chip in here to potentially incentivize others to donate to Apart.