I didn't fund this project, but I did recommended it to JueYan Zhang for funding.
**Main points in favor of this grant**
- It seems possible that AI agent swarms (or fully automated firms) will soon be responsible for writing most new production code. The phenomena of emergent misalignment, self-fulfilling misalignment, and "spiralism" seem particularly concerning when AI agents are writing and reviewing most of the content that later agents read and are trained on, with little human oversight. Moltbook, Gastown, and AI Village showcase concerning multi-agent dymamics. I expect that we will see a surge of interest in multi-agent emergent AI risks as frontier AI researchers increasingly use AI teams or swarms to automate AI development.
- Multi-agent AI risks seem quite neglected in the AI safety field at the moment relative to their importance. It's possible that many single agent/single stakeholder alignment techniques might not generalize to successfully aligning an AI agent swarm with user intentions (i.e., "multi-single" alignment). It's possible that the first "strong AGI" will be built largely by an AI agent swarm; therefore, it seems very important to build techniques to steer and evaluate agent swarms operating on a shared codebase. Also, agent swarms might constitute a "collective superintelligence", pose a threat vector for mass-manipulation, or otherwise coordinate in undesirable ways.
- I know the co-founders quite well and have a high opinion of their competence. Jack seems like an excellent researcher and previously worked at Softmax, a prominent multi-agent AI company. Raf has substantial experience with startups and, from what I can tell, seems like an excellent engineer. I've found their preliminary research on the effects of codebase comment ablation/modification on AI agent benchmark performance quite interesting.
**Main reservations**
- I expect that most of the catastrophic risks associated with AI agent swarms will be first observed inside of frontier AI companies. Therefore, for Antimemetic AI's techniques to be maximally impactful, they should be deployed inside frontier AI companies. It's unclear whether this type of deployment is realistic for Antimemetic AI due to codebase security restrictions, unless they are acquired (in which case only frontier AI company would benefit). However, I think these techniques could improve safety for plenty of smaller AI companies and users regardless.
- It's possible that "codebase alignment" is a relatively ineffective way to steer AI agents compared to techniques that target individual agents. I expect that research into novel control mechanisms is definitely warranted, however!
- It's possible that the techniques Antimemetic AI are exploring could be used to more effectively leverage multi-agent AI systems to accelerate AI R&D. I would consider this a downside risk. To mitigate this risk, I recommend Antimemetic AI carefully consider which clients they should sell their technology to and conduct a risk analysis.
- Antimemetic AI is a for-profit company and could be subject to the usual incentive problems and mission dilution. I recommend that the founders consider incorporating as a PBC, raising from mission-aligned VCs, and consider the dual-use potential of their technology.
**Conflicts of interest**
I consider the co-founders as friends. I made this clear to JueYan when recommending the project. I don't believe that this has affected my judgement here.