Gavin Leech

Thanks very much to all donors! A private donor has offered to fill the difference so please stop sending me money (mods, if there's a way to close projects I can't see it). We've started work.

---

Shallow review of AI safety 2024, Nov 2024

A donor has sent another $10k, which will partly fund the 2025 edition.

I think collections like this add significant value to newcomers to the field, mostly by being a list of all areas worth maybe thinking about, and key links (rather than eg by providing a lot of takes on which areas are more or less important, unless the author has excellent taste). Gavin has convinced me that the previous post gets enough traffic for it be valuable to be kept up to date.

I'm not super convinced that a ton has changed since 2023, but enough has to be worth at least some updating, so I'm funding the MVP version (I expect this to have more errors than higher funding, but for these to largely be found in the comments, and even higher funding would still have errors). I'd be fine to see others funding it higher though

I'm very impressed with Algon's independence and clarity, and think) that games are still a great place for actual evaluation science.

Sahil, TJ), and Pete have a strong reputation for deep thinking about AI, though on the abstract-to-max-abstract end of the greater AI safety and AI flourishing field. I've been chatting to Pete and others in the cluster for a while and think that the philosophical gap discussed here is actually urgent. I'm hoping (15%?) that along with related work by Grietzer this new subfield soon yields a training method with better alignment properties.

My sole reservation is that the work badly needs a project manager, which is why I've funded a good amount more than the minimum.

Good luck!

CoI: I own Arb, who are providing a little project management / cheerleading pro bono. We won't see any of the money.

Tom wrote the best book there is about existential risk and is a true scholar. He has a publisher lined up for this one already and I'm keen to see where he goes with it.

Had a brief exchange with Yuchen and like the energy (four real papers submitted simultaneously, four more promised). The readme here) is a better explanation than the above. The work is cool and indeed nearly completely neglected. Luck!

I've been following Dan's work for years and think this could go pretty far with agents.

The state of the art in collating followup studies is extremely poor for stupid reasons and so easy to beat.

He's not naive about the technical issues nor the likely reception in academia and has some cool ideas for getting the results into actual discussions.

Quite often when I meet some impressive new person in safety, it turns out they got in / went pro through PIBBSS. I'm particuarly excited to see how their fellows' conceptual work goes during the theory revival.

AFAIK they usually only have <1 year runway despite running a long-term hits-based strat. I'd like to see what they can do with more freedom.

Bonus: They have matching funds with SFF right now.

CoI: I've volunteered on the admissions team in the past.

Petr and his team are actual filmmakers making some of the only AI art I admire. I can corroborate the key claim above about his augmented work being > an order of magnitude cheaper than traditional film production. The 12 month timeframe until _screenings_ is crazy fast but doable if the script comes together soon. I trust Petr to not make it manipulative (appropriate to the theme) and he's planning to consult the 'Gradual Disempowerment' authors to keep it real.

Major distribution is a whole other question, but he has one lead. Deals and lawyer latency is what's be most likely make the ambitious timeline slip.

Another concern is whether others will have taken up the theme in 12 months' time (see e.g.)).

(I'd have funded the project's full minimum but I'm tapped out.)

Conflict of interest: I worked with Petr on the Dwarkesh pod.

I like Apart; I like their results.

The top-of-funnel reference class they're in includes Bluedot, Nontrivial, AISC, aisafety.info. Three of those groups also struggle for institutional funding. Two of those groups are also trying to put out real research as legible output. (MATS isn't top-of-funnel in this sense.)

What do we want from this kind of effort?

• leaders who know what they're talking about technically (check)
• friendly (check)
• otherwise good vibes and epistemics (check)
• actually doing stuff; empirical; e.g. seeking (proxy) feedback for ideas from conference reviews (check)
• throughput (check)
• buy-in from ML or safety or governance or whatnot (mixed?)

I find it difficult to estimate the value of openness and friendliness in field-building but it's not small, and this is not their only selling point. Good luck!

Peli is a frugal philosopher who cowrote one of my favourite essays on alignment.

The idea: He wants to invent virtue post-training, inspired by e.g. the actual normative practice of fields like maths, which often appeals to self-instantiation. It's an open question whether current capabilities allow for instilling stable loops like this and I'm glad someone is trying.

Counterfactual: The current project struck out from the usual funders, I guess because theory is de-emphasised now, or because virtue ethics is usually unoperationalisable, or because he's not good at sales, or because they don't know the following about him.

Track record: Besides the (great) above essay: I've worked with Peli on two technical ML research papers and was impressed with his experimental skill, design skill, and precision. He also did some invited technical replication work on Turner 2023. He's competent at ML experiments at the level of a decent PhD student in the field. He is very used to working totally independently from inception to .

Concerns: If I didn't know the above, the doc would worry me in how totally non-ML-technical it is (it's intentionally doing some original philosophical work as prerequisite for that part). He's also been looking for collaborators and not finding much success; I hope that giving him independent funding makes this search an easier pitch.

I expect distribution to be the weakest part of the project. The inferential distance might be too high for the narrower part of the ingroup audience to bridge to him, even with him presenting them good data. But places like PAW, AIES and HAAISS will certainly engage. He's capable of doing the conference paper grind but doesn't seem motivated by it. Maybe a collaborator could bring the will to actively disseminate the results. But if distribution is the worst risk for a speculative ambitious project then we're in a good spot.

Cost-effectiveness: Very high, $40k / FTE for taking an intriguing idea and bringing it to testability.

Conflict of interest: As noted, Peli has worked on several projects at my company Arb and I've known him for years on Twitter.

Thanks very much to all donors! A private donor has offered to fill the difference so please stop sending me money (mods, if there's a way to close projects I can't see it). We've started work.

---

Shallow review of AI safety 2024, Nov 2024

A donor has sent another $10k, which will partly fund the 2025 edition.

The predecessor was my most important project last year. I've personally verified that there's a great deal of demand for some version of this ("customer" orgs and institutions and "supplier" volunteers). Nuno has some rare and essential qualities (honesty, clarity, infovorism) while lacking some others. But the shoestring version still excites me and I vote with my feet.