Joseph Bloom

Main points in favor of this grant

• Neel Nanda's top choice in the Nonlinear Network. Neel says many people want to hire him.
• Joseph is an official maintainer of TransformerLens (the top package for mech interp).
• Teaches at the ARENA program.
• Two really good posts on Decision Transformer Interpretability and Mech Interp Analysis of GridWorld Agent-Simulator.
• Work was listed by Anthropic May 2023 update
• Working on trajectory transfers is a natural progression from decision transformers

Donor's main reservations

I wonder if he is best to be hired by some other alignment team instead since I think he might be young as he might have better mentorship with others.

Process for deciding amount

This just should be fully funded, at least to $110,000. $25000 (but ideally $50000 would put him at ease for 6 months by which time he expects to have enough output to justify further funding. I'd give more but I have a limited budget. This is already half of my budget but I feel quite strongly about this.

Conflicts of interest

Nothing to disclose.

---

Joseph Bloom - Independent AI Safety Research, Jul 2023

I think with his track record so far and endorsements, he's earned the right to go the direction he thinks is best. Maybe it'd be better to have an org that "houses" a bunch of people that just want to work by themselves and the org just formally employs them and helps them raise funds for their project and maybe has some communal resources but I don't think I'd prefer to fund that org vs. fund someone who is just going to do good direct work.

---

Joseph Bloom - Independent AI Safety Research, Dec 2023

Want to update that Joseph has been crushing it. He's made good research progress, updates Dylan and I every month and receives good feedback. Super happy with this grant.

I have also let Joseph know that if he is in need for more funding, he should contact me and I will make sure it happens.

10/10

Joseph applied for independent research funding to continue his research into decision transformer interpretability. I'm happy about Joseph's initial result, which found circuits in a decision transformer in a simple RL environment. I thought the applicant's write up was solid and gave me some updates on what cognitive machinery I expect to be induced by RL. In particular, I was excited about the preference directions in embedding space that they constructed. This seems like a useful initial step for retargeting the search, though more understanding of the circuits that are doing the optimization seems critical for this approach.

I think interpretability on RL models is pretty neglected and very relevant for safety.

According to a reference, the applicant was also in the top 3 ARENA participants, and was very motivated and agentic.

The counterfactual is that Joseph tries to get funding elsewhere, and if that fails, getting a research engineer job at an AI safety org (e.g. Redwood, Conjecture, Ought, etc). I encouraged this person to apply to the AI safety orgs, as I think that working at an org is generally more productive than independent research. These jobs are quite competitive, so it's likely that Joseph won't get hired by any of them, and in this case, it seems great to pay him to do independent alignment research.

Overall, I think that Joseph is a promising researcher, and is working on a useful direction, so I feel excited about supporting this.

Since receiving this grant, Joseph has received some more funding (here), and was mentioned in the Anthropic May Update.