Timaeus

Timaeus focuses on interpretability work and sharing their results. The set of advisors is excellent, including Davidad and Evan Hubinger. Evan, John Wentworth and Vanessa Kosoy have offered high praise, and there is evidence they have impacted top lab research agendas. They’re done what I think is solid work, although I am not so great at evaluating papers directly.

If you’re interested in directly funding interpretability research, that all makes this seem like a slam dunk. I’ve confirmed that this all continues to hold true in 2025\.

To donate, get in touch with Jesse at jesse@timaeus.co. If this is the sort of work that you’re interested in doing, they also have a discord at http://devinterp.com/discord.

I've generally been impressed by how well Timaeus have executed. They've in short order assembled a strong team who are collaborating & working well together, producing substantial research and outreach outputs. They have a distinctive research vision, and I think deserve some credit for popularizing studying the evolution of networks throughout training from an interpretability perspective with e.g. EleutherAI's interpretability team now pursuing their own "development interpretability" flavored research.

I have not had time to conduct an in-depth evaluation of Timaeus's research, so would not want people to defer to me on this point. In particular I don't currently have a good enough understanding of Singular Learning Theory to judge the tractability of this approach. I do have some background skepticism of this category of approaches (having seen a number of theoretically nice approximations like neural tangent kernel not actually have that much explanatory power) that would make me have a higher burden of evidence to buy-into this approach, however if it is a dead-end I would expect Timaeus to eventually recognize that & be able to pivot to ohter valuable directions.

---

Next Steps in Developmental Interpretability, Aug 2024

@AdamGleave Full-disclosure, I have two small COIs here:

• I'm an informal advisor of Timaeus, which consists of me having some advising meetings with team members on a ~once/month basis, and I'm e.g. listed on their website. However, I do not have any hard power: explicitly I am not a board member, and no one at Timaeus reports to me.
• Jesse, the Timaeus founder, works from my organizations co-working space FAR.Labs. If Timaeus expanded and made in-person hires in the Bay it is likely they would also work from FAR.Labs. So FAR.AI might get a small amount of revenue from Timaeus expansion.

**Main points in favor of this grant**

• I think there is merit in developing a breadth of interpretability approaches. If Singular Learning Theory ends up having merit, there is a wealth of knowledge from physics and chemistry that directly applies.
• The bang for your buck with researchers in the project is really good compared to most.
• I am excited about bringing a talented researcher in Daniel Murfet into the field.
• I am excited about this project being a launching ground for a potential future for profit or non profit if this goes well
• I think Jesse is great at explaining his ideas and communicating his reasoning. I think Stan is very smart and worth supporting.

**Donor's main reservations**

• I don’t know about the merits of SLT and I’m not equipped to judge.
• I don’t know the direction they will take things if they are successful and it doesn’t seem like it was yet thought through.

**Process for deciding amount**

I need to save some money for 2 more grants I would like to make but I want to show significant support for this project and gets them across the $125k threshold.

**Conflicts of interest**

I might potentially hire Stan as a consultant for my business. Funding this probably works against me but I felt this was important to disclose.

---

Scoping Developmental Interpretability, Dec 2023

If I am not mistaken, Jesse + team have received additional funding from SFF now of $500k to expand. This puts it off the table for me for further Manifund grants for now but will check in with Jesse/Stan sometime in the new year for where they are at.

**Main points in favor of this grant**

• Developmental interpretability seems like a potentially promising and relatively underexplored research direction for exploring neural network generalization and inductive biases. Hopefully, this research can complement low-level or probe-based approaches for neural network interpretability and eventually help predict, explain, and steer dangerous AI capabilities such as learned optimization and deceptive alignment.
• Jesse made a strong, positive impression on me as a scholar in the SERI MATS Winter 2022-23 Cohort; his research was impressive and he engaged well with criticism and others scholars' diverse research projects. His mentor, Evan Hubinger, endorsed his research at the time and obviously continues to do, as indicated by his recent regrant. While Jesse is relatively young to steer a research team, he has strong endorsements and support from Dan Murfet, David Krueger, Evan Hubinger, and other researchers, and has displayed impressive enterpeneurship in launching Timaeus and organizing the SLT summits.
• I recently met Dan Murfet at EAGxAustralia 2023 and was impressed by his research presentation skills, engagement with AI safety, and determination to build the first dedicated academic AI safety lab in Australia. Dan seems like a great research lead for the University of Melbourne lab, where much of this research will be based.
• Australia has produced many top ML and AI safety researchers, but has so far lacked a dedicated AI safety organization to leverage local talent. I believe that we need more AI safety hubs, especially in academic institutions, and I see Timaeus (although remote) and the University of Melbourne as strong contenders.
• Developmental interpretability seems like an ideal research vehicle to leverage underutilized physics and mathematics talent for AI safety. Jesse is a former physicist and Dan is a mathematician who previously specialized in algebraic geometry. In my experience as Co-Director of MATS, I have realized that many former physicists and mathematicians are deeply interested in AI safety, but lack a transitionary route to adapt their skills to the challenge.
• Other funders (e.g., Open Phil, SFF) seem more reluctant (or at least slower) to fund this project than Manifund or Lightspeed and Jesse/Dan told me that they would need more funds within a week if they were going to hire another RA. I believe that this $20k is a high-expected value investment in reducing the stress associated with founding a potentially promising new AI safety organization and will allow Jesse/Dan to produce more exploratory research early to ascertain the value of SLT for AI safety.

**Donor's main reservations**

• I have read several of Jesse's and Dan's posts about SLT and Dev Interp and watched several of their talks, but still feel that I don't entirely grasp the research direction. I could spend further time on this, but I feel more than confident enough to recommend $20k.
• Jesse is relatively young to run a research organization and Dan is relatively new to AI safety research; however, they seem more than capable for my level of risk tolerance with $20k, even with my current $50k pot.
• The University of Melbourne may not be an ideal (or supportive) home for this research team; however, Timaeus already plans to be somewhat remote and several fiscal sponsors (e.g., Rethink Priorities Special Projects, BERI, Ashgro) would likely be willing to support their researchers.

**Process for deciding amount**

I chose to donate $20k because Jesse said that a single paper would cost $40k (roughly 1 RA-year) and my budget is limited. I encourage further regrantors to join me and fund another half-paper!

**Conflicts of interest**

Jesse was a scholar in the program I co-lead, but I do not believe that this constitutes a conflict of interest.

**Main points in favor of this grant**

I think understanding the inductive biases of modern machine learning processes is extremely important both to being able to accurately asses dangers as well as discover good interventions. Most of my uncertainty over the future is currently tied up in uncertainty regarding the inductive biases of machine learning processes (see here for a good explanation of why: https://www.alignmentforum.org/posts/A9NxPTwbw6r6Awuwt/how-likely-is-deceptive-alignment).

On that front, I think Singular Learning Theory is a real contender for a theory that has a chance of effectively explaining and predicting the mechanisms behind machine learning inductive biases. Furthermore, I'm familiar with the work of many of the people involved in this project and I believe them to be quite capable at tackling this problem.

**Donor's main reservations**

Though I have high hopes for Singular Learning Theory, my modal outcome is that it's mostly just wrong and doesn't explain machine learning inductive biases that well. Inductive biases are very complex and most theories like this in the past have failed. Though I think this is a better bet here than most, I don't think I expect it to succeed.

**Process for deciding amount**

I've committed $100k at this time, which I think is a reasonable amount for the team to, hopefully in combination with funding from other sources, get started spinning up on this project.

**Conflicts of interest**

Jesse was a mentee of mine in the SERI MATS program.