Tegan McCaslin

Continuing research projects in AI forecasting, AI strategy and forecasting.

Tegan applied for funding to continue independent research relevant for AI strategy and AI timelines. The LTFF has twice before funded Tegan to perform independent research. I supported this grant primarily due to Tegan’s research output from these previous two grants, and despite some skepticism about an independent-research environment being the best option for Tegan.

Tegan’s main outputs from her previous grants are a report comparing primate and bird brain architectures in terms of cognition, and a draft of a report that both compares the efficiency of biological evolution vs. deep learning in terms of improving cognitive abilities and sheds light on whether there is a common ordering between biological evolution and deep learning in terms of which cognitive tasks are more difficult to learn. I had previously read the former report, and I skimmed the latter for this evaluation.

My overall evaluation of these two outputs is as follows. These research questions are very ambitious (in a good way), and would be valuable to learn about. I think these questions are also in areas that are unlikely to see significant work from academia; both projects require making somewhat fuzzy or subjective judgments with very limited data and generalizing from there, and my impression is that many academics would dismiss such work as too speculative or unscientific. Additionally, the interdisciplinary nature of this work means there are fewer individuals in academia who would be well-positioned to tackle such research. Furthermore, work along these lines is neglected within the EA community. Despite the fact that biological evolution is the only process we know of that has produced human-level intelligence, there has been very little work within EA to study evolution in hopes of gaining insights about transformative AI. That Tegan’s research focuses on questions that are important and neglected makes me more optimistic about the value of the sorts of projects she will pursue, as well as her research judgement.

Regarding research methodology, my sense is Tegan is going about this in a mostly reasonable way. My inside view is that she’s handling the fuzzy, subjective nature of this research well, her evidence largely supports her conclusions, and she appreciates the limitations of her research. The biggest weakness of both projects, in my mind, is that they suffer from very few data points. Admittedly, both projects are in areas where not much data seems to exist (and that which does exist is fuzzy and may be difficult to use). But the fact remains that with few data points, strong conclusions cannot be drawn, and the research itself ends up less informative than it would be with more data. While finding such data may be difficult, I have a reasonably strong prior that at least some further relevant data exists and that the benefits to the research from finding such data would outweigh the search costs.

In light of all this, I would like to see more research along the lines that Tegan has performed previously, and I am in favor of her continuing to pursue such research. Having said that, I think there are some potential downsides to pursuing long-term independent research. Since this will be the third grant that the LTFF will give to Tegan for independent research, I think it’s worth considering these potential downsides, as well as potential mitigation techniques. (I also think this is a valuable exercise for other long-term independent researchers, which is part of my motivation for spelling out these considerations below.)

The largest pitfall in my mind for long-term independent researchers is one’s research becoming detached from the actual concerns of a field and thereby producing negligible value. Tegan seems to have avoided this pitfall so far, thanks to her research judgment and understanding of the relevant areas, and I see no evidence that she’s headed towards it in the future.

Another potential pitfall of independent research is a general lack of feedback loops, both for specific research projects and for the individual’s research skills. One way that independent researchers may be able to produce stronger feedback loops for their work is by sharing more intermediate work. While Tegan has shared (and received feedback from) some senior longtermist researchers on some of her intermediate work, I think she would probably benefit from sharing intermediate work more broadly, such as on the EA forum.

Finally, independent research can struggle to get as much traction as other work (keeping quality constant), as it’s less likely to be connected to organizations or networks where it will naturally be passed around. My sense is that Tegan’s research hasn’t gotten as much attention as it “deserves” given its level of quality, and that many who would find value in the research aren’t aware of it. Fixing such a dynamic generally requires a more active promotion strategy from the researcher. Again, I think posting more intermediate work could help here, as it would create more instances where others see the work, learn about what the researcher is working on, and perhaps even offer feedback.

Conducting independent research into AI forecasting and strategy questions.

This is in significant part a continuation of our previous grant to Tegan for research into AI forecasting and strategy questions. Since then, Tegan has worked with other researchers I trust, and has received sufficient positive testimonials to make me comfortable with this grant. She also sent us some early drafts of research on comparing evolutionary optimization processes with current deep learning systems, which she is planning to publish soon, and which I think is promising enough to be worth funding. She also sent us some early draft work on long-term technological forecasting (10+ years into the future) that I also thought was promising.

Conducting independent research into AI forecasting and strategy questions

From the application:

1) I’d like to independently pursue research projects relevant to AI forecasting and strategy, including (but not necessarily limited to) some of the following:

• Does the trajectory of AI capability development match that of biological evolution?
• How tractable is long-term forecasting?
• How much compute did evolution use to produce intelligence?
• Benchmarking AI capabilities against insects
• Collect and summarize the views of people other than Paul, Dario and Jacob on timelines
• Short doc with more detail on the first two projects here: [https://docs.google.com/document/d/1hTLrLXewF-_iJiefyZPF6L677bLrUTo2ziy6BQbxqjs/edit?usp=sharing](https://docs.google.com/document/d/1hTLrLXewF-_iJiefyZPF6L677bLrUTo2ziy6BQbxqjs/edit?usp=sharing)

I am actively pursuing opportunities to work with or under more senior AI strategy researchers [..], so my research focus within AI strategy is likely to be influenced by who exactly I end up working with. Otherwise I expect to spend some short period of time at the start generating more research ideas and conducting pilot tests on the order of several hours into their tractability, then choosing which to pursue based on an importance/tractability/neglectedness framework.

[..]

2) There are relatively few researchers dedicated full-time to investigating AI strategy questions that are not immediately policy-relevant. However, there nonetheless exists room to contribute to the research on existential risks from AI with approaches that fit into neither technical AI safety nor AI policy/governance buckets.

My thoughts and reasoning

Tegan has been a member of the X-risk network for several years now, and recently left AI Impacts. She is now looking for work as a researcher. Two considerations made me want to recommend that the LTF Fund make a grant to her.

• It’s easier to relocate someone who has already demonstrated trust and skills than to find someone completely new.
• It’s important to give good researchers runway while they find the right place.

While I haven’t spent the time to look into Tegan’s research in any depth, the small amount I did read looked promising. The methodology of this post is quite exciting, and her work there and on other pieces seems very thorough and detailed.

That said, my brief assessment of Tegan’s work was not the reason why I recommended this grant, and if Tegan asks for a new grant in 6 months to focus on solo research, I will want to spend significantly more time reading her output and talking with her, to understand how these questions were chosen and what precise relation they have to forecasting technological progress in AI.

Overall, I think Tegan is in a good place to find a valuable role in our collective X-risk reduction project, and I’d like her to have the runway to find that role.