Linch Zhang

The Long-Term Future Fund provided a grant of $121,575 to Robert Miles, administered through his fiscal sponsor, Ashgro Inc. The grant includes a $71,000 stipend for Rob and $50,000 for contractors and other expenses to support his work in producing YouTube videos, appearing on podcasts, helping researchers communicate their work, growing an online community to help newcomers get more involved, and building a large online FAQ at aisafety.info.

This grant continues the support we've given Rob in the past; we were his first significant funder and have been consistently happy with his progress. We believe Rob's historical impact with his outreach projects has been surprisingly large. His videos are of very high quality, with high overall production value and message fidelity. My understanding is that many technical researchers report satisfaction with the way their ideas are presented. Additionally, his videos are popular, with a typical Rob Miles video garnering between 100,000 and 200,000 views. While this is significantly lower than top technical YouTube channels, it has gained more traction than almost any other technical AI safety outreach to date.

The grant is also fairly cheap relative to the historical impact. Rob requested a $71,000 stipend, which is substantially lower than both his counterfactual earnings and the pay of other individuals who are competent in both technical AI safety and communications.

However, we think Rob's non-YouTube projects are less successful or impactful than his main YouTube channel. This is hard to definitively assess since much of it is based on private or hard-to-aggregate information, such as Rob privately advising researchers on how to communicate their work. Nonetheless, we (I) think it's unlikely that the non-YouTube work is as valuable as the videos. That said, the primary investigator of the grant believes Rob’s non-YouTube work is still significantly more valuable than our marginal grants and thus worth funding.

Another point against the grant is that Rob's YouTube channel productivity has been rather low, especially recently, with no new videos produced in the last year. However, he recently released a very long and high-quality video that, in addition to being a useful summary about many of the important events in AI (safety) over the last year, also goes into some detail about why he hasn't produced as much content recently.

The Long-Term Future Fund provided a $99,330 grant to Alignment Ecosystem Development (AED), an AI safety field-building nonprofit, through their fiscal sponsor, Ashgro Inc. The grant covered a 1-year stipend for 1.25 full-time equivalents (FTEs) to build and maintain digital infrastructure for the AI safety ecosystem, as well as expenses related to AI safety domains and other assorted costs. AED has built, taken ownership of, closely partnered with, and/or maintained approximately 15 different projects, including the AI Safety Map (aisafety.world), AI Safety Info (aisafety.info), and AI Safety Quest (aisafety.quest), with the overarching objective of growing and improving the AI safety ecosystem. Bryce Robertson, a volunteer, currently works on the ecosystem full-time and would like to transition into a paid role. The grant allocated $66,000 for stipends and the remaining funds for various software and other expenses.

Fund managers had differing opinions on whether this grant represented a good use of marginal funds. The primary investigator strongly supported the grant, while two other fund managers who examined it in detail were not convinced that it met the fund's high funding bar. Habryka, one of the fund managers, expressed excitement for the grant primarily because it continued the work that AED leadership had historically done in the AI safety space. Although not particularly enthusiastic about any individual project in the bundle, Habryka had heard of many people benefiting from AED's support in various ways and held a prior belief that good infrastructure work in this space often involves providing a significant amount of illegible and diffuse support. For example, Habryka noted that plex from AED had been quite useful in moderating and assisting with Rob Miles's Discord, which Habryka considered a valuable piece of community infrastructure, contributing positively to the impact of Rob Miles's videos in informing people about AI safety.

Arguments in favor of the grant included the websites being useful pieces of community infrastructure that can help people orient towards AI safety and the historically surprising popularity of AED's work. However, some fund managers raised concerns that AED might be spreading themselves too thin by working on many different projects rather than focusing on just one. They also questioned the quality of the outputs and worried that if newcomers' first exposure to AI safety was through AED's work, they might become unimpressed with the field as a result.

In accordance with the fund's procedure, the project was put to a vote, and the grant ultimately passed the funding bar. At a more abstract level, the fund's general policy or heuristic has been to lean towards funding when one fund manager is very excited about a grant, and other fund managers are more neutral. The underlying implicit model here is that individual excitement is more likely to identify grants with the potential for significant impact or "hits" in a hits-based giving framework, compared to grant decisions with consensus decisions where everybody is happy about a grant, but no one is actively thrilled. The underlying philosophy is explicated further in an earlier comment I wrote in January 2021, about a year before I joined LTFF.

The Long-Term Future Fund has been supporting Logan Smith for 2 years, providing a stipend of $40,000 every 6 months. Recently, Logan and their team published exciting mechanistic interpretability results where they used Sparse Autoencoders (SAEs) to find highly interpretable directions in language models. The fund has also supported Hoagy Cunningham, the first author of the linked paper, for the same work.

This work was quite similar to research later published by Anthropic, which generated significant excitement in the AI safety community (and was a precursor to the recent excitement of Golden Gate Claude). You may find it helpful to read this post comparing their two papers. I believe some of the independent researchers funded by the Long-Term Future Fund to work on SAEs were subsequently hired by Anthropic to continue their interpretability work at Anthropic.

Although I (Linch) am not an expert in the field, my impression is that the work on sparse autoencoders, both from independent researchers and Anthropic, represents some of the most meaningful advances in AI safety and interpretability in 2023.

Note: When an earlier private version of these notes was circulated, a senior figure in technical AI safety strongly contested my description. They believe that the Anthropic SAE work is much more valuable than the independent SAE work, as both were published around the same time, but the Anthropic work provides sufficient evidence to be worth extending by other researchers, whereas the independent research was not dispositive. I find these arguments plausible but not overwhelmingly convincing. Unfortunately, I lack the technical expertise to be well-equipped to form an accurate independent assessment on this matter.

The Long-Term Future Fund provided a $1,500 grant to Joshua Clymer for compute funds to rent A100 GPUs for experiments on instruction-following generalization. Although Clymer had previously worked on AI safety field-building and communications, this was his first technical AI safety project.

The fund was interested in whether models trained within a specific distribution can generalize "correctly" out of distribution, such as on benchmarks like TruthfulQA, instead of learning the idiosyncrasies of human evaluators. More broadly, the fund wanted to ensure that models can faithfully follow human instructions even when operating in vastly different contexts from their training data. The grantee believed that training models on general instruction-following could be a plausible approach to aligning AI systems with insufficiently specified rewards.

The proposal employed a method called "sandwiching," where the model is trained on data from a non-expert who cannot evaluate the model's performance, but the later evaluation is conducted by an expert. We were excited about the grant as it was cheap and tractable, while addressing an obvious difficulty in AI alignment, making it an attractive funding opportunity.

*Outcomes:* The paper and benchmark have been published, with the authors finding that reward models do not inherently learn to evaluate 'instruction-following' and instead favor personas that resemble internet text. In other words, models trained to follow instructions on easy tasks do not naturally follow instructions on hard tasks, even when we are fairly confident the model has the knowledge to answer the questions correctly. You can also check out Joshua’s thoughts on the project on the alignment forum.

• March 2022 Notes by Linch Zhang: This was my first substantive grant investigation. At the time, I felt shaky about it, but now I feel really good about it. The two main reasons I originally recommended this grant:1. It was an investment into the people who wanted to do EA work – getting 3 ~Google-quality engineers to do more EA/longtermist work (as opposed to counterfactuals that were earning to give or worse) seems well worth it at 200k. 2. It was an investment into the team specifically. Having cohesive software teams seems like an important component for EA becoming formidable in the future, and is somewhat (surprisingly to me) missing in EA, especially outside of AI safety and crypto trading. I heard really good things about Manifold from early users, and they appeared to be developing at a speed that blew other software projects in forecasting (Metaculus, Foretold, Cultivate, Hypermind, etc) out of the water.At the time, it was not an investment into the prediction market itself/theory of change with regards to play-money prediction markets broadly, because the above two factors were sufficient to be decisive.At the time, it was also unclear whether they plan to go the for-profit route or the nonprofit route.They’ve since decided to go the for-profit route.Looking back, still too soon to be sure, but it looks like Manifold is going quite well. Continue to develop features at phenomenal speeds, lots of EAs and others in adjacent communities use the product, team is still producing fast and are excited for the future.From an “investment into team” perspective, I think Manifold now plausibly has the strongest software team in EA outside of AI safety and earning-to-give (not that I’d necessarily have enough visibility to know of all the potentially better teams, especially stealth ones).I have a number of disjunctive ToCs for how Manifold (and forecasting in general) can over time make the future better, some of which is implicitly covered here. Though I am still uncertain about whether this particular project is the best use of the cofounders + team’s time...a lot of the evidence I have to observe this is more an update on the team’s overall skill + cohesiveness rather than an update about their comparative advantage for prediction markets specifically.
• Addendum June 2023:I’ve grown more confused about the total impact or value of this grant. On the one hand, I think Manifold is performing at or moderately above expectations in terms of having a cohesive team that’s executing quickly, and many people in the community appear to find their product useful or at least interesting. On the other hand, the a) zero-interest-rate environment and corresponding high startup evaluations when I recommended this grant has ended in early 2022, and b) recent events have reduced a substantial fraction of EA funding, which meant 200K is arguably much more costly now than a year ago.Still, I think I’m broadly glad to have Manifold in our ecosystem. I think they’re very helpful for people in our and adjacent communities in training epistemics, and I’m excited to see them branch out into experiments in regranting and retroactive funding projects; from a first-principles perspective, it’d be quite surprising if the current status of EA grantmaking is sufficiently close to optimal.

• Solomon Sia wants to talk to a range of advisers, including industry experts, users, and contacts at the CFTC, to see if there are good improvements in ways to regulate prediction markets in the US, while simultaneously protecting users and reducing regulatory risk and friction.
• This was an exploratory grant for seeing how it’s possible to improve the US regulatory environment for prediction markets with a resulting written report provided to EA Funds.
• I think this is a reasonable/great option to explore: I think my position on prediction markets is somewhat more cynical than that of most EAs in the forecasting space, but still, I’m broadly in favor of them and think they can be a critical epistemic intervention, both for uncovering new information and for legibility/common knowledge reasons.It seemed quite plausible to me that the uncertain regulatory environment for prediction markets in the US is impeding the growth of large real-money prediction markets on questions that matter.Solomon seemed unusually competent and knowledgeable about the tech regulations space, a skillset very few EAs have.Cultivating this skillset and having him think about EA issues seemed valuable. A potential new caveat is that in 2023 as AI risk worries heat up, it seems increasingly likely that we might be able to draw from a diverse skillset of experienced and newly interested/worried people.The for-profit motivations for this work are there but not very large, as unless a company is trying very hard to do specific regulatory capture for their company (which is bad and also practically very difficult), easing prediction market regulations has collective benefits and individual costs.(weakly held) I thought trying to nail this during the Biden administration is good because it seemed plausible that the current CFTC will be more predisposed to liking prediction markets than average for the CFTC.One interesting update is that EA connections are likely a mild plus in 2022, and a moderate liability in 2023.NB: Solomon and his collaborator think a) that the EA connection is still a mild to moderate positive b) it’s now unclear whether the Biden administration is better or worse than a counterfactual Republican administration.
• I’ve thought about this grant some afterwards, and I think even with the benefit of hindsight, I'm still a bit confused about how happy I should be about this grant ex-post.One thing is that I’ve grown a bit more confused about the output and tractability of interventions in this domain.The successes(?) Kalshi had confused me and I haven’t had enough time to integrate this into my worldview.My current impression is that CFTC is fairly open to informed opinions from others on this matter.I continue to believe it’s a good grant ex-ante.