Asya Bergal

We recommended a grant of $23,544 to pay Daniel Filan for his time making 12 additional episodes of the AI X-risk Research Podcast (AXRP), as well as the costs of hosting, editing, and transcription.

The reasoning behind this grant was similar to the reasoning behind my last grant to AXRP:

• I’ve listened or read through several episodes of the podcast; I thought Daniel asked good questions and got researchers to talk about interesting parts of their work. I think having researchers talk about their work informally can provide value not provided by papers (and to a lesser extent, not provided by blog posts). In particular:
• I’ve personally found that talks by researchers can help me understand their research better than reading their academic papers (e.g. Jared Kaplan’s talk about his scaling laws paper). This effect seems to have also held for at least one listener of Daniel’s podcast.
• Informal conversations can expose motivations for the research and relative confidence level in conclusions better than published work.

Daniel also shared some survey data in his grant application about how people rated AXRP compared to other AI alignment resources, though I didn't look at this closely when making the grant decision, as I already had a reasonably strong prior towards funding.

• This was funding a research & networking event for the winners of the Eliciting Latent Knowledge contest run in early 2022; the plan for the event was mainly for it to be participant-led, with participants sharing what they were working on and connecting with others, along with professional alignment researchers visiting to share their own work with participants.
• I think the case for this grant is pretty straightforward: the winners of this contest are (presumably) selected for being unusually likely to be able to contribute to problems in AI alignment, and retreats, especially those involving interactions with professionals in the space, have a strong track record of getting people more involved with this work.

• Kristaps, who was based in Belgium, applied for funding to cover 4 years of tuition, housing, and dining fees at UCSD in the U.S., with the ultimate goal of reducing biological existential risk. Kristaps had worked previously with a long-term-future-focused biosecurity researcher, who gave him a positive reference, and demonstrated reasonable understanding of long-term biorisk considerations in my conversation with him.
• It’s generally very difficult for international students to find support for degrees in the U.S., and Kristaps had indicated in his application that his counterfactual would be to either work for a few years to make money to pay for his degree, or to take out a substantial student loan.
• In general, my guess is that going to a good US or UK university increases future impact in expectation, both by boosting someone’s career, and by putting them in closer proximity for longer with other people working on the long-term future. I think this case is stronger the better the university, the closer the university is to a key geographic hub, and the more students at the university itself are thinking seriously about the long-term future. I made the call to fund here partially by referencing this ranking site, which put UCSD 10th in Biological Sciences degrees worldwide.
• I’m somewhat worried that funding undergraduate degrees is unusually likely to attract applicants who feign interest in a priority cause.

• Noemi applied for funding to design, conduct, analyze, and write up survey research for the Centre for the Governance of AI, which hadn’t yet been set up at the time of the application. From her application:> “Ongoing projects include a large-scale cross-cultural survey of the public’s .AI views (follow-up to Zhang & Dafoe, 2019), the analysis and dissemination of an AI researcher survey (Zwetsloot et al., 2021; Zhang et al., 2021), and a survey of economists’ views of AI/HLMI and related economic forecasts. Future work might include e.g., eliciting expert views and forecasts on AI from a variety of epistemic communities (e.g., policy-makers, AI researchers, AI ethics experts) through surveys and a study of the role of anthropomorphism and mind perception in attitudes towards AI governance.”
• I wanted to make this grant because I was interested in some of the concrete surveys being conducted (particularly the economist survey), and also overall like the model of having someone “specialize” in conducting AI-relevant surveys – ideally, a dedicated survey-runner could become unusually efficient at running surveys, and make it cheap for others to request survey data when it was decision-relevant for them.

Stipends, work hours, and retreat costs for four extra students at CHERI’s summer research program.

This grant was for Effective Altruism Switzerland’s new Swiss Existential Risk Initiative (CHERI) to pay for four extra students in their 2021 summer research program. The students were referred by other summer research programs which had run out of capacity; CHERI had told the students that they could accept them, but that it was unclear whether they would be able to provide a research stipend.

FHI’s summer research program was cancelled this year, so it seems plausible to me that there is not enough summer capacity for students who could turn out to be very strong researchers. CHERI is very new, so I am unsure about the quality of their program, but I think some students can learn and get further involved in longtermism just from being given a somewhat structured environment in which to do research. I also think it would be a bad outcome if CHERI accepted some students but didn’t pay them; I think this could create an unhealthy dynamic for the program cohort.

6-month salary to write book on philosophy + history of longtermist thinking, while longer-term funding is arranged.

Thomas Moynihan asked for 6 months of salary to pay for his time writing a book about why longtermism has become a popular philosophy now, as opposed to earlier in history. Thomas previously wrote a similar book about existential risk, which was featured on the 80,000 Hours podcast.

I was in favor of this grant because I think books can be an effective way to do movement-building and to bring longtermist ideas further into the mainstream. I’m not sure how successful Thomas’s book will be, but I expect it to appeal to educated audiences and contribute to the idea that longtermism is worth taking seriously as a philosophical position.

This was a bridge grant to cover Thomas’s salary while a grant from another funder came through.

Participation in a 2-week summer school on science diplomacy to advance my profile in the field of science policy.

This grant was for Fabio Haenel to pay for a 2-week Science Diplomacy Summer Schoolin Barcelona. Fabio has a Masters in International Relations from the University of Wroclaw and has worked in a variety of jobs promoting science and technology for impact, but he has recently decided to try to pursue a career in policy and emerging technologies, with the goal of reducing risks from technological development and global conflict.

Fabio was familiar with this summer school and thought it would be helpful for building relevant expertise. Jonas Vollmer, the head of EA Funds, has previously met Fabio and had a positive impression of him; since this was a very small grant amount, I decided to defer to Fabio’s judgement on the usefulness of the program.

Funding for research assistance in gathering data on the persistence, expansion, and reversal of laws over 5+ decades.

Zach Freitas-Groff is a PhD student in economics at Stanford University. He asked for funding to hire research assistants on Upwork to help gather data on close referenda in the U.S. as part of a project estimating how long the impact lasts from different types of ballot initiatives. Zach suggested this research could:

a) Boost the profile of longtermist economics research, by showing it’s possible to make empirical progress on a concrete question about how to have longer impact; and b) Provide concretely useful information on how to most effectively influence policy via ballot initiative.

I was most interested in the way this grant could further Zach’s career; references described him as one of the most promising global priorities researchers and economists in the EA movement.

Setting up a nonprofit focused on longtermist AI governance field-building.

This grant was funding the setup of the Centre for the Governance of AI (GovAI) as an independent nonprofit, co-led by Allan Dafoe and Ben Garfinkel. GovAI has historically been a research centre within University of Oxford, but Allan Dafoe has recently taken a senior role at DeepMind which prevented him from heading the centre as part of the university. GovAI is now spinning off into a nonprofit, while the AI governance researchers at FHI are forming an AI governance research group led by Ben Garfinkel.

The application sought $257,000 of funding to cover 3 months of Markus Anderljung’s salary as he sets up the nonprofit, legal and miscellaneous expenses, and salaries for the nonprofit’s first two hires. $80,000 of their budget was covered by another funder, and we covered the rest.

From the application:

The nonprofit will occupy a unique role in the longtermist AI governance space, similar to that which The Forethought Foundation has played in the global priorities space. The organisation will be in a particularly good position to e.g. build and maintain a community of scholars and practitioners interested in longtermist AI governance, to engage with established academics in fields such as political science and economics, to convene conferences, and to provide scholarships and prizes. It may also be in a particularly good position to do grantmaking and house AI governance projects.

Looking back on GovAI’s history, I’d guess that it’s had most of its impact through training longtermist researchers and practitioners who have gone on to do good work at GovAI and elsewhere. GovAI has close ties to academia, which I suspect has allowed them to recruit talented people who would be less excited to work at a non-academically-affiliated institution.

GovAI has also produced several research outputs that I have found informative, including O’Keefe et al.’s report on the Windfall Clause, Ding’s report on Deciphering China’s AI dream along with his newsletter, and Zaidi and Dafoe’s report on lessons from the Baruch Plan for nuclear weapons. Some of their work has also informed Luke Muelhauser’s thinking.

I’m interested in supporting GovAI’s new nonprofit in recruiting more talented young people to work on AI governance and policy, as well as in facilitating additional research work. I think as a nonprofit, GovAI risks losing some of the academic prestige it used to attract people when it was part of the University of Oxford, but I’m hopeful that its existing reputation and connections have a comparable effect, especially given that there are still affiliated AI governance researchers working at FHI.

Researching global security, forecasting, and public communication.

Recusal note: Daniel Eth and Ozzie Gooen did not participate in the voting or final discussion around this grant.

Rethink Priorities is an organization that does public-facing research on a variety of EA topics; see their previous work here. Their previous longtermist work has consisted entirely of Luisa Rodriguez’s work on nuclear war. Luisa has since gone on to do other work, including this post, which I’ve referenced multiple times in this very grant round.

Rethink’s longtermist team is very new and is proposing work on fairly disparate topics, so I think about funding them similarly to how I would think about funding several independent researchers. Their longtermist hires are Linchuan Zhang, David Reinstein, and 50% of Michael Aird (he will be spending the rest of his time as a Research Scholar at FHI). I’m not familiar with David Reinstein. Michael Aird has produced a lot of writing over the past year, some of which I’ve found useful. l haven’t looked at any written work Linchuan Zhang has produced (and I’m not aware of anything major), but he has a good track record in forecasting, I’ve appreciated some of his EA forum comments, and my impression is that several longtermist researchers I know think he’s smart. Evaluating them as independent researchers, I think they’re both new and promising enough that I’m interested in paying for a year of their time to see what they produce.

They are proposing to do research in three different areas:

• Global security (conflict, arms control, avoiding totalitarianism)
• Forecasting (estimating existential risk, epistemic challenges to longtermism)
• Polling / message testing (identifying longtermist policies, figuring out how to talk about longtermism to the public)

Broadly, I am most excited about the third of these, because I think there’s a clear and pressing need for it. I think work in the other two areas could be good, but feels highly dependent on the details (their application only described these broad directions).

Here’s some example work in these areas that I could imagine being interested in. Note that I haven’t spent time looking into these ideas; it’s possible that on further reflection I would no longer endorse them, or discover that the work has already been done:

• Finding high-leverage ways to reduce geopolitical conflict that don’t require political influence (or learning that there aren’t any)
• Estimating the expected badness of a totalitarian future compared to an extinction outcome
• Experimenting with daily-rolling forecasts for early warning

We decided to pay 25% of the budget that Rethink requested, which I guessed was our fair share given Rethink’s other funding opportunities.

Doing the first part of a DPhil at Oxford in modelling viral pandemics.

Toby applied for 7 months of funding for his tuition and living expenses during his DPhil at the Nuffield Department of Medicine at the University of Oxford. Toby’s DPhil offer came too late for him to access other sources of funding for these 7 months; this money meant he wouldn’t have to take out a loan or put off his studies.

Toby’s proposed DPhil topic is to model transmission of novel respiratory viruses within hospitals. I don’t think this is a particularly high-leverage strain of work for preventing global catastrophic biological risks (GCBRs), but I think in biosecurity, it makes sense to prioritize building career capital over having an immediate impact early on.

Funding career building means taking a bet that someone will ultimately go on to do longtermist work. I didn’t know Toby at all, so I had a call with him talking about his motivations, future plans, and the interactions between short-term-focused and long-term-focused work in biosecurity. He gave thoughtful answers to those questions that made me feel like he was likely to pursue longtermist work going forward.

Developing a free online course to prepare students for cutting-edge research on the economics of transformative AI.

Update December 2021: This course is now live [here.](https://www.coursera.org/learn/economics-of-ai)

Anton applied to improve and professionalize a free Coursera course on the economics of transformative AI in time for an initial public run this fall. Anton is an associate professor of economics and had previously created a beta version of the course for his own graduate students at the University of Virginia. He applied for funding to pay for his own time working on the course, the time of a graduate student assistant, professional video editing, guest lecturers, copyright fees, and support from the University of Virginia.

I have spoken to Anton before and read large parts of his and Philip Trammel’s paper, [Economic growth under transformative AI](https://globalprioritiesinstitute.org/wp-content/uploads/Philip-Trammell-and-Anton-Korinek_economic-growth-under-transformative-ai.pdf), which I thought was a good summary of existing economics work on the subject. My sense is that Anton is a competent economist and well-suited to creating an academic course.

I also skimmed the course materials for the beta course and watched parts of the associated video lectures. I would summarize the course as starting in more conventional AI economics covering the automation of labor and economic growth, and ending in more longtermist AI economics covering economic singularities, the macroeconomics of AI agents, and the control problem. The course is fairly technical and aimed towards advanced economics students. I am somewhat familiar with the economics work Anton is drawing from; while I didn’t investigate the course materials in depth, I didn’t see anything obviously inaccurate. The presentation of the beta version of the course seemed rough on certain dimensions (in particular, video quality); I can imagine it benefitting substantially from additional work and editing.

I was interested in funding this course because I think it could introduce smart economists to longtermist work, though much of the course isn’t explicitly focused on the long-term impacts of transformative AI. I think the work of many economists, including Philip Trammell and Robin Hanson, has been really influential in longtermist thinking, and a lot of the AI forecasting work I’m most excited about now is based in economics. It doesn’t seem implausible to me that a course like this could find the “low-hanging fruit” of economics students who turn out to be really interested in longtermist work when exposed to it. (As a secondary path to impact, this course is the best collection of materials around the economics of transformative AI that I know of, and I can imagine using it as a resource in the future.)

One risk of funding work that isn’t strictly longtermist is that it misrepresents longtermist ideas in an attempt to have broader appeal, discouraging people who would be on board with longtermism and attracting people who don’t really understand it. Anton strikes me as someone who is careful in his presentation of concepts, so I think this is unlikely to happen here.

I think there’s a significant chance the course doesn’t have much of an impact, at least in its first run. Anton’s goal is to have 50 students complete the course in the fall; even if that goal is met, it seems likely to me that that’s too small a number to find anyone who ends up being really excited about the material. However, given that the course is likely to have multiple runs and iterations in the future, I think the upside from this grant is worth it.

Technical support for a podcast about research aimed at reducing x-risk from AI.

Recusal note: Adam Gleave and Oliver Habryka did not participate in the voting or final discussion around this grant.

Daniel applied for funding to pay for audio editing, transcription, and recording space usage for his AI alignment podcast, the AI X-risk Research Podcast (AXRP).

I’ve listened or read through several episodes of the podcast; I thought Daniel asked good questions and got researchers to talk about interesting parts of their work. I think having researchers talk about their work informally can provide value not provided by papers (and to a lesser extent, not provided by blog posts). In particular:

• I’ve personally found that talks by researchers can help me understand their research better than reading their academic papers (e.g. Jared Kaplan’s talk about his scaling laws paper). This effect seems to have also held for at least one listener of Daniel’s podcast.
• Informal conversations can expose motivations for the research and relative confidence level in conclusions better than published work.

Daniel provided some statistics about his podcast download numbers over time: 200 – 400 per episode as of early March. This guide suggests that Daniel’s podcast is in the top 10% to 25% of podcasts, though this single metric seems like a pretty dubious measure of podcast performance. (I also think it’s too early to tell exactly how well this podcast will do.)

Overall, the existing download counts along with personal anecdotes from people getting value out of this podcast were enough to justify this grant for me.

Research into physical-world interventions to improve productivity and safeguard life.

Josh Jacobson applied for a grant to research physical-world interventions to improve productivity and safeguard life. He plans to brainstorm and research various interventions, such as improving air quality, preparing for nuclear war or earthquakes, or securing second citizenships. He also plans to help longtermists implement these interventions.

I overall voted in favor of this grant because:

• I agree with the basic premise: Improving the productivity of people working on important issues could be very valuable, and protecting their safety might be useful, too.
• I am aware that some longtermist organizations are currently researching some of these issues themselves. Josh may save these organizations time and circulate the results of his research beyond organizational boundaries.
• Josh's concrete ideas seemed relatively detailed to me, so I thought he was likely to make substantial progress on the project. His project plan and milestones seemed focused on delivering concrete output quickly, which I liked. He also previously worked with BERI on a related project that seemed successful.
• Josh seems very motivated to work on this project, such that I expect him to do a good job at it.

Some reasons against this grant that I considered:

• Overestimating the risks involved might lead to overspending on prevention. The potential benefits seem relatively limited (marginal reduction of a low risk), whereas the costs could grow large (many organizations could potentially spend a lot of time on this).
• For instance, I estimate that an earthquake in San Francisco has a ~2% annual likelihood of occurring, and would kill ~0.03% of the population in expectation. I estimate 30% of such deaths to be preventable. The resulting preventable risk seems tiny (just two micromorts per year). If longtermist organizations started taking measures to avert this risk for their staff as a result of this grant, that could be a poor use of their time.
• Detailed reporting on personal safety risks could lead to unhelpful anxiety about them. During the COVID pandemic, I frequently got the impression that some EAs and rationalists seemed anxious about catching COVID to a degree where their worries seemed to take a much bigger toll on their expected quality of life than the actual risk at hand. I worry that a similar dynamic might come into play here.

Josh applied for a $106,000 grant over a year. Because we don't yet know whether this project will be successful longer-term and the requested grant was relatively large, the LEF funding committee decided to make the grant in two installments: an initial $53,000 grant, and another $53,000 subject to a satisfactory progress report after the first six months. Half of each installment ($26,500) will be covered by the LTFF.

Supporting an AI PhD student's research on the effects of removing non-pharmaceutical interventions for COVID-19.

Mrinank is a first-year PhD student at the Autonomous Intelligent Machines and Systems CDT (AIMS) program at Oxford University and a DPhil affiliate at the Future of Humanity Institute. Mrinank was invited to work with the Imperial College London COVID Response team on machine learning (ML) research estimating the effects of removing and then reimplementing COVID-19 restrictions (e.g., opening schools and businesses, and then closing them again). This grant supports his living expenses while he takes a break from his studies to work on this.

I think this research itself is unlikely to have strong effects on the long-term future, but I think it's a good opportunity to support the career of an up-and-coming ML and AI safety researcher. Mrinank has expressed a strong interest in AI safety (e.g. by running the AI safety group at Oxford), and seems committed to using his ML career to do good from a longtermist perspective. My impression from talking to others is that COVID research is an unusually effective way of building credibility in ML right now, and Mrinank has been extremely successful with related work so far -- his paper on the robustness of effectiveness estimations of non-pharmaceutical interventions was accepted as a Spotlight Talk at NeurIPS 2020, his paper on the effectiveness of non-pharmaceutical interventions is currently under review at a top-tier journal (and has an Altmetric score of 587), and he was invited to present his work at the Africa CDC.

In general, I think the fund should be hesitant to recommend grants that could likely get funding from sources focused on near-term impact. In this case, the opportunity to do this research seemed very short-lived, and I wasn't aware of any such sources that would fund Mrinank on short notice (applications for fastgrants.org are paused).

Researching methods to continuously monitor and analyse artificial agents for the purpose of control.

Lee applied to spend the year before his PhD researching methods for monitoring the internal processes of artificial agents while they're running, which would allow us to detect and respond to undesirable internal behavior. His work will consist of doing a literature review of existing work before trying to make progress on methods that are potentially promising. This research direction has a lot in common with work on the interpretability of neural networks, though it is additionally focused on finding methods that can be used to understand agent internals in real time.

Lee was previously a Masters student at ETH Zurich studying Neural Systems and Computation. During his Masters, Lee worked with Valerio Mante, who was doing work on understanding recurrent neural network models of the brain using an interpretability method called dynamical systems analysis. I'm not sure that dynamical systems analysis in particular is likely to be a scalable solution to real-time monitoring, but I'm interested in supporting Lee doing more work in this area and think he will benefit from his connections to existing academic researchers. Lee also shared some preliminary work he did on a related research direction with the fund; I thought the work was interesting and showed that he was likely to make progress working as an independent reinforcement learning (RL) researcher.

I also shared Lee's proposal with a few AI safety researchers, including people working on interpretability; overall, they thought this was a potentially promising research direction.

Doing a PhD on understanding the analogy between the development of human intelligence and AGI.

This grant was made prior to our standard cycle.

This grant covers the tuition and living expenses for Richard Ngo's 3-year PhD in the philosophy of machine learning at Cambridge University. Richard applied to do this PhD after the standard funding deadline for PhD students had passed, so he was not able to acquire university funding. This grant enabled him to start the PhD now rather than wait another year for university funding to be available. Richard is also applying for funding from other sources, and will return some of this grant if his other applications are successful.

Richard previously worked as a research engineer at DeepMind, and his continued work there would have been primarily engineering-focused. I was excited to enable him to work on AI philosophy full-time instead. I've been extremely impressed with Richard's previous philosophical work -- in particular, his report on AGI safety from first principles is the best articulation I'm aware of about why AGI might pose an existential threat. He also has a strong background in philosophy and AI: a BA in Computer Science and Philosophy from Oxford, and an MPhil in machine learning from Cambridge. His proposed PhD topic is on the analogy between the development of human intelligence and AGI.

From his application:

I think this is one of the most crucial foundational issues in AI safety. Most of our current beliefs about future AI systems (e.g. that they'll be generally intelligent and pursue large-scale goals) are based on loose analogies to humans. However, early arguments for why AI might be dangerous have been critiqued for relying on formalisms like utility maximisation in inapplicable ways (e.g. by Rohin Shah, Eric Drexler, and myself). Instead, I believe that taking a perspective closer to cognitive science and evolutionary analysis will allow us to generate concepts which, while less precise, may be more useful for predicting and shaping AGI behaviour.

I'd classify Hubinger et al.'s paper on mesa-optimisation (https://arxiv.org/abs/1906.01820) as a good example of how we can get more clarity about AGI by drawing on the analogy to humans. I'd also note that the foundations of the field are still in flux - a significant number of safety researchers think that inner alignment is the hardest problem, yet we didn't even have a term for it until very recently. So I expect further work on clarifying the conceptual foundations of the field to have major benefits for the research that's being done elsewhere.

I agree with Richard's justification for this work.

Conducting a computational study on using a light to vibrations mechanism as a targeted antiviral.

We awarded Gavin $30,000 to work on a computational study assessing the feasibility of using a light to vibrations (L2V) mechanism as a targeted antiviral. Light to vibrations is an emerging technique that could destroy viruses by vibrating them at their resonant frequency using tuned pulses of light. In an optimistic scenario, this study would identify a set of viruses that are theoretically susceptible to L2V inactivation. Results would be published in academic journals and would pave the way for further experimental work, prototypes, and eventual commercial production of L2V antiviral equipment. L2V techniques could be generalizable and rapidly adaptable to new pathogens, which would provide an advantage over other techniques used for large-scale control of future viral pandemics.

On this grant, I largely deferred to the expertise of colleagues working in physics and biorisk. My ultimate take after talking to them was that the described approach was plausible and could meaningfully affect the course of future pandemics, although others have also recently started working on L2V approaches.

My impression is that Gavin's academic background is well-suited to doing this work, and I received positive personal feedback on his competence from other EAs working in biorisk.

My main uncertainty recommending this grant was in how the LTFF should compare relatively narrow biorisk interventions with other things we might fund. I ultimately decided that this project was worth funding, but still don't have a good way of thinking about this question.

Surveying experts on AI risk scenarios and working on other projects related to AI safety.

We awarded Alexis $5,000, primarily to support his work on a survey aimed at identifying the arguments and related beliefs motivating top AI safety and governance researchers to work on reducing existential risk from AI.

I think the views of top researchers in the AI risk space have a strong effect on the views and research directions of other effective altruists. But as of now, only a small and potentially unrepresentative set of views exist in written form, and many are stated in imprecise ways. I am hopeful that a widely-taken survey will fill this gap and have a strong positive effect on future research directions.

I thought Alexis's previous work on the principal-agent literature and AI risk was useful and thoughtfully done, and showed that he was able to collaborate with prominent researchers in the space. This collaboration, as well as details of the application, suggested to me that the survey questions would be written with lots of input from existing researchers, and that Alexis was likely to be able to get widespread survey engagement.

Since recommending this grant, I have seen the survey circulated and taken it myself. I thought it was a good survey and am excited to see the results.