Oliver Habryka

This is a grant to cover half of Vanessa’s stipend for two years (the other half being paid by MIRI). We also made another grant to Vanessa in Q4 2020 for a similar amount.

My model of the quality of Vanessa’s work is primarily indirect, having engaged relatively little with the central learning-theoretic agenda that Vanessa has worked on. The work is also quite technically dense, and I haven’t found anyone else who could explain the work to me in a relatively straightforward way (though I have heard that Daniel Filan’s AXRP podcast with Vanessa is a better way to get started than previous material, though it hadn’t been published when I was evaluating this grant).

I did receive a decent number of positive references for Vanessa’s work, and I have seen her make contributions to other conversations online that struck me as indicative of a pretty deep understanding of the AI Alignment problem.

If I had to guess at the effects of this kind of work, though I should clarify I am substantially deferring to other people here in a way that makes me not particularly trust my specific predictions, I expect that the primary effect would be that the kind of inquiry Vanessa is pursuing highlights important confusions and mistaken assumptions in how we expect machine intelligence to work, which when resolved, will make researchers better at navigating the very large space of potential alignment approaches. I would broadly put this in the category of “Deconfusion Research”.

Vanessa’s research resulted in various public blog posts, which can be found *here*.

Especially since the collapse of FTX, I am quite interested in further diversifying the set of communities that are working on things I think are important to the future. AstralCodexTen and SlateStarCodex meetups seem among the best candidates for creating additional thriving communities with overlapping, but still substantially different norms.

I do feel currently quite confused about what a good relationship between adjacent communities like this and Effective Altruism-labeled funders like the Long Term Future Fund should be. Many of these meetups do not aim to do as much as good as possible, or have much of an ambitious aim to affect the long term future of humanity, and I think pressures in that direction would likely be more harmful than helpful, by introducing various incentives for deception and potentially preventing healthy local communities from forming by creating a misaligned relationship between the organizers (who are paid by EA institutions to produce as much talent for longtermist priorities) and the members (who are interested in learning cool things about rationality and the world and want to meet other people with similar interests).

Since this is a relatively small grant, I didn’t really resolve this confusion, and mostly decided to just go ahead with this. I also talked a bunch to Skyler about this, and currently think we can figure out a good relationship into the future on how it’s best to distribute funding like this, and I expect to think more about this in the coming weeks.

This grant has been approved but has not been paid out at the time of writing.

We’ve made grants to Alex to pursue AI Alignment research before:

• 2019: Building towards a “Limited Agent Foundations” thesis on mild optimization and corrigibility
• 2020: Understanding when and why proposed AI designs seek power over their environment ($30,000)
• 2021: Alexander Turner - Formalizing the side effect avoidance problem ($30,000)
• 2022: Alexander Turner - 12-month stipend supplement for CHAI research fellowship ($31,500)

We also made another grant in 2023 to a team led by Alex Turner for their post on steering vectors for $115,411 (total includes payment to 5 team members, including, without limitation, travel expenses, office space, and stipends).

This grant is an additional grant to Alex, this time covering his full-time stipend for a year to do more research in AI Alignment.

Only the first one has a public grant write-up, and the reasoning and motivation behind all of these grants is pretty similar, so I will try to explain the reasoning behind all of them here.

As is frequently the case with grants I evaluate in the space of AI Alignment, I disagree on an inside-view level pretty strongly with the direction of the research that Alex has been pursuing for most of his AI Alignment career. Historically I have been, on my inside-view, pretty unexcited about Alex’s work on formalizing power-seekingness, and also feel not that excited about his work on shard theory. Nevertheless, I think these are probably among the best grants the LTFF has made in recent years.

The basic reasoning here is that despite me not feeling that excited about the research directions Alex keeps choosing, within the direction he has chosen, Alex has done quite high-quality work, and also seems to often have interesting and useful contributions in online discussions and private conversations. I also find his work particularly interesting, since I think that within a broad approach I often expected to be fruitless, Alex has produced more interesting insight than I expected. This in itself has made me more interested in further supporting Alex, since someone producing work that shows that I was at least partially wrong about a research direction being not very promising is more important to incentivize than work whose effects I am pretty certain of.

I would like to go into more detail on my models of how Alex’s research has updated me, and why I think it has been high quality, but I sadly don’t have the space or time here to go into that much depth. In-short, the more recent steering vector work seems like the kind of “obvious thing to try that could maybe help” that I would really like to saturate with work happening in the field, and the work on formalizing power-seeking theorems is also the kind of stuff that seems worth having done, though I do pretty deeply regret the overly academic/formal presentation which has somewhat continuously caused people to overinterpret the strength of its results (which Alex also seems to have regretted, and is also a pattern I have frequently observed in academic work that was substantially motivated by trying to “legitimize the field”).

Another aspect of this grant that I expect to have somewhat wide-ranging consequences is the stipend level we set on. Some basic principles that have lead me to suggest this stipend level:

• I have been using the anchor of “industry stipend minus 30%” as a useful heuristic for setting stipend levels for LTFF grants. The goal in that heuristic was to find a relatively objective standard that would allow grantees to think about stipend expectations on their own without requiring a lot of back and forth, while hitting a middle ground in the incentive landscape between salaries being so low that lots of top talent would just go into industry instead of doing impactful work, and avoiding grifter problems with people asking for LTFF grants because they expect they will receive less supervision and can probably get away without a ton of legible progress.
• In general I think self-employed salaries should be ~20-40% higher, to account for additional costs like health insurance, payroll taxes, administration overhead, and other things that an employer often takes care of.

I have been rethinking stipend policies, as I am sure many people in the EA community have been since the collapse of FTX, and I haven’t made up my mind on the right principles here. It does seem like a pretty enormous number of good projects are no longer having the funding to operate at their previous stipend levels, and it’s plausible to me that we should take the hit, lose out on a bunch of talent, and reduce stipend levels to a substantially lower level again to be more capable of handling funding shocks. But I am really uncertain on this, and at least in the space of AI Alignment, I can imagine the recent rise to prominence of AI Risk concerns could potentially alleviate funding shortfalls (or it could increase competition by having more talent flow into the space, which could reduce wages, which would also be great).

See the Stipend Appendix below, “How we set grant and stipend amounts”, for more information on EA Funds’ determination of grant and stipend amounts.

• We’ve given multiple grants to David in the past (example). In this case, David was planning to work with FHI, but FHI was unable to pay him for his time. To enable him to continue doing work on longtermist policy, we offered to cover his salary at ALTER, the new organization he has set up. I did not evaluate this grant in great depth, given that FHI would have been happy to pay for his time otherwise.
• so we offered to cover his salary. I did not evaluate this grant in great depth, given that FHI would have been happy to pay for this.

• Peter Hartree worked at 80,000 Hours for multiple years, and was interested in exploring a broader career shift – to take more time to study and think about core longtermist problem areas.
• He received great references from his colleagues at 80k, and I am generally in favor of people at EA organizations reconsidering their career trajectory once in a while and being financially supported while doing so (especially given that current salaries at most EA organizations make building runway for this kind of reflection hard).
• Note: We recommended this grant to a private funder, rather than funding it through LTFF donations, since “independent study” is sometimes hard to prove public benefit for.

• The core principle of the program is to realistically simulate normal forecasting, but on questions which have already been resolved (backcasting). This creates the possibility of rapid feedback. The answer can be revealed immediately after a backcast is made, whereas forecasts are often made on questions which take months or years to resolve. The fundamental challenge of backcasting is gathering information without gaining an unfair advantage or accidentally stumbling on the answer. This project addresses the challenge in a simple way: by forming teams of two, an information gatherer and a forecaster.
• Since this was a small grant, we didn’t evaluate this grant in a lot of depth. The basic idea seemed reasonable to me, and seemed like it might indeed improve training for people who want to get better at forecasting.

Independent research into the nature of optimization, knowledge, and agency, with relevance to AI alignment.

Alex Flint has been doing independent AI alignment research for about a year, and his work strikes me as among the best that independent researchers have produced. From his application:

My research focuses on the frames being used by the AI and AI safety communities to understand and construct intelligent systems. My goal is to identify the frames that we don’t know we’re using and elucidate them, so that we can choose whether to endorse or discard them. I am most interested in the frames that currently underlie technical work in AI and AI safety. [...] I am seeking funding to continue thinking and writing about the frames that underlie our thinking in technical AI alignment.

AI alignment as a field is in a very early stage, which means that a substantial number of the problems we are trying to solve are very underspecified and often don't have clear formulations. I have found Alex's work to be particularly good at clarifying these problem statements, and teasing out the implicit assumptions made by those who formulated the problems of the field. This strikes me as an important step towards enabling more researchers to gain traction on solving the important problems in AI alignment.

It also seems like a crucial step to ensure that the field at large still maintains a focus on its core problems, instead of substituting the hard problems of AI alignment with easier ones. (This is a serious risk, because easier problems are easier to make progress on and might let researchers accrue prestige more easily — which makes them tempting to work on, even if the results aren’t as useful.)

Some specific posts of Alex that I thought did a good job here:

• AI Risk for Epistemic Minimalists
• Agency in Conway's Game of Life
• Parsing Chris Mingard on Neural Networks
• The ground of optimization
• Our take on CHAI's research agenda in under 1500 words

All of these feel to me like they take some high-level problem formulation and pare it down towards something more essential. Importantly, I think that I sometimes see something important being lost in the process, but that in itself is also often useful, since it helps us identify where the important elements lie.

Overall, I am happy to fund Alex to do more independent alignment research and am looking forward to what he can produce with the funding provided by this grant.

Running a virtual and physical camp where selected applicants test their fit for AI safety research.

We’ve made multiple grants to the AI Safety Camp in the past. From the April 2019 grant report:

I’ve talked with various participants of past AI Safety camps and heard broadly good things across the board. I also generally have a positive impression of the people involved, though I don’t know any of the organizers very well.

The material and testimonials that I’ve seen so far suggest that the camp successfully points participants towards a technical approach to AI Alignment, focusing on rigorous reasoning and clear explanations, which seems good to me.

I am not really sure whether I’ve observed significant positive outcomes of camps in past years, though this might just be because I am less connected to the European community these days.

I also have a sense that there is a lack of opportunities for people in Europe to productively work on AI Alignment related problems, and so I am particularly interested in investing in infrastructure and events there. This does however make this a higher-risk grant, since I think this means this event and the people surrounding it might become the main location for AI Alignment in Europe, and if the quality of the event and the people surrounding it isn’t high enough, this might cause long-term problems for the AI Alignment community in Europe.

Concerns

• I think organizing long in-person events is hard, and conflict can easily have outsized negative effects. The reviews that I read from past years suggest that interpersonal conflict negatively affected many participants. Learning how to deal with conflict like this is difficult. The organizers seem to have considered this and thought a lot about it, but the most likely way I expect this grant to have large negative consequences is still if there is some kind of conflict at the camp that results in more serious problems.
• I think it’s inevitable that some people won’t get along with organizers or other participants at the camp for cultural reasons. If that happens, I think it’s important for these people to have some other way of getting connected to people working on AI Alignment. I don’t know the best way to arrange this, but I would want the organizers to think about ways to achieve it.

[...]

I would want to engage with the organizers a fair bit more before recommending a renewal of this grant, but I am happy about the project as a space for Europeans to get engaged with alignment ideas and work on them for a week together with other technical and engaged people.

Broadly, the effects of the camp seem very likely to be positive, while the (financial) cost of the camp seems small compared to the expected size of the impact. This makes me relatively confident that this grant is a good bet.

When we next funded them, I said:

This grant is for the AI Safety Camp, to which we made a grant in the last round. Of the grants I recommended this round, I am most uncertain about this one. The primary reason is that I have not received much evidence about the performance of either of the last two camps [1], and I assign at least some probability that the camps are not facilitating very much good work. (This is mostly because I have low expectations for the quality of most work of this kind and haven’t looked closely enough at the camp to override these — not because I have positive evidence that they produce low-quality work.)

My biggest concern is that the camps do not provide a sufficient level of feedback and mentorship for the attendees. When I try to predict how well I’d expect a research retreat like the AI Safety Camp to go, much of the impact hinges on putting attendees into contact with more experienced researchers and having a good mentoring setup. Some of the problems I have with the output from the AI Safety Camp seem like they could be explained by a lack of mentorship.

From the evidence I observe on their website, I see that the attendees of the second camp all produced an artifact of their research (e.g. an academic writeup or code repository). I think this is a very positive sign. That said, it doesn’t look like any alignment researchers have commented on any of this work (this may in part have been because most of it was presented in formats that require a lot of time to engage with, such as GitHub repositories), so I’m not sure the output actually lead to the participants to get any feedback on their research directions, which is one of the most important things for people new to the field.

During this grant round, I spent additional time reviewing and evaluating the AI Safety Camp application, which seemed important given that we are the camp’s most central and reliable funder.

To evaluate the camp, I sent out a follow-up survey to a subset of past participants of the AI Safety camp, asking them some questions about how they benefited from the camp. I also spent some time talking to alumni of the camp who have since done promising work.

Overall, my concern above about mentorship still seems well-placed, and I continue to be concerned about the lack of mentorship infrastructure at the event, which, as far as I can tell, doesn’t seem to have improved very much.

However, some alumni of the camp reported very substantial positive benefits from attending the camp, while none of them reported noticing any substantial harmful consequences. And as far as I can tell, all alumni I reached out to thought that the camp was at worst, only a slightly less valuable use of their time than what they would have done instead, so the downside risk seems relatively limited.

In addition to that, I also came to believe that the need for social events and workshops like this is greater than I previously thought, and that they are in high demand among people new to the AI Alignment field. I think there is enough demand for multiple programs like this one, which reduces the grant’s downside risk, since it means that AI Safety Camp is not substantially crowding out other similar camps. There also don’t seem to be many similar events to AI Safety Camp right now, which suggests that a better camp would not happen naturally, and makes it seem like a bad idea to further reduce the supply by not funding the camp.

Formalizing the side effect avoidance problem.

Alex is planning to continue and potentially finish formalizing his work on impact measures that he has been working on for the past few years during his PhD. We’ve given two grants to Alex in the past:

• April 2019 – $30,000: Building towards a “Limited Agent Foundations” thesis on mild optimization and corrigibility
• September 2020 – $30,000: Understanding when and why proposed AI designs seek power over their environment.

Since then, Alex has continued to produce research that seems pretty good to me, and has also helped other researchers who seem promising find traction in the field of AI alignment. I’ve also received references from multiple other researchers who have found his work valuable.

Overall, I didn’t investigate this grant in very much additional detail this round, since I had already evaluated his last two applications in much more detail, and it seems the value proposition for this grant is very similar in nature. Here are some of the most relevant quotes from past rounds.

From the April 2019 report:

I’m excited about this because:

• Alex’s approach to finding personal traction in the domain of AI Alignment is one that I would want many other people to follow. On LessWrong, he read and reviewed a large number of math textbooks that are useful for thinking about the alignment problem, and sought public input and feedback on what things to study and read early on in the process.
• He wasn’t intimidated by the complexity of the problem, but started thinking independently about potential solutions to important sub-problems long before he had “comprehensively” studied the mathematical background that is commonly cited as being the foundation of AI Alignment.
• He wrote up his thoughts and hypotheses in a clear way, sought feedback on them early, and ended up making a set of novel contributions to an interesting sub-field of AI Alignment quite quickly (in the form of his work on impact measures, on which he recently collaborated with the DeepMind AI Safety team)

Potential concerns

These intuitions, however, are a bit in conflict with some of the concrete research that Alex has actually produced. My inside views on AI Alignment make me think that work on impact measures is very unlikely to result in much concrete progress on what I perceive to be core AI Alignment problems, and I have talked to a variety of other researchers in the field who share that assessment. I think it’s important that this grant not be viewed as an endorsement of the concrete research direction that Alex is pursuing, but only as an endorsement of the higher-level process that he has been using while doing that research.

As such, I think it was a necessary component of this grant that I have talked to other people in AI Alignment whose judgment I trust, who do seem excited about Alex’s work on impact measures. I think I would not have recommended this grant, or at least this large of a grant amount, without their endorsement. I think in that case I would have been worried about a risk of diverting attention from what I think are more promising approaches to AI Alignment, and a potential dilution of the field by introducing a set of (to me) somewhat dubious philosophical assumptions.

From the September 2020 report:

I've been following Alex's work closely since [the 2019 grant round], and overall have been quite happy with its quality. I still have high-level concerns about his approach, but have over time become more convinced that Alex is aware of some of the philosophical problems that work on impact measures seems to run into, and so am more confident that he will navigate the difficulties of this space correctly. His work also updated me on the tractability of impact-measure approaches, and though I am still skeptical, I am substantially more open to interesting insights coming out of an analysis of that space than I was before. (I think it is generally more valuable to pursue a promising approach that many people are skeptical about, rather than one already known to be good, because the former is much less likely to be replaceable).

I've also continued to get positive feedback from others in the field of AI alignment about Alex's work, and have had multiple conversations with people who thought it made a difference to their thinking on AI alignment.

One other thing that has excited me about Alex's work is his pedagogical approach to his insights. Researchers frequently produce ideas without paying attention to how understandable those ideas are to other people, and enshrine formulations that end up being clunky, unintuitive or unwieldy, as well as explanations that aren't actually very good at explaining. Over time, this poor communication often results in substantial research debt. Alex, on the other hand, has put large amounts of effort into explaining his ideas clearly and in an approachable way, with his "Reframing Impact" sequence on the AI Alignment Forum.

I have not made any substantial updates since September, so the above still summarizes most of my perspective on this.

Building understanding of the structure of risks from AI to inform prioritization.

Recusal note: Daniel Eth and Ozzie Gooen did not participate in the voting or final discussion around this grant.

We made a grant to David Manheim in the August 2019 round. In that grant report, I wrote:

However, I am excited about this grant, because I have a good amount of trust in David’s judgment. To be more specific, he has a track record of identifying important ideas and institutions and then working on/with them. Some concrete examples include:

• Wrote up a paper on Goodhart’s Law with Scott Garrabrant (after seeing Scott’s very terse post on it)
• Works with the biorisk teams at FHI and OpenPhil
• Completed his PhD in public policy and decision theory at the RAND Corporation, which is an unusually innovative institution (e.g. this study);
• Writes interesting comments and blog posts on the internet (e.g. LessWrong)
• Has offered mentoring in his fields of expertise to other people working or preparing to work projects in the x-risk space; I’ve heard positive feedback from his mentees

Another major factor for me is the degree to which David shares his thinking openly and transparently on the internet, and participates in public discourse, so that other people interested in these topics can engage with his ideas. (He’s also a superforecaster, which I think is predictive of broadly good judgment.) If David didn’t have this track record of public discourse, I likely wouldn’t be recommending this grant, and if he suddenly stopped participating, I’d be fairly hesitant to recommend such a grant in the future.

As I said, I’m not excited about the specific project he is proposing, but have trust in his sense of which projects might be good to work on, and I have emphasized to him that I think he should feel comfortable working on the projects he thinks are best. I strongly prefer a world where David has the freedom to work on the projects he judges to be most valuable, compared to the world where he has to take unrelated jobs (e.g. teaching at university).

Since then, I haven’t received any evidence that contradicts this perspective, so I continue to think that it’s valuable for David to be able to work on projects he thinks are valuable, especially if he plans to write up his findings and thoughts publicly.

However, the scope of this project is broader than just David’s personal work, so it seems worthwhile to further explain my take on the broader project. From the application:

This grant will fund a group of researchers who are collaborating on a project working to improve understanding of the structure of risk from AI, with the aim of making results publicly available to AI safety researchers and academia. The project started as a collaboration between researchers from several institutions including FHI and APL, so while several members of the group already have funding, many do not. APL has declined funding for year 2, so we are going to take the project forward with a slightly different focus.

The project goals for year 2 (2020-2021) are first, to refine the model for uncertainties that drive the different risks which emerge from AGI and/or ASI and to write up our preliminary understanding in a series of posts on the Alignment Forum. Second, to continue developing and perform a series of elicitations to understand what AI safety experts think about both the specific uncertainties, and the structure of how these uncertainties interrelate to create risk, and then to publicise those results along with an interactive probabilistic model. In the process we will refine some inaccessible or unclear arguments so that they are suitable for academic treatment, and hopefully improve the quality of communication around AGI and ASI risk scenarios. The people who will receive the funding are being given time to think through and write more about AI safety, and we will continue to encourage them to write papers and LessWrong or Alignment Forum posts about their insights or questions.

I am co-leading the project with Daniel Eth, with assistance from Sammy Martin. The group of collaborators are involved in AI safety research already, but most have other work and/or do not have funding that allows them to focus on AI safety. The individuals are David Manheim (myself), Issa Rice, Ben Cottier, Ross Greutzmacher, Jeremy Perret, Alexis Carlier, and Sammy Martin. Daniel Eth and Aryeh Englander are also working extensively on the project, but they are funded at FHI as a Research Scholar and at APL respectively. We are also consulting with other groups, including AI Impacts and GCRI. The direct near-term impact of the project is expected to be several-fold. We believe, and have been told by others, that the direct impact of the work in clarifying and cataloguing which uncertainties exist and how they are related is of high value, and in general this issue is not being tackled by individual researchers outside our group. Our team members agree that there is a real need for a formalization and an organization of the arguments around AGI and ASI risk.

I’ve observed the work of a good number of the people involved in this project, and the group seems pretty promising to me, though I don’t know all of the people involved.

I do have some uncertainty about the value of this project. In particular, it feels quite high-level and vague to me — which isn’t necessarily bad, but I feel particularly hesitant about relatively unfocused proposals for teams as large as this. My current best guess as to the outcome of this grant is that a number of people who seem like promising researchers have more resources and time available to think in a relatively unconstrained way about AI risk.

Developing and sharing an investigative method to improve traction in pre-theoretic fields.

Logan has previously worked with the Center for Applied Rationality, but is now running an independent project aiming to help researchers in AI alignment and related pre-paradigmatic fields find more traction on philosophically confusing questions. I find work in this space potentially very valuable, but also very high-variance, so I assign high probability to this project not producing very much value. However, I think there’s at least some chance that it helps a substantial number of future AI alignment researchers in a way that few other interventions are capable of.

My general experience of talking to people who are trying to think about the long-term future (and AI alignment in particular) is that they often find it very difficult to productively make progress on almost any of the open problems in the field, with the problems often seeming far too big and ill-defined.

Many well-established research fields seem to have had these problems in their early stages. I think that figuring out how to productively gain traction on these kinds of ill-defined questions is one of the key bottlenecks for thinking about the long-term future. Substantially more so than, for example, getting more people involved in those fields, since my current sense is that marginal researchers currently struggle to find any area to meaningfully contribute and most people in the field have trouble productively integrating others’ contributions into their own thinking.

The basic approach Logan is aiming for seems descendant of some of Eugene Gendlin’s “Focusing)” material, which I know has been quite useful for many people working in AI alignment and related research fields, based on many conversations I’ve had with researchers in the last few years. It seems to frequently come up as the single most useful individual thinking technique, next to the ability to make rough quantitative back-of-the-envelope estimates for many domains relevant to events and phenomena on a global scale. This makes me more optimistic than I would usually be about developing techniques in this space.

The approach itself still seems to be in very early stages, with only the very first post of the planned sequence of posts being available here. The current tentative name for it is “naturalism”:

“Naturalism” (an allusion to 19th century naturalists) is the name I use for a specific way of engaging curiously with the world. It’s a method of inquiry that uses patient observation and original seeing to build models that were previously unthinkable. It takes longer to learn than the spontaneous curiosity of pure exploration, but this specialized method helps you to make deliberate progress in areas where your existing concepts are leading you astray.

Logan is planning to work closely with a small number of interested researchers, which I think is generally the right approach for work like this, and is planning to err on the side of working with people in practical contexts instead of writing long blog posts full of theory. Armchair philosophizing about how thinking is supposed to work can sometimes be useful (particularly when combined with mathematical arguments — a combination which sparked, for example, the Bayesian theory of cognition). But most of the time, it seems to me to produce suggestions that are only hypothetically interesting, but ultimately ungrounded and hard to use for anything in practice. On the margin, investigations that integrate people’s existing curiosities and problems seem better suited to making progress on this topic.

In evaluating this grant, we received a substantial number of highly positive references for Logan, both through their historical work at CFAR, and from people involved in the early stages of their “naturalism” program. I’ve worked a bit with Logan in the context of a few rationality-related workshops and have generally been impressed by their thinking. I’ve also found many of their blog posts quite valuable.

Supporting multiple SPARC project operations during 2021.

This is a relatively small grant to SPARC. I’ve written in the past about SPARC when evaluating our grant to CASPAR:

[...] SPARC or ESPR, two programs with somewhat similar goals that have been supported by other funders in the long-term future space. I currently think interventions in this space are quite valuable, and have been impressed with the impact of SPARC; multiple very promising people in the long-term future space cite it as the key reason they became involved.

SPARC has been funded by Open Philanthropy for the past few years, and they applied for a small supplement to that funding, which seemed worthwhile to me.

Historically, SPARC seems to have been quite successful in causing some of the world’s most promising high-school students (e.g. multiple IMO gold medalists) to develop an interest in the long-term future, and to find traction on related problems. In surveys of top talent within the EA community, SPARC has performed well on measures of what got people involved. Informally, I’ve gotten a sense that SPARC seems to have been a critical factor in the involvement of a number of people I think are quite competent, and also seems to have substantially helped them to become that competent.

I do have some hesitations around the future of SPARC: the pandemic harmed their ability to operate over the past year, and I also have a vague sense that some parts of the program have gotten worse (possibly not surprisingly, given that relatively few of the original founders are still involved). But I haven’t had the time to engage with those concerns in more depth, and am not sure I will find the time to do so, given that this is a relatively small grant.

Creating a mathematical theory of AGI and AGI alignment, based on *the Learning-Theoretic AI Alignment Research Agenda*.

This grant is to support Vanessa for independent research on her learning-theoretic research agenda for AI alignment. Vanessa has been a long-time contributor to AI alignment research, with dozens of posts and hundreds of comments on the AI Alignment Forum.

I sadly haven't had the opportunity to come to a full understanding of Vanessa's research agenda myself, so I primarily relied on external references in order to evaluate this grant. We received strongly positive references from researchers at MIRI and FHI, as well as a few other independent researchers in the field. I wouldn't say that all researchers think that her research is the single most promising open direction we have for AI alignment, but almost all of them agreed that it seems to be an angle worth pursuing, and that Vanessa should have the relevant funding to do more research in this space.

I am familiar with and have benefited from Vanessa's research on quantilization and IRL. She has also written many critiques and comments on the research of other contributors in AI alignment. Examples include this recent comment on Andrew Critch's overview of a number of research areas and their relevance to AI existential risk, these excellent reviews of a number of articles during the 2018 LessWrong Review, and many other critiques and comments.

Overall, of the grants we are making this round, this is the one I am most excited about: While I think that technical work on existential risks from AI is currently one of the most valuable interventions to support with additional funding, I've found it historically very hard to find researchers in this space that I am excited about. My sense is that most people find it very hard to find traction in this space, and to orient on the seemingly insurmountable problems we are facing. Vanessa's research seems like a trailhead that can open more avenues of research, and seems to strike right at the heart of the core problems in technical AI alignment research.

Making Metaculus useful and available to EA and other organizations

I’ve written up rationales for grants to Metaculus in the past, which you can see here.

In my last writeup on Metaculus, I said the following:

My current model is that Metaculus will struggle as a platform without a fully dedicated team or at least individual champion, though I have not done a thorough investigation of the Metaculus team and project, so I am not very confident of this. One of the major motivations for this grant is to ensure that Metaculus has enough resources to hire a potential new champion for the project (who ideally also has programming skills or UI design skills to allow them to directly work on the platform). That said, Metaculus should use the money as best they see fit.

I am also concerned about the overlap of Metaculus with the Good Judgment Project, and currently have a sense that it suffers from being in competition with it, while also having access to substantially fewer resources and people.

The requested grant amount was for $150k, but I am currently not confident enough in this grant to recommend filling the whole amount. If Metaculus finds an individual new champion for the project, I can imagine strongly recommending that it gets fully funded, if the new champion seems competent.

I’ve since thought a lot more about Metaculus, have used the platform more myself, and have broadly been very happy with the progress that the platform has made since I wrote the summary above. As far as I know, Metaculus now has a full-time champion for the project (Tamay Besiroglu), and has demonstrated to me significant advantages over platforms like the Good Judgement Project, in particular with its ability to enter probability distributions over events and a question curation team that is much better at producing forecasts that I care about and seem important to me (and, I would guess, others working on global catastrophic risk).

This overall makes me excited about Metaculus’s future and the effects of this grant.

Understanding when and why proposed AI designs seek power over their environment.

Grant date: January 2020

We previously made a grant to Alex Turner at the beginning of 2019. Here is what I wrote at the time:

My thoughts and reasoning

I'm excited about this because:

Alex's approach to finding personal traction in the domain of AI Alignment is one that I would want many other people to follow. On LessWrong, he read and reviewed a large number of math textbooks that are useful for thinking about the alignment problem, and sought public input and feedback on what things to study and read early on in the process.

He wasn't intimidated by the complexity of the problem, but started thinking independently about potential solutions to important sub-problems long before he had "comprehensively" studied the mathematical background that is commonly cited as being the foundation of AI Alignment.

He wrote up his thoughts and hypotheses in a clear way, sought feedback on them early, and ended up making a set of novel contributions to an interesting sub-field of AI Alignment quite quickly (in the form of his work on impact measures, on which he recently collaborated with the DeepMind AI Safety team)

Potential concerns

These intuitions, however, are a bit in conflict with some of the concrete research that Alex has actually produced. My inside views on AI alignment make me think that work on impact measures is very unlikely to result in much concrete progress on what I perceive to be core AI alignment problems, and I have talked to a variety of other researchers in the field who share that assessment. I think it's important that this grant not be viewed as an endorsement of the concrete research direction that Alex is pursuing, but only as an endorsement of the higher-level process that he has been using while doing that research.

As such, I think it was a necessary component of this grant that I have talked to other people in AI alignment whose judgment I trust, who do seem excited about Alex's work on impact measures. I think I would not have recommended this grant, or at least this large of a grant amount, without their endorsement. I think in that case I would have been worried about a risk of diverting attention from what I think are more promising approaches to AI Alignment, and a potential dilution of the field by introducing a set of (to me) somewhat dubious philosophical assumptions.

Overall, while I try my best to form concrete and detailed models of the AI alignment research space, I don't currently devote enough time to it to build detailed models that I trust enough to put very large weight on my own perspective in this particular case. Instead, I am mostly deferring to other researchers in this space that I do trust, a number of whom have given positive reviews of Alex's work.

In aggregate, I have a sense that the way Alex went about working on AI alignment is a great example for others to follow, I'd like to see him continue, and I am excited about the LTF Fund giving out more grants to others who try to follow a similar path.

I've been following Alex's work closely since then, and overall have been quite happy with its quality. I still have high-level concerns about his approach, but have over time become more convinced that Alex is aware of some of the philosophical problems that work on impact measures seems to run into, and so am more confident that he will navigate the difficulties of this space correctly. His work also updated me on the tractability of impact-measure approaches, and though I am still skeptical, I am substantially more open to interesting insights coming out of an analysis of that space than I was before. (I think it is generally more valuable to pursue a promising approach that many people are skeptical about, rather than one already known to be good, because the former is much less likely to be replaceable).

I've also continued to get positive feedback from others in the field of AI alignment about Alex's work, and have had multiple conversations with people who thought it made a difference to their thinking on AI alignment.

One other thing that has excited me about Alex's work is his pedagogical approach to his insights. Researchers frequently produce ideas without paying attention to how understandable those ideas are to other people, and enshrine formulations that end up being clunky, unintuitive or unwieldy, as well as explanations that aren't actually very good at explaining. Over time, this poor communication often results in substantial research debt. Alex, on the other hand, has put large amounts of effort into explaining his ideas clearly and in an approachable way, with his "Reframing Impact" sequence on the AI Alignment Forum.

This grant would fund living expenses and tuition, helping Alex to continue his current line of research during his graduate program at Oregon State.

Creating quality videos on AI safety, and offering communication and media support to AI safety orgs.

We’ve funded Rob Miles in the past, and since Rob’s work has continued to find traction and maintain a high quality bar, I am viewing this mostly as a grant renewal. Back then, I gave the following rationale for the grant:

The videos on [Rob's] YouTube channel pick up an average of ~20k views. His videos on the official Computerphile channel often pick up more than 100k views, including for topics like logical uncertainty and corrigibility (incidentally, a term Rob came up with).

More things that make me optimistic about Rob’s broad approach:

• He explains that AI alignment is a technical problem. AI safety is not primarily a moral or political position; the biggest chunk of the problem is a matter of computer science. Reaching out to a technical audience to explain that AI safety is a technical problem, and thus directly related to their profession, is a type of ‘outreach’ that I’m very happy to endorse.
• He does not make AI safety a politicized matter. I am very happy that Rob is not needlessly tribalising his content, e.g. by talking about something like “good vs bad ML researchers”. He seems to simply portray it as a set of interesting and important technical problems in the development of AGI.
• His goal is to create interest in these problems from future researchers, and not to simply get as large of an audience as possible. As such, Rob’s explanations don’t optimize for views at the expense of quality explanation. His videos are clearly designed to be engaging, but his explanations are simple and accurate. Rob often interacts with researchers in the community (at places like DeepMind and MIRI) to discuss which concepts are in need of better explanations. I don’t expect Rob to take unilateral action in this domain.

Rob is the first skilled person in the X-risk community working full-time on producing video content. Being the very best we have in this skill area, he is able to help the community in a number of novel ways (for example, he’s already helping existing organizations produce videos about their ideas).

Since then, the average views on his videos appear to have quintupled, usually eclipsing 100k views on YouTube. While I have a lot of uncertainty about what level of engagement those views represent, it would not surprise me if more than 15% of people introduced to the topic of AI alignment in the last year discovered it through Rob’s YouTube channel. This would be a substantial figure, and I also consider Rob’s material one of the best ways to be introduced to the topic (in terms of accurately conveying what the field is about).

In most worlds where I think this grant turns out to be bad, it is because it is currently harmful for the field of AI alignment to grow rapidly, because it might cause the field to become harder to coordinate, cause more bad ideas to become popular, or lead too many people to join who don’t have sufficient background or talent to make strong contributions. I think it is relatively unlikely that we are in that world, and I continue to think that the type of outreach Rob is doing is quite valuable, but I still think there’s at least a 5% probability to it being bad for the AI Alignment field to grow right now.

I trust Rob to think about these considerations and to be careful about how he introduces people to the field; thus, I expect that if we were to end up in a world where this kind of outreach is more harmful than useful, Rob would take appropriate action.

Hiring a research engineer to support CHAI’s technical research projects.

Over the last few years, CHAI has hosted a number of people who I think have contributed at a very high quality level to the AI alignment problem, most prominently Rohin Shah, who has been writing and updating the AI Alignment Newsletter and has also produced a substantial number of other high-quality articles, like this summary of AI alignment progress in 2018-2019.

Rohin is leaving CHAI soon, and I'm unsure about CHAI's future impact, since Rohin made up a large fraction of the impact of CHAI in my mind.

I have read a number of papers and articles from other CHAI grad students, and I think that the overall approach I see most of them taking has substantial value, but I also maintain a relatively high level of skepticism about research that tries to embed itself too closely within the existing ML research paradigm. That paradigm, at least in the past, hasn't really provided any space for what I consider the most valuable safety work (though I think most other members of the Fund don't share my skepticism). I don't think I have the space in this report to fully explain where that skepticism is coming from, so the below should only be seen as a very cursory exploration of my thoughts here.

A concrete example of the problems I have seen (chosen for its simplicity more than its importance) is that, on several occasions, I've spoken to authors who, during the publication and peer-review process, wound up having to remove some of their papers' most important contributions to AI alignment. Often, they also had to add material that seemed likely to confuse readers about the paper's purpose. One concrete class of examples: adding empirical simulations of scenarios whose outcome is trivially predictable, where the specification of the scenario adds a substantial volume of unnecessary complexity to the paper, while distracting from the generality of the overall arguments.

Another concern: Most of the impact that Rohin contributed seemed to be driven more by distillation and field-building work than by novel research. As I have expressed in the past (and elsewhere in this report), I believe distillation and field-building to be particularly neglected and valuable at the margin. I don't currently see the rest of CHAI engaging in that work in the same way.

On the other hand, since it appears that CHAI has probably been quite impactful on Rohin's ability to produce work, I am somewhat optimistic that there are more people whose work is amplified by the existence of CHAI, even if I am less familiar with their work, and I am also reasonably optimistic that CHAI will be able to find other contributors as good as Rohin. I've also found engaging with Andrew Critch's thinking on AI alignment quite valuable, and I am hopeful about more work from Stuart Russell, who obviously has a very strong track record in terms of general research output, though my sense is that marginal funding to CHAI is unlikely to increase Stuart's output in particular (and might in fact decrease it, since managing an organization takes time away from research).

While I evaluated this funding request primarily as unrestricted funding to CHAI, the specific project that CHAI is requesting money for seems also quite reasonable to me. Given the prosaic nature of a lot of CHAI's AI alignment works, it seems quite important for them to be able to run engineering-heavy machine learning projects, for which it makes sense to hire research engineers to assist with the associated programming tasks. The reports we've received from students at CHAI also suggest that past engineer hiring has been valuable and has enabled students at CHAI to do substantially better work.

Having thought more recently about CHAI as an organization and its place in the ecosystem of AI alignment,I am currently uncertain about its long-term impact and where it is going, and I eventually plan to spend more time thinking about the future of CHAI. So I think it's not that unlikely (~20%) that I might change my mind on the level of positive impact I'd expect from future grants like this. However, I think this holds less for the other Fund members who were also in favor of this grant, so I don't think my uncertainty is much evidence about how LTFF will think about future grants to CHAI.

(Recusal note: Due to being a grad student at CHAI, Adam Gleave recused himself from the discussion and voting surrounding this grant.)

Developing a research project on how to infer human's internal mental models from their behaviour.

Sofia Jativa Vega wants to work together with Stuart Armstrong (from the Future of Humanity Institute) on developing methods for AI agents to infer human mental models and use those to predict human preferences. I have many disagreements with Stuart’s agenda, but overall trust his methodology and judgement, and have been following the research he has been posting online for a long time. Stuart was very excited about this opportunity, and Sofia seems to have the relevant background to make progress on these problems (with a PhD in neuroscience).

Exploring crucial considerations for decision-making around information hazards.

Will Bradshaw has been working with Anders Sandberg from the Future of Humanity Institute (FHI) on analysis of good decision-making protocols around information hazards. I trust the judgement of many of the people who have reviewed his work at FHI, and they expressed significant excitement about his work. I also personally think that work on better information hazard protocols is quite valuable, and that there has been relatively little public work on analyzing various sources of infohazards and how to navigate them, making marginal work quite valuable.

After we made this recommendation, Will reached out to us and asked whether it’s possible for him to broaden the grant to also include some immediate crisis response relating to the coronavirus pandemic, in particular trying to make sure that this crisis translates into more long-term work on biorisk. We decided that it would be fine for him to work on either this or his work on infohazards, depending on his judgement and the judgement of his collaborators.

Independent research on agent foundations.

Daniel attended the MIRI Summer Fellows Program in 2017 and 2018, as well as the AI Summer Fellows program in 2018. During those periods, he developed some research directions that multiple researchers I contacted were excited about, and he received positive testimonials from the people he worked with at MIRI.

From his application:

MSFP 2018

here

Subsidized therapy/coaching/mediation for those working on the future of humanity.

We are aware of a significant number of people (including many full-time employees) within the EA and longtermist communities who struggle with depression, anxiety, and other mental health problems. I think it makes sense to provide members of those communities with therapy and coaching sessions, which seem to be relatively effective at helping with those problems (the exact effect sizes are a highly disputed question, but it seems to me that on net, therapy and coaching seem to help a good amount). I also think that a major benefit is that some EAs are unwilling to see therapists who they expect to not understand their values or beliefs; they may be more willing to pursue therapy, and make more progress, with someone familiar with those values and beliefs.

Damon is a licensed therapist who has been offering services to people working in high-impact areas for the past year, and this grant is to allow him to spend a larger fraction of his time over the next year helping people working on high-impact projects, as well as to relocate to California to be able to offer his services to a larger number of people (he is currently located in Florida, where none of his current clients are located).

We’ve received a very large number of overwhelmingly positive testimonials that were sent to us from his current clients via an independent channel (i.e. Damon did not filter those testimonials for positive ones). This was one of the key pieces of evidence that led me to recommend this grant.

Writing fiction to convey EA and rationality-related topics.

This is a continuation of a grant we made last round, so our reasoning remains effectively the same. Miranda sent us some drafts and documents that seem promising enough to be worth further funding, though we think that after this round she should likely seek independent funding, since we hope that her book will be far enough along by then to get more of an outside perspective on the project and potentially get funding from other funders.

Investigation of AI Safety Via Debate and ML training.

From the application:

I think that David Krueger is doing good work in the space of AI alignment, and that funding Joe to work on things David thinks are important seems worth the small amount of requested funding. We recommended this grant mostly on the basis of referrals and testimonials. David has been collaborating with many good people I trust quite a bit over the past few years (FHI, Deepmind, CHAI and 80k), so that’s where a lot of my trust comes from.

A note-taking tool for networked thought, actively used by many EA researchers.

We have previously made a grant to Roam Research. Since then, a large number of researchers and other employees at organizations working in priority areas have started using Roam and seem to have benefited a lot from it. We received a large number of positive testimonials, and I’ve also found the product to be well-designed.

Despite that, our general sense is that Roam should try to attract external funding after this round, and we are not planning to recommend future grants to Roam (mostly due to it being well-suited to seeking more broader funding).

Conducting independent research into AI forecasting and strategy questions.

This is in significant part a continuation of our previous grant to Tegan for research into AI forecasting and strategy questions. Since then, Tegan has worked with other researchers I trust, and has received sufficient positive testimonials to make me comfortable with this grant. She also sent us some early drafts of research on comparing evolutionary optimization processes with current deep learning systems, which she is planning to publish soon, and which I think is promising enough to be worth funding. She also sent us some early draft work on long-term technological forecasting (10+ years into the future) that I also thought was promising.

Telling the story of human progress to the world, and promote progress as a moral imperative.

All of Jason's work is in the domain of Progress Studies. He works on understanding what the primary causes of historical technological progress were, and what the broad effects of different types of technological progress have been. Since I consider most catastrophic risks to be the result of badly controlled emerging technologies, understanding the historical causes of technological progress, and humanity’s track record in controlling those technologies, is an essential part of thinking about global catastrophic risk and the long-term future.

I also consider this grant to be valuable because Jason seems to be a very capable researcher who has attracted the attention of multiple people whose thinking I respect a lot (he also received a grant from Tyler Cowen's Emergent Ventures). I think there is a good chance he could become a highly influential public writer, and having him collaborate with researchers and thinkers working on global catastrophic risks could be very valuable in the long-run.

I also think that his current research is going to be highly and directly relevant in worlds where catastrophic risks are not the primary type of issue that turns out to be important, and where human flourishing through technological progress may be the most important cause area. (This reasoning is similar to that behind last round’s grant toward improving reproducibility in science.)

A summit for the world's leading applied category theorists to engage with human flourishing experts.

David Spivak and Brendan Fong, the two co-founders of the Topos Institute, are applying category theory to various problems in AI alignment and other areas I think are important, and are organizing a conference to facilitate exchange between the category theory community and people currently working on various technical problems around the long-term future.

More recently various researchers I have talked to in AI alignment have found various aspects of category theory quite useful when trying to solve certain technical problems, and David Spivak has a strong track record of academic and educational contributions.

Identifying [white space opportunities](https://hbr.org/2010/02/where-is-your-white-space) for technical projects to improve biosecurity.

From his application:

We also reached out to a variety of researchers we trust in the domain of biosecurity who gave strong positive feedback about Kyle’s project and his skills. He’s also spoken in the past at EA Events about biotech initiatives in clean meat, and has been working as a researcher on clean meat for the last few years, which provides him with a lot of the relevant biotech background to work in this space.

Building a theory of abstraction for embedded agency using real-world systems for a tight feedback loop.

John participated in the recent MIRI Summer Fellows Program, where he proposed some research directions to other MIRI researchers that they were excited about. As well as receiving multiple strong testimonials from various AI alignment researchers, he has also been very actively posting his ideas to the AI Alignment Forum, where he has received substantive engagement and positive comments from several top researchers; this is one of the main reasons for this grant.

Research funding for a year, to enable a transition to AI safety work.

Vojtěch previously did research in mathematics and game theory. He just finished an internship at FHI and is now interested in exploring a full-time career in AI Safety. To do so, he plans to spend a year doing research visits at various organizations and exploring some research directions he is excited about.

According to an FHI researcher we spoke to, Vojtěch seems to have performed well during his time at FHI, so it seemed good to allow him to try transitioning into a full-time AI safety role.

Upskilling in ML in order to be able to do productive AI safety research sooner than otherwise.

Joar was one of the co-authors on the Mesa-Optimisers paper, which I found surprisingly useful and clearly written, especially considering that its authors had relatively little background in alignment research or research in general. I think it is probably the second most important piece of writing on AI alignment that came out in the last 12 months, after the Embedded Agency sequence. My current best guess is that this type of conceptual clarification / deconfusion is the most important type of research in AI alignment, and the type of work I’m most interested in funding. While I don’t know exactly how Joar contributed to the paper, my sense is that all the authors put in a significant effort (bar Scott Garrabrant, who played a supervising role).

This grant is for projects during and in between terms at Oxford. I want to support Joar producing more of this kind of research, which I expect this grant to help with. He’s also been writing further thoughts online (example), which I think has many positive effects (personally and as externalities).

My brief thoughts on the paper (nontechnical):

• The paper introduced me to a lot of of terminology that I’ve continued to use over the past few months (which is not true for most terminology introduced in this space)
• It helped me deconfuse my thinking on a bunch of concrete problems (in particular on the question of whether things like Alpha Go can be dangerous when “scaled up”)
• I’ve seen multiple other researchers and thinkers I respect refer to it positively
• In addition to being published as a paper, it was written up as a series of blogposts in a way that made it a lot more accessible

More of my thoughts on the paper (technical):

Note: If you haven’t read the paper, or you don’t have other background in the subject, this section will likely be unclear. It’s not essential to the case for the grant, but I wanted to share it in case people with the requisite background are interested in more details about the research

I was surprised by how helpful the conceptual work in the paper was - helping me think about where the optimization was happening in a system like AlphaGo Zero improved my understanding of that system and how to connect it to other systems that do optimization in the world. The primary formalism in the paper was clarifying rather than obscuring (and the ratio of insight to formalism was very high - see my addendum below for more thoughts on that).

Once the basic concepts were in place, clarifying different basic tools that would encourage optimization to happen in either the base optimizer or the mesa optimizer (e.g. constraining and expanding space/time offered to the base or mesa optimizers has interesting effects), plus clarifying the types of alignment / pseudo-alignment / internalizing of the base objective, all helped me think about this issue very clearly. It largely used basic technical language I already knew, and put it together in ways that would’ve taken me many months to achieve on my own - a very helpful conceptual piece of work.

Note: Funding from this grant will go to the Leverhulme Centre for the Future of Intelligence, which will fund Jess in turn. The LTF Fund is not replacing funding that CFI would have supplied instead; without this grant, Jess would need to pursue grants from sources outside CFI.

Research on the links between short- and long-term AI policy while skilling up in technical ML.

The main work I know of Jess’s is her early involvement in 80,000 Hours. In the first 1-2 years of their existence, she wrote dozens of articles for them, and contributed to their culture and development. Since then I’ve seen her make positive contributions to a number of projects over the years - she has helped in some form with every EA Global conference I’ve organized (two in 2015 and one in 2016), and she’s continued to write publicly in places like the EA Forum, the EA Handbook, and news sites like Quartz and Vox. This background implies that Jess has had a lot of opportunities for members of the fund to judge her output. My sense is that this is the main reason that the other members of the fund were excited about this grant — they generally trust Jess’s judgment and value her experience (while being more hesitant about CFI’s work).

There are three things I looked into for this grant writeup: Jess’s policy research output, Jess’s blog, and the institutional quality of Leverhulme CFI. The section on Leverhulme CFI became longer than the section on Jess and was mostly unrelated to her work, so I’ve taken it out and included it as an addendum.

Impressions of Policy Papers

First is her policy research. The papers I read were from those linked on her blog. They were:

• The Role and Limits of Principles in AI Ethics: Towards a Focus on Tensions by Jess Whittlestone, Rune Nyrup, Anna Alexandrova and Stephen Cave
• Reducing Malicious Use of Synthetic Media Research: Considerations and Potential Release Practices for Machine Learning by Aviv Ovadya and Jess Whittlestone

On the first paper, about focusing on tensions: the paper said that many “principles of AI ethics” that people publicly talk about in industry, non-profit, government and academia are substantively meaningless, because they don’t come with the sort of concrete advice that actually tells you how to apply them - and specifically, how to trade them off against each other. The part of the paper I found most interesting were four paragraphs pointing to specific tensions between principles of AI ethics. They were:

• Using data to improve the quality and efficiency of services vs. respecting the privacy and autonomy of individuals
• Using algorithms to make decisions and predictions more accurate vs. ensuring fair and equal treatment
• Reaping the benefits of increased personalization in the digital sphere vs. enhancing solidarity and citizenship
• Using automation to make people’s lives more convenient and empowered vs. promoting self-actualization and dignity

My sense is that while there is some good public discussion about AI and policy (e.g. OpenAI’s work on release practices seems quite positive to me), much conversation that brands itself as ‘ethics’ is often not motivated by the desire to ensure this novel technology improves society in accordance with our deepest values, but instead by factors like reputation, PR and politics.

There are many notions, like Peter Thiel’s “At its core, artificial intelligence is a military technology” or the common question “Who should control the AI?” which don’t fully account for the details of how machine learning and artificial intelligence systems work, or the ways in which we need to think about them in very different ways from other technologies; in particular, that we will need to build new concepts and abstractions to talk about them. I think this is also true of most conversations around making AI fair, inclusive, democratic, safe, beneficial, respectful of privacy, etc.; they seldom consider how these values can be grounded in modern ML systems or future AGI systems. My sense is that much of the best conversation around AI is about how to correctly conceptualize it. This is something that (I was surprised to find) Henry Kissinger’s article on AI did well; he spends most of the essay trying to figure out which abstractions to use, as opposed to using already existing ones.

The reason I liked that bit of Jess’s paper is that I felt the paper used mainstream language around AI ethics (in a way that could appeal to a broad audience), but then:

• Correctly pointed out that AI is a sufficiently novel technology that we’re going to have to rethink what these values actually mean, because the technology causes a host of fundamentally novel ways for them to come into tension
• Provided concrete examples of these tensions

In the context of a public conversation that I feel is often substantially motivated by politics and PR rather than truth, seeing someone point clearly at important conceptual problems felt like a breath of fresh air.

That said, given all of the political incentives around public discussion of AI and ethics, I don’t know how papers like this can improve the conversation. For example, companies are worried about losing in the court of Twitter’s public opinion, and also are worried about things like governmental regulation, which are strong forces pushing them to primarily take popular but ineffectual steps to be more "ethical". I’m not saying papers like this can’t improve this situation in principle, only that I don’t personally feel like I have much of a clue about how to do it or how to evaluate whether someone else is doing it well, in advance of their having successfully done it.

Personally, I feel much more able to evaluate the conceptual work of figuring out how to think about AI and its strategic implications (two standout examples are this paper by Bostrom and this LessWrong post by Christiano), rather than work on revising popular views about AI. I’d be excited to see Jess continue with the conceptual side of her work, but if she instead primarily aims to influence public conversation (the other goal of that paper), I personally don’t think I’ll be able to evaluate and recommend grants on that basis.

From the second paper I read sections 3 and 4, which lists many safety and security practices in the fields of biosafety, computer information security, and institutional review boards (IRBs), then outlines variables for analysing release practices in ML. I found it useful, even if it was shallow (i.e. did not go into much depth in the fields it covered). Overall, the paper felt like a fine first step in thinking about this space.

In both papers, I was concerned with the level of inspiration drawn from bioethics, which seems to me to be a terribly broken field (cf. Scott Alexander talking about his IRB nightmare or medicine’s ‘culture of life’). My understanding is that bioethics coordinated a successful power grab (cf. OpenPhil’s writeup) from the field of medicine, creating hundreds of dysfunctional and impractical ethics boards that have formed a highly adversarial relationship with doctors (whose practical involvement with patients often makes them better than ethicists at making tradeoffs between treatment, pain/suffering, and dignity). The formation of an “AI ethics” community that has this sort of adversarial, unhealthy relationship with machine learning researchers would be an incredible catastrophe.

Overall, it seems like Jess is still at the beginning of her research career (she’s only been in this field for ~1.5 years). And while she’s spent a lot of effort on areas that don’t personally excite me, both of her papers include interesting ideas, and I’m curious to see her future work.

Impressions of Other Writing

Jess also writes a blog, and this is one of the main things that makes me excited about this grant. On the topic of AI, she wrote three posts (1, 2, 3), all of which made good points on at least one important issue. I also thought the post on confirmation bias and her PhD was quite thoughtful. It correctly identified a lot of problems with discussions of confirmation bias in psychology, and came to a much more nuanced view of the trade-off between being open-minded versus committing to your plans and beliefs. Overall, the posts show independent thinking written with an intent to actually convey understanding to the reader, and doing a good job of it. They share the vibe I associate with much of Julia Galef’s work - they’re noticing true observations / conceptual clarifications, successfully moving the conversation forward one or two steps, and avoiding political conflict.

I do have some significant concerns with the work above, including the positive portrayal of bioethics and the absence of any criticism toward the AAAI safety conference talks, many of which seem to me to have major flaws.

While I’m not excited about Leverhulme CFI’s work (see the addendum for details), I think it will be good for Jess to have free rein to follow her own research initiatives within CFI. And while she might be able to obtain funding elsewhere, this alternative seems considerably worse, as I expect other funding options would substantially constrain the types of research she’d be able to conduct.

Productivity coaching for effective altruists to increase their impact.

I plan to continue coaching high-impact EAs on productivity. I expect to have 600+ sessions with about 100 clients over the next year, focusing on people working in AI safety and EA orgs. I’ve worked with people at FHI, Open Phil, CEA, MIRI, CHAI, DeepMind, the Forethought Foundation, and ACE, and will probably continue to do so. Half of my current clients (and a third of all clients I’ve worked with) are people at these orgs. I aim to increase my clients’ output by improving prioritization and increasing focused work time.

I would use the funding to: offer a subsidized rate to people at EA orgs (e.g. between $10 and $50 instead of $125 per call), offer free coaching for select coachees referred by 80,000 Hours, and hire contractors to help me create materials to scale coaching.

You can view my impact evaluation (linked below) for how I’m measuring my impact so far.

(Lynette’s public self-evaluation is here.)

I generally think it's pretty hard to do "productivity coaching" as your primary activity, especially when you are young, due to a lack of work experience. This means I have a high bar for it being a good idea that someone should go full-time into the "help other people be more productive” business.

My sense is that Lynette meets that bar, but only barely (to be clear, I consider it to be a high bar). The main thing that she seems to be doing well is being very organized about everything that she is doing, in a way that makes me confident that her work has had a real impact — if not, I think she’d have noticed and moved on to something else.

However, as I say in the CFAR writeup, I have a lot of concerns with primarily optimising for legibility, and Lynette’s work shows some signs of this. She has shared around 60 testimonials on her website (linked here). Of these, not one of them mentioned anything negative, which clearly indicates that I can't straightforwardly interpret those testimonials as positive evidence (since any unbiased sampling process would have resulted in at least some negative datapoints). I much prefer what another applicant did here: they asked people to send us information anonymously, which increased the chance of our hearing opinions that weren’t selected to create a positive impression. As is, I think I actually shouldn't update much on the testimonials, in particular given that none of them go into much detail on how Lynette has helped them, and almost all of them share a similar structure.

Reflecting on the broader picture, I think that Lynette’s mindset reflects how I think many of the best operations staff I’ve seen operate: aim to be productive by using simple output metrics, and by doing things in a mindful, structured way (as opposed to, for example, trying to aim for deep transformative insights more traditionally associated with psychotherapy). There is a deep grounded-ness and practical nature to it. I have a lot of respect for that mindset, and I feel as though it's underrepresented in the current EA/rationality landscape. My inside-view models suggest that you can achieve a bunch of good things by helping people become more productive in this way.

I also think that this mindset comes with a type of pragmatism that I am more concerned about, and often gives rise to what I consider unhealthy adversarial dynamics. As I discussed above, it’s difficult to get information from Lynette’s positive testimonials. My sense is that she might have produced them by directly optimising for “getting a grant” and trying to give me lots of positive information, leading to substantial bias in the selection process. The technique of ‘just optimize for the target’ is valuable in lots of domains, but in this case was quite negative.

That said, framing her coaching as achieving a series of similar results generally moves me closer to thinking about this grant as "coaching as a commodity". Importantly, few people reported very large gains in their productivity; the testimonials instead show a solid stream of small improvements. I think that very few people have access to good coaching, and the high variance in coach quality means that experimenting is often quite expensive and time-consuming. Lynette seems to be able to consistently produce positive effects in the people she is working with, making her services a lot more valuable due to greater certainty around the outcome. (However, I also assign significant probability that the way the evaluation questions were asked reduced the rate at which clients reported either negative or highly positive experiences.)

I think that many productivity coaches fail to achieve Lynette’s level of reliability, which is one of the key things that makes me hopeful about her work here. My guess is that the value-add of coaching is often straightforwardly positive unless you impose significant costs on your clients, and Lynette seems quite good at avoiding that by primarily optimizing for professionalism and reliability.

A research experience program for prospective AI safety researchers.

https://aisafetycamp.com/

http://bit.ly/2FFFcK1

http://bit.ly/2KKjPLB

This grant is for the AI Safety Camp, to which we made a grant in the last round. Of the grants I recommended this round, I am most uncertain about this one. The primary reason is that I have not received much evidence about the performance of either of the last two camps [1], and I assign at least some probability that the camps are not facilitating very much good work. (This is mostly because I have low expectations for the quality of most work of this kind and haven’t looked closely enough at the camp to override these — not because I have positive evidence that they produce low-quality work.)

My biggest concern is that the camps do not provide a sufficient level of feedback and mentorship for the attendees. When I try to predict how well I’d expect a research retreat like the AI Safety Camp to go, much of the impact hinges on putting attendees into contact with more experienced researchers and having a good mentoring setup. Some of the problems I have with the output from the AI Safety Camp seem like they could be explained by a lack of mentorship.

From the evidence I observe on their website, I see that the attendees of the second camp all produced an artifact of their research (e.g. an academic writeup or code repository). I think this is a very positive sign. That said, it doesn’t look like any alignment researchers have commented on any of this work (this may in part have been because most of it was presented in formats that require a lot of time to engage with, such as GitHub repositories), so I’m not sure the output actually lead to the participants to get any feedback on their research directions, which is one of the most important things for people new to the field.

After some followup discussion with the organizers, I heard about changes to the upcoming camp (the target of this grant) that address some of the above concerns (independent of my feedback). In particular, the camp is being renamed to “AI Safety Research Program”, and is now split into two parts — a topic selection workshop and a research retreat, with experienced AI Alignment researchers attending the workshop. The format change seems likely to be a good idea, and makes me more optimistic about this grant.

I generally think hackathons and retreats for researchers can be very valuable, allowing for focused thinking in a new environment. I think the AI Safety Camp is held at a relatively low cost, in a part of the world (Europe) where there exist few other opportunities for potential new researchers to spend time thinking about these topics, and some promising people have attended. I hope that the camps are going well, but I will not fund another one without spending significantly more time investigating the program.

Footnotes

[1] After signing off on this grant, I found out that, due to overlap between the organizers of the events, some feedback I got about this camp was actually feedback about the Human Aligned AI Summer School, which means that I had even less information than I thought. In April I said I wanted to talk with the organizers before renewing this grant, and I expected to have at least six months between applications from them, but we received another application this round and I ended up not having time for that conversation.

Writing EA-themed fiction that addresses X-risk topics.

https://archiveofourown.org/series/936480

https://arxiv.org/pdf/1710.05060v1.pdf

One natural decomposition of whether this grant is a good idea is to first ask whether writing fiction of this type is valuable, then whether Miranda is capable of actually creating that type of fiction, and last whether funding Miranda will make a significant difference in the amount/quality of her fiction.

I think that many people reading this will be surprised or confused about this grant. I feel fairly confident that grants of this type are well worth considering, and I am interested in funding more projects like this in the future, so I’ve tried my best to summarize my reasoning. I do think there are some good arguments for why we should be hesitant to do so (partly summarized by the section below that lists things that I think fiction doesn’t do as well as non-fiction), so while I think that grants like this are quite important, and have the potential to do a significant amount of good, I can imagine changing my mind about this in the future.

The track record of fiction

In a general sense, I think that fiction has a pretty strong track record of both being successful at conveying important ideas, and being a good attractor of talent and other resources. I also think that good fiction is often necessary to establish shared norms and shared language.

Here are some examples of communities and institutions that I think used fiction very centrally in their function. Note that after the first example, I am making no claim that the effect was good, I’m just establishing the magnitude of the potential effect size.

• Harry Potter and the Methods of Rationality (HPMOR) was instrumental in the growth and development of both the EA and Rationality communities. It is very likely the single most important recruitment mechanism for productive AI alignment researchers, and has also drawn many other people to work on the broader aims of the EA and Rationality communities.
• Fiction was a core part of the strategy of the neoliberal movement; fiction writers were among the groups referred to by Hayek as "secondhand dealers in ideas.” An example of someone whose fiction played both a large role in the rise of neoliberalism and in its eventual spread would be Ayn Rand.
• Almost every major religion, culture and nation-state is built on shared myths and stories, usually fictional (though the stories are often held to be true by the groups in question, making this data point a bit more confusing).
• Francis Bacon’s (unfinished) utopian novel “The New Atlantis” is often cited as the primary inspiration for the founding of the Royal Society, which may have been the single institution with the greatest influence on the progress of the scientific revolution.

On a more conceptual level, I think fiction tends to be particularly good at achieving the following aims (compared to non-fiction writing):

• Teaching low-level cognitive patterns by displaying characters that follow those patterns, allowing the reader to learn from very concrete examples set in a fictional world. (Compare Aesop’s Fables to some nonfiction book of moral precepts — it can be much easier to remember good habits when we attach them to characters.)
• Establishing norms, by having stories that display the consequences of not following certain norms, and the rewards of following them in the right way
• Establishing a common language, by not only explaining concepts, but also showing concepts as they are used, and how they are brought up in conversational context
• Establishing common goals, by creating concrete utopian visions of possible futures that motivate people to work towards them together
• Reaching a broader audience, since we naturally find stories more exciting than abstract descriptions of concepts

(I wrote in more detail about how this works for HPMOR in the last grant round.)

In contrast, here are some things that fiction is generally worse at (though a lot of these depend on context; since fiction often contains embedded non-fiction explanations, some of these can be overcome):

• Carefully evaluating ideas, in particular when evaluating them requires empirical data. There is a norm against showing graphs or tables in fiction books, making any explanation that rests on that kind of data difficult to access in fiction.
• Conveying precise technical definitions
• Engaging in dialogue with other writers and researchers
• Dealing with topics in which readers tend to come to better conclusions by mentally distancing themselves from the problem at hand, instead of engaging with concrete visceral examples (I think some ethical topics like the trolley problem qualify here, as well as problems that require mathematical concepts that don’t neatly correspond to easy real-world examples)

Overall, I think current writing about both existential risk, rationality, and effective altruism skews too much towards non-fiction, so I’m excited about experimenting with funding fiction writing.

Miranda’s writing

The second question is whether I trust Miranda to actually be able to write fiction that leverages these opportunities and provides value. This is why I think Miranda can do a good job:

• Her current fiction project is read by a few people whose taste I trust, and many of them describe having developed valuable skills or insights as a result (for example, better skills for crisis management, a better conception of moral philosophy, an improved moral compass, and some insights about decision theory)
• She wrote frequently on LessWrong and her blog for a few years, producing content of consistently high quality that, while not fictional, often displayed some of the same useful properties as fiction writing.
• I’ve seen her execute a large variety of difficult projects outside of her writing, which means I am a lot more optimistic about things like her ability to motivate herself on this project, and excelling in the non-writing aspects of the work (e.g. promoting her fiction to audiences beyond the EA and rationality communities)
• She worked in operations at CEA and received strong reviews from her coworkers
• She helped CFAR run the operations for SPARC in two consecutive years and performed well as a logistics volunteer for 11 of their other workshops
• I’ve seen her organize various events and provide useful help with logistics and general problem-solving on a large number of occasions

My two biggest concerns are:

• Miranda losing motivation to work on this project, because writing fiction with a specific goal requires a significantly different motivation than doing it for personal enjoyment
• The fiction being well-written and engaging, but failing to actually help people better understand the important issues it tries to cover.

I like the fact that this grant is for an exploratory 3 months rather than a longer period of time; this allows Miranda to pivot if it doesn’t work out, rather than being tied to a project that isn’t going well.

The counterfactual value of funding

It would be reasonable to ask whether a grant is really necessary, given that Miranda has produced a huge amount of fiction in the last two years without receiving funding explicitly dedicated to that. I have two thoughts here:

• I generally think that we should avoid declining to pay people just because they’d be willing to do valuable work for free. It seems good to reward people for work even if this doesn’t make much of a difference in the quality/consistency of the work, because I expect this promise of reward to help people build long-term motivation and encourage exploration.
• To explain this a bit more, I think this grant will help other people build motivation towards pursuing similar projects in the future, by setting a precedent for potential funding in this space. For example, I think the possibility of funding (and recognition) was also a motivator for Miranda in starting to work on this project.
• I expect this grant to have a significant effect on Miranda’s productivity, because I think that there is often a qualitative difference between work someone produces in their spare time and work that someone can focus full-time on. In particular, I expect this grant to cause Miranda’s work to improve in the dimensions that she doesn’t naturally find very stimulating, which I expect will include editing, restructuring, and other forms of “polish”.

This grant was recommended by the Fund, but ultimately was funded by a private donor, who (prior to CEA finalizing its standard due diligence checks) had personally offered to make this donation instead. As such, the grant recommendation was withdrawn.

Oliver Habryka had created a full writeup by that point, so it is included below.

Help promising people to reason more effectively and find high-impact work, such as reducing x-risk.

top reported bottleneck

In our last grant round, I took an outside view on CFAR and said that, in terms of output, I felt satisfied with CFAR's achievements in recruitment, training and the establishment of communal epistemic norms. I still feel this way about those areas, and my writeup last round still seems like an accurate summary of my reasons for wanting to grant to CFAR. I also said that most of my uncertainty about CFAR lies in its long-term strategic plans, and I continue to feel relatively confused about my thoughts on that.

I find it difficult to explain my thoughts on CFAR, and I think that a large fraction of this difficulty comes from CFAR being an organization that is intentionally not optimizing towards being easy to understand from the outside, having simple metrics, or more broadly being legible [1]. CFAR is intentionally avoiding being legible to the outside world in many ways. This decision is not obviously wrong, as I think it brings many positives, but I think it is the cause of me feeling particularly confused about how to talk coherently about CFAR.

Considerations around legibility

Summary: CFAR’s work is varied and difficult to evaluate. This has some good features — it can avoid focusing too closely on metrics that don’t measure impact well — but also forces evaluators to rely on factors that aren’t easy to measure, like the quality of its internal culture. On the whole, while I wish CFAR were somewhat more legible, I appreciate the benefits to CFAR’s work of not maximizing “legibility” at the cost of impact or flexibility.

To help me explain my point, let's contrast CFAR with an organization like AMF, which I think of as exceptionally legible. AMF’s work, compared to many other organizations with tens of millions of dollars on hand, is easy to understand: they buy bednets and give them to poor people in developing countries. As long as AMF continues to carry out this plan, and provides basic data showing its success in bednet distribution, I feel like I can easily model what the organization will do. If I found out that AMF was spending 10% of its money funding religious leaders in developing countries to preach good ethical principles for society, or funding the campaigns of government officials favorable to their work, I would be very surprised and feel like some basic agreement or contract had been violated — regardless of whether I thought those decisions, in the abstract, were good or bad for their mission. AMF claims to distribute anti-malaria bednets, and it is on this basis that I would choose whether to support them.

AMF could have been a very different organization, and still could be if it wanted to. For example, it could conduct research on various ways to effect change, and give its core staff the freedom to do whatever they thought was best. This new AMF (“AMF 2.0”) might not be able to tell you exactly what they’ll do next year, because they haven’t figured it out yet, but they can tell you that they’ll do whatever their staff determine is best. This could be distributing deworming pills, pursuing speculative medical research, engaging in political activism, funding religious organizations, etc.

If GiveWell wanted to evaluate AMF 2.0, they would need to use a radically different style of reasoning. There wouldn’t be a straightforward intervention with RCTs to look into. There wouldn’t be a straightforward track record of impact from which to extrapolate. Judging AMF 2.0 would require GiveWell to form much more nuanced judgments about the quality of thinking and execution of AMF’s staff, to evaluate the quality of its internal culture, and to consider a host of other factors that weren’t previously relevant.

I think that evaluating CFAR requires a lot of that kind of analysis, which seems inherently harder to communicate to other people without summarizing one’s views as: "I trust the people in that organization to make good decisions."

The more general idea here is that organizations are subject to bandwidth constraints - they often want to do lots of different things, but their funders need to be able to understand and predict their behavior with limited resources for evaluation. As I've written about recently, a key variable for any organization is the people and organizations by which they are trying to be understood and held accountable. For charities that receive most of their funding in small donations from a large population of people who don’t know much about them, this is a very strong constraint; they must communicate their work so that people can understand it very quickly with little background information. If a charity instead receives most of its funding in large donations from a small set of people who follow it closely, it can communicate much more freely, because the funders will be able to spend a lot of their time talking to the org, exchanging models, and generally coming to an understanding of why the org is doing what it’s doing.

This idea partly explains why most organizations tend to focus on legibility, in how they talk about their work and even in the work they choose to pursue. It can be difficult to attract resources and support from external parties if one’s work isn’t legible.

I think that CFAR is still likely optimizing too little towards legibility, compared to what I think would be ideal for it. Being legible allows an organization to be more confident that its work is having real effects, because it acquires evidence that holds up to a variety of different viewpoints. However, I think that far too many organizations (nonprofit and otherwise) are trying too hard to make their work legible, in a way that reduces innovation and also introduces a variety of adversarial dynamics. When you make systems that can be gamed, and which carry rewards for success (e.g. job stability, prestige, etc), people will reliably turn up to game them [2].

(As Jacob has written in his post on Unconscious Economics, this doesn’t mean people are consciously gaming your system, but merely that this behavior will eventually transpire. The many causes of this include selection effects, reinforcement learning, and memetic evolution.)

In my view, CFAR, by not trying to optimize for a single, easy-to-explain metric, avoids playing the “game” many nonprofits play of focusing on work that will look obviously good to donors, even if it isn’t what the nonprofit believes would be most impactful. They also avoid a variety of other games that come from legibility, such as job applicants getting very good at faking the signals that they are a good fit for an organization, making it harder for them to find good applicants.

Optimizing for communication with the goal of being given resources introduces adversarial dynamics; someone asking for money may provide limited/biased information that raises the chance they’ll be given a grant but reduces the accuracy of the grantmaker’s understanding. (See my comment in Lynette’s writeup below for an example of how this can arise.) This optimization can also tie down your resources, forcing you to carry out commitments you made for the sake of legibility, rather than doing what you think would be most impactful [3].

So I think that it's important that we don't force all organizations towards maximal legibility. (That said, we should ensure that organizations are encouraged to pursue at least some degree of legibility, since the lack of legibility also gives rise to various problems.)

Do I trust CFAR to make good decisions?

As I mentioned in my initial comments on CFAR, I generally think that the current projects CFAR is working on are quite valuable and worth the resources they are consuming. But I have a lot of trouble modeling CFAR’s long-term planning, and I feel like I have to rely instead on my models of how much I trust CFAR to make good decisions in general, instead of being able to evaluate the merits of their actual plans.

That said, I do generally trust CFAR's decision-making. It’s hard to explain the evidence that causes me to believe this, but I’ll give a brief overview anyway. (This evidence probably won’t be compelling to others, but I still want to give an accurate summary of where my beliefs come from):

• I expect that a large fraction of CFAR's future strategic plans will continue to be made by Anna Salamon, from whom I have learned a lot of valuable long-term thinking skills, and who seems to me to have made good decisions for CFAR in the past.
• I think CFAR's culture, while imperfect, is still based on strong foundations of good reasoning with deep roots in the philosophy of science and the writings of Eliezer Yudkowsky (which I think serve as a good basis for learning how to think clearly).
• I have made a lot of what I consider my best and most important strategic decisions in the context of, and aided by, events organized by CFAR. This suggests to me that at least some of that generalizes to CFAR's internal ability to think strategically.
• I am excited about a number of individuals who intend to complete CFAR's latest round of instructor training, which gives me some optimism about CFAR's future access to good talent and its ability to establish and sustain a good internal culture.

Footnotes

[1] The focus on ‘legibility’ in this context I take from James C. Scott’s book “Seeing Like a State.” It was introduced to me by Elizabeth Van Nostrand in this blogpost discussing it in the context of GiveWell and good giving; Scott Alexander also discussed it in his review of the book . Here’s an example from Scott regarding centralized planning and governance:

The centralized state wanted the world to be “legible”, i.e. arranged in a way that made it easy to monitor and control. An intact forest might be more productive than an evenly-spaced rectangular grid of Norway spruce, but it was harder to legislate rules for, or assess taxes on.

[2] The errors that follow are all forms of Goodhart’s Law, which states that “any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.”

[3] The benefits of (and forces that encourage) stability and reliability can maybe be most transparently understood in the context of menu costs and the prevalence of highly sticky wages.

Multi-model approach to corporate and state actors relevant to existential risk mitigation.

I feel a lot of skepticism toward much of the work done in the academic study of international relations. Judging from my models of political influence and its effects on the quality of intellectual contributions, and my models of research fields with little ability to perform experiments, I have high priors that work in international relations is of significantly lower quality than in most scientific fields. However, I have engaged relatively little with actual research on the topic of international relations (outside of unusual scholars like Nick Bostrom) and so am hesitant in my judgement here.

I also have a fair bit of worry around biorisk. I haven’t really had the opportunity to engage with a good case for it, and neither have many of the people I would trust most in this space, in large part due to secrecy concerns from people who work on it (more on that below). Due to this, I am worried about information cascades. (An information cascade is a situation where people primarily share what they believe but not why, and because people update on each others' beliefs you end up with a lot of people all believing the same thing precisely because everyone else does.)

I think is valuable to work on biorisk, but this view is mostly based on individual conversations that are hard to summarize, and I feel uncomfortable with my level of understanding of possible interventions, or even just conceptual frameworks I could use to approach the problem. I don’t know how most people who work in this space came to decide it was important, and those I’ve spoken to have usually been reluctant to share details in conversation (e.g. about specific discoveries they think created risk, or types of arguments that convinced them to focus on biorisk over other threats).

I’m broadly supportive of work done at places like FHI and by the people at OpenPhil who care about x-risks, so I am in favor of funding their work (e.g. Soren’s grant above). But I don’t feel as though I can defer to the people working in this domain on the object level when there is so much secrecy around their epistemic process, because I and others cannot evaluate their reasoning.

However, I am excited about this grant, because I have a good amount of trust in David’s judgment. To be more specific, he has a track record of identifying important ideas and institutions and then working on/with them. Some concrete examples include:

• Wrote up a paper on Goodhart’s Law with Scott Garrabrant (after seeing Scott’s very terse post on it)
• Works with the biorisk teams at FHI and OpenPhil
• Completed his PhD in public policy and decision theory at the RAND Corporation, which is an unusually innovative institution (e.g. this study);
• Writes interesting comments and blog posts on the internet (e.g. LessWrong)
• Has offered mentoring in his fields of expertise to other people working or preparing to work projects in the x-risk space; I’ve heard positive feedback from his mentees

Another major factor for me is the degree to which David is shares his thinking openly and transparently on the internet, and participates in public discourse, so that other people interested in these topics can engage with his ideas. (He’s also a superforecaster, which I think is predictive of broadly good judgment.) If David didn’t have this track record of public discourse, I likely wouldn’t be recommending this grant, and if he suddenly stopped participating, I’d be fairly hesitant to recommend such a grant in the future.

As I said, I’m not excited about the specific project he is proposing, but have trust in his sense of which projects might be good to work on, and I have emphasized to him that I think he should feel comfortable working on the projects he thinks are best. I strongly prefer a world where David has the freedom to work on the projects he judges to be most valuable, compared to the world where he has to take unrelated jobs (e.g. teaching at university).

Supporting aspiring researchers of AI alignment to boost themselves into productivity

From the application:

(1) We would like to apply for a grant to fund an upcoming camp in Madrid that we are organizing. The camp consists of several weeks of online collaboration on concrete research questions, culminating in a 9-day intensive in-person research camp. Participants will work in groups on tightly-defined research projects in strategy and technical AI safety. Expert advisors from AI Safety/Strategy organizations will help refine proposals to be tractable and relevant. This allows for time-efficient use of advisors’ knowledge and research experience, and ensures that research is well-aligned with current priorities. More information: [https://aisafetycamp.com/](https://aisafetycamp.com/)

(2) The field of AI alignment is talent-constrained, and while there is a significant number of young aspiring researchers who consider focussing their career on research on this topic, it is often very difficult for them to take the first steps and become productive with concrete and relevant projects. This is partially due to established researchers being time-constrained and not having time to supervise a large number of students. The goals of AISC are to help a relatively large number of high-talent people to take their first concrete steps in research on AI safety, connect them to collaborate, and efficiently use the capacities of experienced researchers to guide them on their path.

(3) We send out evaluation questionnaires directly after the camp and in regular intervals after the camp has passed. We measure impact on career decisions and collaborations and keep track of concrete output produced by the teams, such as blog posts or published articles.

We have successfully organized two camps before and are in the preparation phase for the third camp taking place in April 2019 near Madrid. I was the main organizer for the second camp and am advising the core team of the current camp, as well as organizing funding.

An overview of previous research projects from the first 2 camps can be found here:

[https://aisafetycamp.com/2018/06/05/aisc-1-research-summaries](https://aisafetycamp.com/2018/06/05/aisc-1-research-summaries)/

[https://aisafetycamp.com/2018/12/07/aisc2-research-summaries/](https://aisafetycamp.com/2018/12/07/aisc2-research-summaries/)

We have evaluated the feedback from participants of the first two camps in the following two documents:

[https://docs.google.com/document/d/1f8wvsvQTv4wdBaggCaK8aKC5gFdIHUDcihnmVkZPM6I/edit?usp=sharing](https://docs.google.com/document/d/1f8wvsvQTv4wdBaggCaK8aKC5gFdIHUDcihnmVkZPM6I/edit?usp=sharing)

[https://docs.google.com/document/d/18v2e-S3iZrOPbE7d9n26sUs1K6CkUAvezRvRj_xlcj8/edit?usp=sharing](https://docs.google.com/document/d/18v2e-S3iZrOPbE7d9n26sUs1K6CkUAvezRvRj_xlcj8/edit?usp=sharing)

My thoughts and reasoning

I’ve talked with various participants of past AI Safety camps and heard broadly good things across the board. I also generally have a positive impression of the people involved, though I don’t know any of the organizers very well.

The material and testimonials that I’ve seen so far suggest that the camp successfully points participants towards a technical approach to AI Alignment, focusing on rigorous reasoning and clear explanations, which seems good to me.

I am not really sure whether I’ve observed significant positive outcomes of camps in past years, though this might just be because I am less connected to the European community these days.

I also have a sense that there is a lack of opportunities for people in Europe to productively work on AI Alignment related problems, and so I am particularly interested in investing in infrastructure and events there. This does however make this a higher-risk grant, since I think this means this event and the people surrounding it might become the main location for AI Alignment in Europe, and if the quality of the event and the people surrounding it isn’t high enough, this might cause long-term problems for the AI Alignment community in Europe.

Concerns

• I think organizing long in-person events is hard, and conflict can easily have outsized negative effects. The reviews that I read from past years suggest that interpersonal conflict negatively affected many participants. Learning how to deal with conflict like this is difficult. The organizers seem to have considered this and thought a lot about it, but the most likely way I expect this grant to have large negative consequences is still if there is some kind of conflict at the camp that results in more serious problems.
• I think it’s inevitable that some people won’t get along with organizers or other participants at the camp for cultural reasons. If that happens, I think it’s important for these people to have some other way of getting connected to people working on AI Alignment. I don’t know the best way to arrange this, but I would want the organizers to think about ways to achieve it.

I also coordinated with Nicole Ross from CEA’s EA Grants project, who had considered also making a grant to the camp. We decided it would be better for the LTF Fund team to make this grant, though we wanted to make sure that some of the concerns Nicole had with this grant were summarized in our announcement:

• AISC could potentially turn away people who would be very good for AI Safety or EA, if those people have negative interactions at the camp or if they are much more talented than other participants (and therefore develop a low opinion of AI Safety and/or EA).
• Some negative interactions with people at the camp could, as with all residential programs, lead to harm and/or PR issues, (for example, if someone at the camp were sexually harassed). Being able to handle such issues thoughtfully and carefully is a hard task, and additional support or advice may be beneficial.

This seems to roughly mirror my concerns above.

I would want to engage with the organizers a fair bit more before recommending a renewal of this grant, but I am happy about the project as a space for Europeans to get engaged with alignment ideas and work on them for a week together with other technical and engaged people.

Broadly, the effects of the camp seem very likely to be positive, while the (financial) cost of the camp seems small compared to the expected size of the impact. This makes me relatively confident that this grant is a good bet.

Conducting independent research into AI forecasting and strategy questions

From the application:

1) I’d like to independently pursue research projects relevant to AI forecasting and strategy, including (but not necessarily limited to) some of the following:

• Does the trajectory of AI capability development match that of biological evolution?
• How tractable is long-term forecasting?
• How much compute did evolution use to produce intelligence?
• Benchmarking AI capabilities against insects
• Collect and summarize the views of people other than Paul, Dario and Jacob on timelines
• Short doc with more detail on the first two projects here: [https://docs.google.com/document/d/1hTLrLXewF-_iJiefyZPF6L677bLrUTo2ziy6BQbxqjs/edit?usp=sharing](https://docs.google.com/document/d/1hTLrLXewF-_iJiefyZPF6L677bLrUTo2ziy6BQbxqjs/edit?usp=sharing)

I am actively pursuing opportunities to work with or under more senior AI strategy researchers [..], so my research focus within AI strategy is likely to be influenced by who exactly I end up working with. Otherwise I expect to spend some short period of time at the start generating more research ideas and conducting pilot tests on the order of several hours into their tractability, then choosing which to pursue based on an importance/tractability/neglectedness framework.

[..]

2) There are relatively few researchers dedicated full-time to investigating AI strategy questions that are not immediately policy-relevant. However, there nonetheless exists room to contribute to the research on existential risks from AI with approaches that fit into neither technical AI safety nor AI policy/governance buckets.

My thoughts and reasoning

Tegan has been a member of the X-risk network for several years now, and recently left AI Impacts. She is now looking for work as a researcher. Two considerations made me want to recommend that the LTF Fund make a grant to her.

• It’s easier to relocate someone who has already demonstrated trust and skills than to find someone completely new.
• It’s important to give good researchers runway while they find the right place.

While I haven’t spent the time to look into Tegan’s research in any depth, the small amount I did read looked promising. The methodology of this post is quite exciting, and her work there and on other pieces seems very thorough and detailed.

That said, my brief assessment of Tegan’s work was not the reason why I recommended this grant, and if Tegan asks for a new grant in 6 months to focus on solo research, I will want to spend significantly more time reading her output and talking with her, to understand how these questions were chosen and what precise relation they have to forecasting technological progress in AI.

Overall, I think Tegan is in a good place to find a valuable role in our collective X-risk reduction project, and I’d like her to have the runway to find that role.

Upskilling in contemporary AI techniques, deep RL and AI safety, before pursuing a ML PhD

From the application :

Notable planned subprojects:

• Engaging with David Krueger’s AI safety reading group at Mila
• Starting & maintaining a public index of AI safety papers, to help future literature reviews and to complement [https://vkrakovna.wordpress.com/ai-safety-resources/](https://vkrakovna.wordpress.com/ai-safety-resources/), as a standalone wiki-page (eg at [http://aisafetyindex.net](http://aisafetyindex.net/))
• From-scratch implementation of seminal deep RL algorithms
• Going through textbooks: Goodfellow Bengio Courville 2016, Sutton Barto 2018, …
• Possibly doing the next AI Safety camp
• Building a prioritization tool for English Wikipedia using NLP, building on the literature of quality assessment ([https://paperpile.com/shared/BZ2jzQ](https://paperpile.com/shared/BZ2jzQ))
• Studying the AI Alignment literature

My thoughts and reasoning

We funded Orpheus in our last grant round to run an AI Safety Unconference just after NeurIPS. We’ve gotten positive testimonials from the event, and I am overall happy about that grant.

I do think that of the grants I recommended this round, this is probably the one I feel least confident about. I don’t know Orpheus very well, and while I have received generally positive reviews of their work, I haven’t yet had the time to look into any of those reviews in detail, and haven’t seen clear evidence about the quality of their judgment. However, what I have seen seems pretty good, and if I had even a tiny bit more time to spend on evaluating this round’s grants, I would probably have spent it reaching out to Orpheus and talking with them more in person.

In general, I think time for self-study and reflection can be exceptionally important for people starting to work in AI Alignment. This is particularly true if they are following a more conventional academic path which could easily cause them to try to immediately work on contemporary AI capabilities research, because I generally think this has negative value even for people concerned about safety (though I do have some uncertainty here). I think giving people working on more classical ML research the time and resources to explore the broader implications of their work on safety, if they are already interested in that, is a good use of resources.

I am also excited about building out the Montreal AI Alignment community, and having someone who both has the time and skills to organize events and can understand the technical safety work seems likely to have good effects.

This grant is also the smallest grant we are funding this round, making me more comfortable with a bit less due diligence than the other grants, especially since this grant seems unlikely to have any large negative consequences.

An offline community hub for rationalists and EAs

From the application:

Our team is working on the offline community hub for rationalists and EAs in Moscow called Kocherga (details on Kocherga are [here](https://www.lesswrong.com/posts/WmfapdnpFfHWzkdXY/rationalist-community-hub-in-moscow-3-years-retrospective)).

We want to make sure it keeps existing and grows into the working model for building new flourishing local EA communities around the globe.

Our key assumptions are:

• There’s a gap between the “monthly meetup” EA communities and the larger (and significantly more productive/important) communities. That gap is hard to close for many reasons.
• Solving this issue systematically would add a lot of value to the global EA movement and, as a consequence, the long-term future of humanity.
• Closing the gap requires a lot of infrastructure, both organizational and technological.

So we work on building such an infrastructure. We also keep in mind the alignment and goodharting issues (building a big community of people who call themselves EAs but who don’t actually share EA virtues would be bad, obviously).

[..]

Concretely, we want to:

• Add 2 more people to our team.
• Implement our new community building strategy (which includes both organizational tasks such as new events and processes for seeding new working groups, and technological tasks such as implementing a website which allows people from the community to announce new private meetups or team up for coaching or mastermind groups)
• Improve our rationality workshops (in terms of scale and content quality). Workshops are important for attracting new community members, for keeping the high epistemic standards of the community and for making sure that community members can be as productive as possible.

To be able to do this, we need to cover our current expenses somehow until we become profitable on our own.

My thoughts and reasoning

The Russian rationality community is surprisingly big, which suggests both a certain level of competence from some of its core organizers and potential opportunities for more community building. The community has:

• Successfully translated The Sequences and HPMOR into Russian, as can be seen at the helpful LessWrong.ru site.
• Executed a successful kickstarter campaign to distribute physical copies of HPMOR (over 7,000 copies).
• Built a community hub in Moscow called Kocherga, which is a financially self-sustaining anti-cafe (a cafe where you pay for time spent there rather than drinks/snacks) that hosts a variety of rationality events for roughly 100 attendees per week.

This grant is to the team that runs the Kocherga anti-cafe.

Their LessWrong write-up suggests:

• They have good skills at building spaces, running events, and generally preserving their culture while still being financially sustainable
• They’ve seen steady increases over time in available funding and attendees
• They’ve succeeded at being largely self-sufficient for 4 years
• They’ve successfully engaged with other local intellectual communities
• Their culture seems to value careful thinking and good discourse a lot, and they seem to have put serious effort into developing the art of rationality, including caring about the technical aspects and incorporating CFAR’s work into their thinking

I find myself having slightly conflicted feelings about the Russian rationality community trying to identify and integrate more with the EA community. I think a major predictor of how excited I have historically been about community building efforts has been a group’s emphasis on improving members’ judgement and thinking skills, as well as the degree to which it emphasizes high epistemic standards and careful thinking. I am quite excited about how Kocherga seems to have focused on those issues so far, and I am worried that this integration and change of identity will reduce that focus (as I think it has for some local and student groups that made a similar transition). That said, I think the Kocherga group has shown quite good judgement on this dimension (see here), which addresses many of my concerns, though I am still interested in thinking and talking about these issues further.

I’m somewhat concerned that I’m not aware of any major insights or unusually talented people from this community, but I expect the language barrier to be a big part of what is preventing me from hearing about those things. And I am somewhat confused about how to account for interesting ideas that don’t spread to the projects I care most about.

I think there are benefits to having an active Russian community that can take opportunities that are only available for people in Russia, or at least people who speak Russian. This particularly applies to policy-oriented work on AI alignment and other global catastrophic risks, which is also a domain that I feel confused about and have a hard time evaluating.

For a lot of the work that I do feel comfortable evaluating, I expect the vast majority of intellectual progress to be made in the English-speaking world, and as such, the question of how talent can flow from Russia to the existing communities working on the long-term future seems quite important. I hope this grant can facilitate a stronger connection between the rest of the world and the Russian community, to improve that talent and idea flow.

—

This grant seemed like a slightly better fit for the EA Meta fund. They decided not to fund it, so we made it instead, since it still seemed like a strong proposal to us.

What I have seen so far makes me confident that this grant is a good idea. However, before we make more grants like this, I would want to talk more to the organizers involved and generally get more information on the structure and culture of the Russian EA and rationality communities.

My thoughts and reasoning

• MIRI is a 20-year-old research organization that seeks to resolve the core difficulties in the way of AGI having a positive impact.
• Only in recent years have there been routes to working on alignment that have also given you funding, status, and a stable social life. Nowadays many others are helping out the work of solving alignment, but MIRI core staff worked on the problem while all the incentives pulled in other directions. For me this is a strong sign of their integrity, and makes me expect they will make good decisions in many contexts where the best action isn't the locally incentivized action. It is also evidence that if I can’t understand why their weird action is good, that they will often still be correct to do it, and this is an outside view in favor of funding them in cases where I don't have my own inside-view model of why the project they're working on is good.
• On that note, MIRI has also worked on a number of other projects that have attempted to teach the skills behind their general methodology for reasoning quantitatively and scientifically about the world and taking right action. I regret not having the time to detail all the impacts of these projects, but they include (and are not limited to): LessWrong, The Sequences, HPMOR, Inadequate Equilibria, Embedded Agency, and CFAR (an organization I discuss below). I view these as some of the main reasons the x-risk community exists.
• Another outside view to consider is the support of MIRI by so many others whom I trust. Their funders have included Open Phil, BERI, FLI, and Jaan Tallinn, plus a variety of smaller donors I trust, and they are advised by Stuart Russell and Nick Bostrom. They’ve also been supported by other people who I don’t necessarily trust directly, but who I do think have interesting and valuable perspectives on the world, like Peter Thiel and Vitalik Buterin .
• I also judge the staff to be exceptionally competent. Some examples:

In sum, I think MIRI is one of the most competent and skilled teams attempting to improve the long-term future, I have a lot of trust in their decision-making, and I’m strongly in favor of ensuring that they’re able to continue their work.

Thoughts on funding gaps

Despite all of this, I have not actually recommended a large grant to MIRI.

• This is due to MIRI’s funding situation being solid at its current level (I would be thinking very differently if I annually had tens of millions of dollars to give away). But MIRI’s marginal use of dollars at this point of funding seems lower-impact, so I only recommended $50k.
• I feel conflicted about whether it might be better to give MIRI more money. Historically, it has been common in the EA funding landscape to only give funding to organizations when they have demonstrated concrete room for more funding, or when funding is the main bottleneck for the organization. I think this has allowed us to start many small organizations that are working on a variety of different problems.

However, this is all complicated by a variety of countervailing considerations, such as the following three:

• Power law distributions of impact only really matter in this way if we can identify which interventions we expect to be in the right tail of impact, and I have a lot of trouble properly bounding my uncertainty here.
• If we are faced with significant uncertainty about cause areas, and we need organizations to have worked in an area for a long time before we can come to accurate estimates about its impact, then it’s a good idea to invest in a broad range of organizations in an attempt to get more information. This is related to common arguments around “explore/exploit tradeoffs”.
• Sometimes, making large amounts of funding available to one organization can have negative consequences for the broader ecosystem of a cause area. Also, giving an organization access to more funding than it can use productively may cause it to make too many hires or lose focus by trying to scale too quickly. Having more funding often also attracts adversarial actors and increases competitive stakes within an organization, making it a more likely target for attackers.

I can see arguments that we should expect additional funding for the best teams to be spent well, even accounting for diminishing margins, but on the other hand I can see many meta-level concerns that weigh against extra funding in such cases. Overall, I find myself confused about the marginal value of giving MIRI more money, and will think more about that between now and the next grant round.

Build infrastructure for the future of effective forecasting efforts

From the application:

What I will do

I applied a few months ago and was granted $20,000 (thanks!). My purpose for this money is similar but greater in scope to the previous round. The previous funding has given me the security to be more ambitious, but I’ve realized that additional guarantees of funding should help significantly more. In particular, engineers can be costly and it would be useful to secure additional funding in order to give possible hires security.

My main overall goal is to advance the use of predictive reasoning systems for purposes most useful for Effective Altruism. I think this is an area that could eventually make use of a good deal of talent, so I have come to see my work at this point as foundational.

This work is in a few different areas that I think could be valuable. I expect that after a while a few parts will emerge as the most important, but think it is good to experiment early when the most effective route is not yet clear.

I plan to use additional funds to scale my general research and development efforts. I expect that most of the money will be used on programming efforts.

Foretold

Foretold is a forecasting application that handles full probability distributions. I have begun testing it with users and have been asked for quite a bit more functionality. I’ve also mapped out the features that I expect people will eventually desire, and think there is a significant amount of work that would be significantly useful.

One particular challenge is figuring out the best way to handle large numbers of questions (1000 active questions plus, at a time.) I believe this requires significant innovations in the user interface and backend architecture. I’ve made some wireframes and have experimented with different methods, and believe I have a pragmatic path forward, but will need to continue to iterate.

I’ve talked with members of multiple organizations at this point who would like to use Foretold once it has a specific set of features, and cannot currently use any existing system for their purposes. […]

Ken

Ken is a project to help organizations set up and work with structured data, in essence allowing them to have private versions of Wikidata. Part of the project is Ken.js, a library which I’m beginning to integrate with Foretold.

Expected Impact

The main aim of EA forecasting would be to better prioritize EA actions. I think that if we could have a powerful system set up, it could make us better at predicting the future, better at understanding what things are important and better at coming to a consensus on challenging topics.

Measurement

In the short term, I’m using heuristics like metrics regarding user activity and upvotes on LessWrong. I’m also getting feedback by many people in the EA research community. In the medium to long term, I hope to set up evaluation/estimation procedures for many projects and would include this one in that process.

My thoughts and reasoning

This grant is to support Ozzie Gooen in his efforts to build infrastructure for effective forecasting. Ozzie requested $70,000 to hire a software engineer who would support him on his work on the prediction platform www.foretold.iothat he is working on.

• When thinking about how to improve the long-term future, I think we are confused about what counts as progress and what specific problems need solving. We can already see that there are a lot of technical and conceptual problems that have to be solved to make progress on a lot of the big problems we think are important.
• I think that in order to make effective intellectual progress, you need some way for many people to collaborate on solving problems and to document the progress they have made so far.
• I think there is potentially a lot of low-hanging fruit in designing better online platforms for making intellectual progress (which is why I chose to work on LessWrong + AI Alignment Forum + EA Forum). Ozzie works in this space too, and previously built Guesstimate (a spreadsheet where every cell is a probability distribution), which I think displayed some real innovation in the way we can use technology to communicate and clarify ideas. It was also produced to a very high standard of quality.
• Forecasting platforms in particular have already displayed significant promise and tractability, with recent work by Philip Tetlock showing that a simple prediction platform can outperform major governmental institutions like the CIA, and older work by Robin Hanson, showing ways that prediction markets could help us make progress on a number of interesting problems.
• The biggest concerns I have with Ozzie’s work, as well as the work on other prediction and aggregation platforms, is that the problem of getting people to actually use the product turns out to be very hard. Matt Fallshaw’s team at Trike Apps built https://predictionbook.com/, but then found it hard to get people to actually use it. Ozzie’s last project, Guesstimate, seemed quite well-executed, but similarly faltered due to low user numbers and a lack of interest from potential customers in industry. As such, I think it’s important not to underestimate the difficulty of making the product good enough that people actually use it.
• I do think that the road to building knowledge aggregation platforms will include many failed projects and many experiments that never get traction; as such, I do think that one should not over-update on the lack of users for some of the existing platforms. As a positive counterexample, the Good Judgment Project seems to have a consistently high number of people making predictions.
• I’ve also frequently interacted with Ozzie in person, and generally found his reasoning and judgment in this domain to be good. I also think it is quite good that he has been writing up his thinking for the community to read and engage with, which will allow other people to build off of his thinking and efforts, even if he doesn’t find traction with this particular project.

Building towards a “Limited Agent Foundations” thesis on mild optimization and corrigibility

From the application:

I am a third-year computer science PhD student funded by a graduate teaching assistantship; to dedicate more attention to alignment research, I am applying for one or more trimesters of funding (spring term starts April 1).

[…]

Last summer, I designed an approach to the “impact measurement” subproblem of AI safety: “what equation cleanly captures what it means for an agent to change its environment, and how do we implement it so that an impact-limited paperclip maximizer would only make a few thousand paperclips?”. I believe that my approach, [Attainable Utility Preservation](https://www.lesswrong.com/posts/yEa7kwoMpsBgaBCgb/towards-a-new-impact-measure) (AUP), goes a long way towards answering both questions robustly, concluding:

> By changing our perspective from “what effects on the world are ‘impactful’?” to “how can we stop agents from overfitting their environments?”, a natural, satisfying definition of impact falls out. From this, we construct an impact measure with a host of desirable properties […] AUP agents seem to exhibit qualitatively different behavior […]

Primarily, I aim both to output publishable material for my thesis and to think deeply about the corrigibility and mild optimization portions of MIRI’s machine learning research agenda. Although I’m excited by what AUP makes possible, I want to lay the groundwork of deep understanding for multiple alignment subproblems. I believe that this kind of clear understanding will make positive AI outcomes more likely.

My thoughts and reasoning

I’m excited about this because:

• Alex’s approach to finding personal traction in the domain of AI Alignment is one that I would want many other people to follow. On LessWrong, he read and reviewed a large number of math textbooks that are useful for thinking about the alignment problem, and sought public input and feedback on what things to study and read early on in the process.
• He wasn’t intimidated by the complexity of the problem, but started thinking independently about potential solutions to important sub-problems long before he had “comprehensively” studied the mathematical background that is commonly cited as being the foundation of AI Alignment.
• He wrote up his thoughts and hypotheses in a clear way, sought feedback on them early, and ended up making a set of novel contributions to an interesting sub-field of AI Alignment quite quickly (in the form of his work on impact measures, on which he recently collaborated with the DeepMind AI Safety team)

Potential concerns

These intuitions, however, are a bit in conflict with some of the concrete research that Alex has actually produced. My inside views on AI Alignment make me think that work on impact measures is very unlikely to result in much concrete progress on what I perceive to be core AI Alignment problems, and I have talked to a variety of other researchers in the field who share that assessment. I think it’s important that this grant not be viewed as an endorsement of the concrete research direction that Alex is pursuing, but only as an endorsement of the higher-level process that he has been using while doing that research.

As such, I think it was a necessary component of this grant that I have talked to other people in AI Alignment whose judgment I trust, who do seem excited about Alex’s work on impact measures. I think I would not have recommended this grant, or at least this large of a grant amount, without their endorsement. I think in that case I would have been worried about a risk of diverting attention from what I think are more promising approaches to AI Alignment, and a potential dilution of the field by introducing a set of (to me) somewhat dubious philosophical assumptions.

Overall, while I try my best to form concrete and detailed models of the AI Alignment research space, I don’t currently devote enough time to it to build detailed models that I trust enough to put very large weight on my own perspective in this particular case. Instead, I am mostly deferring to other researchers in this space that I do trust, a number of whom have given positive reviews of Alex’s work.

In aggregate, I have a sense that the way Alex went about working on AI Alignment is a great example for others to follow, I’d like to see him continue, and I am excited about the LTF Fund giving out more grants to others who try to follow a similar path.

Working to prevent burnout and boost productivity within the EA and X-risk communities

From the application:

(1) After 2 years as a CFAR instructor/researcher, I’m currently in a 6-12 month phase of reorienting around my goals and plans. I’m requesting a grant to spend the coming year thinking about rationality and testing new projects.

(2) I want to help individuals and orgs in the x-risk community orient towards and achieve their goals.

(A) I want to train the skill of dependability, in myself and others.

This is the skill of a) following through on commitments and b) making prosocial / difficult choices in the face of fear and aversion. The skill of doing the correct thing, despite going against incentive gradients, seems to be the key to virtue.

One strategy I’ve used is to surround myself with people with shared values (CFAR, Bay Area) and trust the resulting incentive gradients. I now believe it is also critical to be the kind of person who can take correct action despite prevailing incentive structures.

Dependability is also related to thinking clearly. Your ability to make the right decision depends on your ability to hold and be with all possible realities, especially painful and aversive ones. Most people have blindspots that actively prevent this.

I have some leads on how to train this skill, and I’d like both time and money to test them.

(B) Thinking clearly about AI risk

Most people’s decisions in the Bay Area AI risk community seem model-free. They themselves don’t have models of why they’re doing what they’re doing; they’re relying on other people “with models” to tell them what to do and why. I’ve personally carried around such premises. I want to help people explore where their ‘placeholder premises’ are and create safety for looking at their true motivations, and then help them become more internally and externally aligned.

(C) Burnout

Speaking of “not getting very far.” My personal opinion is that most ex-CFAR employees left because of burnout; I’ve written what I’ve learned here, see top 2 comments: [[https://forum.effectivealtruism.org/posts/NDszJWMsdLCB4MNoy/burnout-what-is-it-and-how-to-treat-it#87ue5WzwaFDbGpcA7](https://forum.effectivealtruism.org/posts/NDszJWMsdLCB4MNoy/burnout-what-is-it-and-how-to-treat-it#87ue5WzwaFDbGpcA7)]. I’m interested in working with orgs and individuals to prevent burnout proactively.

(3) Some possible measurable outputs / artifacts:

• A program where I do 1-on-1 sessions with individuals or orgs; I’d create reports based on whether they self-report improvements
• X-risk orgs (e.g. FHI, MIRI, OpenPhil, BERI, etc.) deciding to spend time/money on my services may be a positive indicator, as they tend to be thoughtful with how they spend their resources
• Writings or talks
• Workshops with feedback forms
• A more effective version of myself (notable changes = gaining the ability to ride a bike / drive a car / exercise—a PTSD-related disability, ability to finish projects to completion, others noticing stark changes in me)

My thoughts and reasoning

Lauren worked as an instructor at CFAR for about 2 years, until Fall 2018. I review CFAR’s impact as an institution below; in general, I believe it has helped set a strong epistemic foundation for the community and been successful in recruitment and training. I have a great appreciation for everyone who helps them with their work.

Lauren is currently in a period of reflection and reorientation around her life and the problem of AGI, in part due to experiencing burnout in the months before she left CFAR. To my knowledge, CFAR has never been well-funded enough to offer high salaries to its employees, and I think it is valuable to ensure that people who work at EA orgs and burn out have the support to take the time for self-care after quitting due to long-term stress. Ideally, I think this should be improved by higher salaries that allow employees to build significant runway to deal with shocks like this, but I think that the current equilibrium of salary levels in EA does not make that easy. Overall, I think it’s likely that staff at highly valuable EA orgs will continue burning out, and I don’t currently see it as an achievable target to not have this happen (though I am in favor of people people working on solving the problem).

I do not know Lauren well enough to evaluate the quality of her work on the art of human rationality, but multiple people I trust have given positive reviews (e.g. see Alex Zhu above), so I am also interested to read her output on the subjects she is thinking about.

I think it’s very important that people who work on developing an understanding of human rationality take the time to add their knowledge into our collective understanding, so that others can benefit from and build on top of it. Lauren has begun to write up her thoughts on topics like burnout, intentions, dependability, circling, and curiosity, and her having the space to continue to write up her ideas seemed like a significant additional positive outcome of this grant.

I think that she should probably aim to make whatever she does valuable enough that individuals and organizations in the community wish to pay her directly for her work. It’s unlikely that I would recommend renewing this grant for another 6 month period in the absence of a relatively exciting new research project/direction, and if Lauren were to reapply, I would want to have a much stronger sense that the projects she was working on were producing lots of value before I decided to recommend funding her again.

In sum, this grant hopefully helps Lauren to recover from burning out, get the new rationality projects she is working on off the ground, potentially identify a good new niche for her to work in (alone or at an existing organization), and write up her ideas for the community.

A major expansion of the [Metaculus](https://metaculus.com/) prediction platform and its community

From the application:

The funds would be used to expand the Metaculus prediction platform along with its community. Metaculus.com is a fully-functional prediction platform with ~10,000 registered users and >120,000 predictions made to date on more than >1000 questions. The goals of Metaculus are:

• Short-term: Provide a resource to science, tech, and (especially) EA-related communities already interested in generating, aggregating, and employing accurate predictions, and training to be better predictors.
• Medium-term: Improve decision-making by individuals and groups by providing well-calibrated numerical predictions.
• Long-term: encourage and backstop a widespread culture of accountable and accurate predictions and scenario planning.

There are two major high-priority expansions possible with funding in place. The first would be an integrated set of extensions to improve user interaction and information-sharing. This would include private messaging and notifications, private groups, a prediction “following” system to create micro-teams within individual questions, and various incentives and systems for information-sharing.

The second expansion would link questions into a network. Users would express links between questions, from very simple (“notify me regarding question Y when P(X) changes substantially) to more complex (“Y happens only if X happens, but not conversely”, etc.) Information can also be gleaned from what users actually do. The strength and character of these relations can then generate different graphical models that can be explored interactively, with the ultimate goal of a crowd-sourced quantitative graphical model that could structure event relations and propagate new information through the network.

My thoughts and reasoning

For this grant, and also the grants to Ozzie Gooen and Jacob L., I did not have enough time to write up my general thoughts on forecasting platforms and communities. I hope to later write a post with my thoughts here. But for a short summary, see my thoughts on Ozzie Gooen’s grant.

I am generally excited about people building platforms for coordinating intellectual labor, particularly on topics that are highly relevant to the long-term future. I think Metaculus has been providing a valuable service for the past few years, both in improving our collective ability to forecast a large variety of important world events and in allowing people to train and demonstrate their forecasting skills, which I expect to become more relevant in the future.

I am broadly impressed with how cooperative and responsive the Metaculus team has been in helping organizations in the X-risk space get answers to important questions, or provide software services to them (e.g. I know that they are helping Jacob L. and Ben Goldhaber set up a private Metaculus instance focused on AI)

I don’t know Anthony well, and overall I am quite concerned that there is no full-time person on this project. My model is that projects like this tend to go a lot better if they have one core champion who has the resources to fully dedicate themselves to the project, and it currently doesn’t seem that Anthony is able to do that.

My current model is that Metaculus will struggle as a platform without a fully dedicated team or at least individual champion, though I have not done a thorough investigation of the Metaculus team and project, so I am not very confident of this. One of the major motivations for this grant is to ensure that Metaculus has enough resources to hire a potential new champion for the project (who ideally also has programming skills or UI design skills to allow them to directly work on the platform). That said, Metaculus should use the money as best they see fit.

I am also concerned about the overlap of Metaculus with the Good Judgment Project, and currently have a sense that it suffers from being in competition with it, while also having access to substantially fewer resources and people.

The requested grant amount was for $150k, but I am currently not confident enough in this grant to recommend filling the whole amount. If Metaculus finds an individual new champion for the project, I can imagine strongly recommending that it gets fully funded, if the new champion seems competent.

Broad project support for rationality and community building interventions

Eli has worked on a large variety of interesting and valuable projects over the last few years, many of them too small to have much payment infrastructure, resulting in him doing a lot of work without appropriate compensation. I think his work has been a prime example of picking low-hanging fruit by using local information and solving problems that aren’t worth solving at scale, and I want him to have resources to continue working in this space.

Concrete examples of projects he has worked on that I am excited about:

• Facilitating conversations between top people in AI alignment (I’ve in particular heard very good things about the 3-day conversation between Eric Drexler and Scott Garrabrant that Eli helped facilitate)
• Organizing advanced workshops on Double Crux and other key rationality techniques
• Doing a variety of small independent research projects, like this evaluation of birth order effects in mathematicians
• Providing many new EAs and rationalists with advice and guidance on how to get traction on working on important problems
• Helping John Salvatier develop techniques around skill transfer

I think Eli has exceptional judgment, and the goal of this grant is to allow him to take actions with greater leverage by hiring contractors, paying other community members for services, and paying for other varied expenses associated with his projects.

Perform independent research in collaboration with John Salvatier

I am recommending this grant with more hesitation than most of the other grants in this round. The reasons for hesitation are as follows:

• I was the primary person on the grant committee on whose recommendation this grant was made.
• Connor lives in the same group house that I live in, which I think adds a complicating conflict of interest to my recommendation.
• I have generally positive impressions of Connor, but I have not personally seen concrete, externally verifiable evidence that clearly demonstrates his good judgment and competence, which in combination with the other two factors makes me more hesitant than usual.

However, despite these reservations, I think this grant is a good choice. The two primary reasons are:

• Connor himself has worked on a variety of research and community building projects, and both by my own assessment and other people I talked to, has significant potential in becoming a strong generalist researcher, which I think is an axis on which a lot of important projects are bottlenecked.
• This grant was strongly recommended to me by John Salvatier, who is funded by an EA Grant and whose work I am generally excited about.

John did some very valuable community organizing while he lived in Seattle and is now working on developing techniques to facilitate skill transfer between experts in different domains. I think it is exceptionally hard to develop effective techniques for skill transfer, and more broadly techniques to improve people’s rationality and reasoning skills, but am sufficiently impressed with John’s thinking that I think he might be able to do it anyway (though I still have some reservations).

John is currently collaborating with Connor and requested funding to hire him to collaborate on his projects. After talking to Connor I decided it would be better to recommend a grant to Connor directly, encouraging him to continue working with John but also allowing him to switch towards other research projects if he finds he can’t contribute as productively to John’s research as he expects.

Overall, while I feel some hesitation about this grant, I think it’s very unlikely to have any significant negative consequences, and I assign some significant probability that this grant can help Connor develop into an excellent generalist researcher of a type that I feel like EA is currently quite bottlenecked on.

Giving copies of Harry Potter and the Methods of Rationality to the winners of EGMO 2019 and IMO 2020

From the application:

*EA Russia has the oral agreements with IMO [International Math Olympiad] 2020 (Saint Petersburg, Russia) & EGMO [European Girls’ Mathematical Olympiad] 2019 (Kyiv, Ukraine) organizers to give HPMORs [copies of [Harry Potter and the Methods of Rationality](http://www.hpmor.com/)] to the medalists of the competitions. We would also be able to add an EA / rationality leaflet made by CFAR (I contacted Timothy Telleen-Lawton on that matter).*

My thoughts and reasoning

[Edit & clarification: The books will be given out by the organisers of the IMO and EGMO as prizes for the 650 people who got far enough to participate, all of which are "medalists".]

My model for the impact of this grant roughly breaks down into three questions:

• What effects does reading HPMOR have on people?
• How good of a target group are Math Olympiad winners for these effects?
• Is the team competent enough to execute on their plan?

What effects does reading HPMOR have on people?

My models of the effects of HPMOR stem from my empirical observations and my inside view on rationality training.

• Empirically, a substantial number of top people in our community have (a) entered due to reading and feeling a deep connection to HPMOR and (b) attributed their approach to working on the long term future in substantial part to the insights they learned from reading HPMOR. This includes some individuals receiving grants on this list, and some individuals on the grant-making team.
• I also weight here my inside view of the skills that HPMOR helps to teach. I’ll try to point at the things I think HPMOR does exceptionally and uniquely well at, though I find it a bit hard to make my models fully explicit here in an appropriate amount of space.
• It is finally worth noting that HPMOR does all of the above things while also being a highly engaging book that has been read by hundreds of thousands of readers (if not more) primarily for pleasure. It is the most reviewed Harry Potter fan fiction on fanfiction.net, which is a remarkable state of affairs.

How good of a target group are Math Olympiad winners for these effects?

I think that Math Olympiad winners are a very promising demographic within which to find individuals who can contribute to improving the long-term future. I believe Math Olympiads select strongly on IQ as well as (weakly) on conscientiousness and creativity, which are all strong positives. Participants are young and highly flexible; they have not yet made too many major life commitments (such as which university they will attend), and are in a position to use new information to systematically change their lives’ trajectories. I view handing them copies of an engaging book that helps teach scientific, practical and quantitative thinking as a highly asymmetric tool for helping them make good decisions about their lives and the long-term future of humanity.

I’ve also visited and participated in a variety of SPARC events, and found the culture there (which is likely to be at least somewhat representative of Math Olympiad culture) very healthy in a broad sense. Participants displayed high levels of altruism, a lot of willingness to help one another, and an impressive amount of ambition to improve their own thinking and affect the world in a positive way. These observations make me optimistic about efforts that build on that culture.

I think it’s important when interacting with minors, and attempting to improve (and thus change) their life trajectories, to make sure to engage with them in a safe way that is respectful of their autonomy and does not put social pressures on them in ways they may not yet have learned to cope with. In this situation, Mikhail is working with/through the institutions that run the IMO and EGMO, and I expect those institutions to (a) have lots of experience with safeguarding minors and (b) have norms in place to make sure that interactions with the students are positive.

Is the team competent enough to execute on their plan?

I don’t have a lot of information on the team, don’t know Mikhail, and have not received any major strong endorsement for him and his team, which makes this the weakest link in the argument. However, I know that they are coordinating both with SPARC (which also works to give books like HPMOR to similar populations) and the team behind the highly successful Russian printing of HPMOR, two teams who have executed this kind of project successfully in the past. So I felt comfortable recommending this grant, especially given its relatively limited downside.

[Edit: It seems relevant to mention that LessWrong is currently receiving operational support from CFAR, in a way that makes me technically an employee of CFAR (similar to how ACE and 80K were/are part of CEA for a long time). However, LessWrong operates as a completely separate entity with its own fundraising and hiring procedures, and I don't feel any hesitation or pressure to critique CFAR openly because of that relation. Though I find myself a tiny bit more hesitant to speak harshly of specific individuals, simply because I am only working a floor away from the CFAR offices and that does have some psychological effect on me. Though the same was true for CEA while LessWrong was located in the CEA office for a few months, and was true for residents of my group house while LessWrong was located in the living room of my group house for most of the past two years, so I don't think this effect is particularly large.]

I think that CFAR’s intro workshops have historically had a lot of positive impact. I think they have done so via three pathways.

• Establishing epistemic norms: I think CFAR workshops are quite good at helping the EA and rationality community establish norms about what good discourse and good reasoning look like. As a concrete example of this, the concept of Double Crux has gotten traction in the EA and rationality communities, which has improved the way ideas and information spread throughout the community, how ideas get evaluated, and what kinds of projects get resources. More broadly, I think CFAR workshops have helped in establishing a set of common norms about what good reasoning and understanding look like, similar to the effect of the sequences on LessWrong.
• Training: A percentage of intro workshop participants (many of whom were already working on important problems within X-risk) have seen significant improvements in competence; as a result, they became substantially more effective in their work.
• Recruitment: CFAR has helped many people move from passive membership in the EA and rationality community to having strong social bonds in the X-risk network.

While I do think that CFAR has historically caused a significant amount of impact, I feel hesitant about this grant because I am unsure whether CFAR can continue to create the same amount of impact in the future. I have a few reasons for this:

• First, all of its founding staff and many other early staff have left. I broadly expect organizations to get a lot worse once their early staff leaves.
• From recent conversations with CFAR, I’ve gotten a sense that the staff isn’t interested in increasing the number of intro workshops, that the intro workshops don’t feel particularly exciting for the staff, and that most staff are less interested in improving the intro workshops than other parts of CFAR. This makes it less likely that those workshops will maintain their quality and impact, and I currently think that those workshops are likely one of the best ways for CFAR to have a large impact.
• I have a general sense that CFAR is struggling to attract top talent, partially because some of the best staff left, and partially due to a general sense of a lack of forward momentum for the organization. This is a bad sign, because I think CFAR in particular benefits from having highly talented individuals teach at their workshops and serve as a concrete example of the skills they’re trying to teach.
• My impression is that while the intro workshops were historically focused on instrumental rationality and personal productivity, the original CFAR staff was oriented quite strongly around truth-seeking. Core rationality concepts were conveyed indirectly by the staff in smaller conversations and in the broader culture of the organization. The current staff seems less oriented around that kind of epistemic rationality, and so I expect that if they continue their current focus on personal productivity and instrumental rationality, the epistemic benefits of CFAR workshops will be reduced significantly, and those are the benefits I care about most.

However, there are some additional considerations that led me to recommending this grant.

• First, CFAR and MIRI are collaborating on a set of programmer-focused workshops that I am also quite positive on. I think those workshops are less directly influenced by counterfactual donations than the mainline workshops, since I expect MIRI to fund them in any case, but they do still rely on CFAR existing as an institution that can provide instructors. I am excited about the opportunities the workshops will enable in terms of curriculum development, since they can focus almost solely on epistemic rationality
• Second, I think that if CFAR does not receive a grant now, there’s a good chance they’d be forced to let significant portions of their staff go, or take some other irreversible action. CFAR decided not to run a fundraiser last fall because they felt like they’d made significant mistakes surrounding a decision made by a community dispute panel that they set up and were responsible for, and they felt like it would be in poor taste to ask the community for money before they thoroughly investigated what went wrong and released a public statement.

In the last year, I had some concerns about the way CFAR communicated a lot of its insights, and I sensed an insufficient emphasis on a kind of robust and transparent reasoning that I don’t have a great name for. I don’t think the communication style I was advocating for is always the best way to make new discoveries, but is very important for establishing broader community-wide epistemic norms and enables a kind of long-term intellectual progress that I think is necessary for solving the intellectual challenges we’ll need to overcome to avoid global catastrophic risks. I think CFAR is likely to respond to last year’s events by improving their communication and reasoning style in this respect (from my perspective).

My overall read is that CFAR is performing a variety of valuable community functions and has a strong enough track record that I want to make sure that it can continue existing as an institution. I didn’t have enough time this grant round to understand how the future of CFAR will play out; the current grant amount seems sufficient to ensure that CFAR does not have to take any drastic action until our next grant round. By the next grant round, I plan to have spent more time learning and thinking about CFAR’s trajectory and future, and to have a more confident opinion about what the correct funding level for CFAR is.

Producing video content on AI alignment

From the application:

My goals are:

• To communicate to intelligent and technically-minded young people that AI Safety:
• To help others in the field communicate and advocate better, by providing high quality, approachable explanations of AIS concepts that people can share, instead of explaining the ideas themselves, or sharing technical documents that people won’t read
• To motivate myself to read and internalise the papers and textbooks, and become a technical AIS researcher in future

My thoughts and reasoning

I think video is a valuable medium for explaining a variety of different concepts (for the best examples of this, see 3Blue1Brown, CGP Grey, and Khan Academy). While there are a lot of people working directly on improving the long term future by writing explanatory content, Rob is the only person I know who has invested significantly in getting better at producing video content. I think this opens a unique set of opportunities for him.

The videos on his Youtube channel pick up an average of ~20k views. His videos on the official Computerphile channel often pick up more than 100k views, including for topics like logical uncertainty and corrigibility (incidentally, a term Rob came up with).

More things that make me optimistic about Rob’s broad approach:

• He explains that AI alignment is a technical problem. AI safety is not primarily a moral or political position; the biggest chunk of the problem is a matter of computer science. Reaching out to a technical audience to explain that AI safety is a technical problem, and thus directly related to their profession, is a type of ‘outreach’ that I’m very happy to endorse.
• He does not make AI safety a politicized matter. I am very happy that Rob is not needlessly tribalising his content, e.g. by talking about something like “good vs bad ML researchers”. He seems to simply portray it as a set of interesting and important technical problems in the development of AGI.
• His goal is to create interest in these problems from future researchers, and not to simply get as large of an audience as possible. As such, Rob’s explanations don’t optimize for views at the expense of quality explanation. His videos are clearly designed to be engaging, but his explanations are simple and accurate. Rob often interacts with researchers in the community (at places like DeepMind and MIRI) to discuss which concepts are in need of better explanations. I don’t expect Rob to take unilateral action in this domain.

Rob is the first skilled person in the X-risk community working full-time on producing video content. Being the very best we have in this skill area, he is able to help the community in a number of novel ways (for example, he’s already helping existing organizations produce videos about their ideas).

Rob made a grant request during the last round, in which he explicitly requested funding for a collaboration with RAISE to produce videos for them. I currently don’t think that working with RAISE is the best use of Rob’s talent, and I’m skeptical of the product RAISE is currently trying to develop. I think it’s a better idea for Rob to focus his efforts on producing his own videos and supporting other organizations with his skills, though this grant doesn’t restrict him to working with any particular organization and I want him to feel free to continue working on RAISE if that is the project he thinks is currently most valuable.

Overall, Rob is developing a new and valuable skill within the X-risk community, and executing on it in a very competent and thoughtful way, making me pretty confident that this grant is a good idea.