Anthony Aguirre

Making Metaculus useful and available to EA and other organizations

I’ve written up rationales for grants to Metaculus in the past, which you can see here.

In my last writeup on Metaculus, I said the following:

My current model is that Metaculus will struggle as a platform without a fully dedicated team or at least individual champion, though I have not done a thorough investigation of the Metaculus team and project, so I am not very confident of this. One of the major motivations for this grant is to ensure that Metaculus has enough resources to hire a potential new champion for the project (who ideally also has programming skills or UI design skills to allow them to directly work on the platform). That said, Metaculus should use the money as best they see fit.

I am also concerned about the overlap of Metaculus with the Good Judgment Project, and currently have a sense that it suffers from being in competition with it, while also having access to substantially fewer resources and people.

The requested grant amount was for $150k, but I am currently not confident enough in this grant to recommend filling the whole amount. If Metaculus finds an individual new champion for the project, I can imagine strongly recommending that it gets fully funded, if the new champion seems competent.

I’ve since thought a lot more about Metaculus, have used the platform more myself, and have broadly been very happy with the progress that the platform has made since I wrote the summary above. As far as I know, Metaculus now has a full-time champion for the project (Tamay Besiroglu), and has demonstrated to me significant advantages over platforms like the Good Judgement Project, in particular with its ability to enter probability distributions over events and a question curation team that is much better at producing forecasts that I care about and seem important to me (and, I would guess, others working on global catastrophic risk).

This overall makes me excited about Metaculus’s future and the effects of this grant.

A major expansion of the [Metaculus](https://metaculus.com/) prediction platform and its community

From the application:

The funds would be used to expand the Metaculus prediction platform along with its community. Metaculus.com is a fully-functional prediction platform with ~10,000 registered users and >120,000 predictions made to date on more than >1000 questions. The goals of Metaculus are:

• Short-term: Provide a resource to science, tech, and (especially) EA-related communities already interested in generating, aggregating, and employing accurate predictions, and training to be better predictors.
• Medium-term: Improve decision-making by individuals and groups by providing well-calibrated numerical predictions.
• Long-term: encourage and backstop a widespread culture of accountable and accurate predictions and scenario planning.

There are two major high-priority expansions possible with funding in place. The first would be an integrated set of extensions to improve user interaction and information-sharing. This would include private messaging and notifications, private groups, a prediction “following” system to create micro-teams within individual questions, and various incentives and systems for information-sharing.

The second expansion would link questions into a network. Users would express links between questions, from very simple (“notify me regarding question Y when P(X) changes substantially) to more complex (“Y happens only if X happens, but not conversely”, etc.) Information can also be gleaned from what users actually do. The strength and character of these relations can then generate different graphical models that can be explored interactively, with the ultimate goal of a crowd-sourced quantitative graphical model that could structure event relations and propagate new information through the network.

My thoughts and reasoning

For this grant, and also the grants to Ozzie Gooen and Jacob L., I did not have enough time to write up my general thoughts on forecasting platforms and communities. I hope to later write a post with my thoughts here. But for a short summary, see my thoughts on Ozzie Gooen’s grant.

I am generally excited about people building platforms for coordinating intellectual labor, particularly on topics that are highly relevant to the long-term future. I think Metaculus has been providing a valuable service for the past few years, both in improving our collective ability to forecast a large variety of important world events and in allowing people to train and demonstrate their forecasting skills, which I expect to become more relevant in the future.

I am broadly impressed with how cooperative and responsive the Metaculus team has been in helping organizations in the X-risk space get answers to important questions, or provide software services to them (e.g. I know that they are helping Jacob L. and Ben Goldhaber set up a private Metaculus instance focused on AI)

I don’t know Anthony well, and overall I am quite concerned that there is no full-time person on this project. My model is that projects like this tend to go a lot better if they have one core champion who has the resources to fully dedicate themselves to the project, and it currently doesn’t seem that Anthony is able to do that.

My current model is that Metaculus will struggle as a platform without a fully dedicated team or at least individual champion, though I have not done a thorough investigation of the Metaculus team and project, so I am not very confident of this. One of the major motivations for this grant is to ensure that Metaculus has enough resources to hire a potential new champion for the project (who ideally also has programming skills or UI design skills to allow them to directly work on the platform). That said, Metaculus should use the money as best they see fit.

I am also concerned about the overlap of Metaculus with the Good Judgment Project, and currently have a sense that it suffers from being in competition with it, while also having access to substantially fewer resources and people.

The requested grant amount was for $150k, but I am currently not confident enough in this grant to recommend filling the whole amount. If Metaculus finds an individual new champion for the project, I can imagine strongly recommending that it gets fully funded, if the new champion seems competent.