Marcus Abramovitch

Animal Welfare Midtraining Data Creation

$20,000

**Project summary**

There is a short window of opportunity to provide mid-training data that helps AI models reason well about animal welfare. Constance Li has been leading a sprint project to get this done with a few people I like (Allen Lu, among others).

This funding slightly funges with Sentient Futures funding, but I’m very okay with that. I’m expecting more money to come into the Falcon Fund, so I am awarding $20k here.

The idea is to train on data where AIs treat animals well and behave admirably towards them, but, most importantly, to explain why the decisions are being made so that, hopefully, this generalizes out of distribution. Fiction is completely fine for this.

I think it's quite reasonable to think “animal welfare people” should be doing this as opposed to lab employees. They have much more nuanced, well-thought-out views on why things are bad for animals, from a variety of philosophical perspectives, and will do a better job of explaining why.

**What are this project's goals and how can they be achieved?**

Create mid-training data to give to the AI Labs

**How will this funding be used?**

Pay people to create the mid-training data.

**Who is on the team, and what's their track record on similar projects?**

Constance Li - Leads Sentient Futures

Allen Lu - Created the MANTA benchmark

Aidan Kanknyoku - Animal welfare alignment team at Anima

Sentient Futures Residents

**What are the most likely causes and outcomes if this project fails? (premortem)**

This turns out not to be that useful; it's quite speculative. It could also just get trained out in RL.

**What other funding is this person or project getting?**

It's part of Sentient Futures, but they don't have enough money.

**Main points in favor of this grant**

The project seems good and promising, and I don't know of any other grantmaker who would make this.

**Donor's main reservations**

I think this is unlikely to have very large effects on models, at least at this scale.

**Process for deciding the amount**

I think it could use more than this. My main uncertainty is how much funding the Falcon Fund will get.

**Conflicts of interest**

N/A

Please disclose e.g. any romantic, professional, financial, housemate, or familial relationships you have with the grant recipient(s).

Get in contact with @OzzieGooen. Ill fund a continuation of what he has

Emptying the rest of my Manifold balance for now into this and planning to give more. I hope this gets the full $150k required.

After a ~45 min call with Krishna, I was very impressed with how clear she was communicating what she was doing, why she was doing it and the metrics for results she had. Furthermore, she had good reasons for wanting this work to be done outside of a major lab.

Furthermore, I think this is a very promising plausible safety feature to combat CBRN risk from models, isolate the part of them that knows about it while keeping the remainder of capabilities intact.

Overall, I found Krishna to be very smart. In my brief conversation with her mentor Alec from a while ago, I also think he has great insights.

I don't really have major reservations here. I think this grant will be similar to my grant to Joseph Bloom of a couple years ago.

Main points in favor of this grant

Donor's main reservations

• Justin is currently making AI safety Youtube videos and I think he’s been great at writing these scripts. The channel he has done this for has skyrocketed and this will take him away from that. That said, I generally think people should work on that which they want to work on
• Growth may take longer than expected, since YouTube’s discovery algorithm can be slow early on.
• The best format is not known yet; Justin may be more effective as an off-screen narrator than an on-camera host.

Process for deciding amount

Justin made the budget. I thought it was reasonable though we discussed a lot what goes into making Youtube videos, making them good, etc. and reasonable costs.

Conflicts of interest

Please disclose e.g. any romantic, professional, financial, housemate, or familial relationships you have with the grant recipient(s).

None

I find it hard to imagine a future that goes well where AI doesn't value all sentient life. I think this is a great first crack at the problem and something I think needs to be tried.

Conflicts I'm on the board of CaML and I helped Miles write this grant write-up

I'd like to see a lot more coverage of AI in journalism and Tarbell seems like the people who've done the best/most of this job so far.

Independently, a couple people have said they are fans of Tarbell's reporters which I think is a good sign and a sign that they should get continued funding. Marius' grant below is a great sign in my opinion.

On the negative side, I somewhat worry that Tarbell wants to grow extremely quickly and they haven't been that cost effective in doing so though I haven't done a lot of calculations to support this.

I'm also not a fan of trying very hard for non-OP dollars and not a fan of matching campaigns. I think these are both a sort of tragedy of the commons and the EA community should try not to succumb to these wishes.

Overall, I'm a fan and so making a donation. I've spoken to Cillian for ~1.5 hours

I spoke with Fred for ~1 hr, and we talked about what he has done so far and what he wants to do. I am most excited about him taking this work to train a model to detect scam emails and eventually get this into email providers and/or test how many scam emails people click on with and without this kind of plugin to try to measure some reductions in scams.

A lot of the work needed to make AI go well is going to take the form of looking at real world harms of AIs and mitigating them to make AI go well for humanity.

Hopefully someone else can come in and fill the rest of the grant here.

I would also be particularly excited if Fred were to commercialize this and/or sell scam filters to email providers to prevent the billions of dollars per year inflicting people.

It's hard to describe the state of free speech in the UK as anything short of shocking. ~10000 people per year are arrested for non-violent speech "crimes" and so this isn't merely accidental or happenstance. Police forces across the country have teams that monitor social media posts. Private messages are being intruded upon. Simply expressing support for groups is leading to arrests.

It's hard to know what is good to do to set humanity on a good longterm trajectory but I think free speech norms are of particular importance for figuring out good values and allowing humanity to come to good values. The UK has historically been an important country for shaping precedent and still has an outsized influence

Sam and Co are running a fairly lean operation and working hard on something that seems time sensitive and tractable and are working to make free speech issues non-partisan. I spoke to Sam for ~2 hours on this and he far surpassed many of my informal "tests". He seems reasonable, committed, honest and principled. I have furthermore read some of his writings and they are good. I further support not working on forecasting research and doing something tractable and tangible instead.

@ I want to quickly note that it's a bit unfair for me to specifically only call you out on this or rather, that this is a thing I find with many AI safety projects. It just came up high on Manifund when I logged on for other reasons and saw donations from people I respect.

That said, no, this doesn't exactly assuage my concerns/make me want to donate here. I don't think this work really takes that much time but more importantly, I just don't think it passes the cost-effectiveness bar that I hold my donations and my regrants to given the other opportunities I know of that are out there.

I don't think people doing "non-profit" work should have to make some kind of bare minimum amount to barely cover expenses but I do think that $100k/year to repost short clips on tiktok shows that spending habits in EA have gotten a bit out of control and we should do a bit of a sanity check. I also don't think that the correct metric here is that it's reasonable to pay the equivalent salary of an ML engineer in France. I think people usually/often inflate their earning potential but more importantly, I just don't think it's reasonable for non-profit work to get paid your maximum earning potential.

I think there are two relevant cost-effectiveness factors to consider in grantmaking: raw impact per dollar, and also the reasonable cost to reproduce. I feel uncertain on the former here (and I'm going to be doing a project to address this), but on the latter, I feel pretty confident that I could get 4-5x this impact by having low-cost EAs in 2nd/3rd world countries or college students doing this work.

Based on what you wrote, I think you might feel undercompensated for previous work (on the SB 1047 documentary) and thus want to recoup some of that on this work but I don't think that is what philanthropic funders should do here. Perhaps I'm somewhat open to retroactive funding for the documentary if I/others investigated but I don't think it's good to

---

Grow An AI Safety Tiktok Channel To Reach Ten Million People, Aug 2025

@Austin, I also like that he's being responsive. I also think many different POVs can be reasonable. I did not mean for my comments to tell people that they shouldn't in any case donate/give money here. I'm mainly adding my take here. My only point is that money is limited/has opportunity costs and I was mainly explaining why I chose not to.

We can discuss elsewhere on appropriate EA salaries.

@Michael sure, I agree I would need to pay for this warm up period and for them to scale up. I disagree though that this is highly skilled work that couldn't be picked up. I think your average EA who is interested in the topic could do this. I didn't mean my comments to say "you should return this money". Lots of grants/spending in EA ecosystems I consider to be wasteful, ineffective etc. And again, apologies for singling you out on a gripe I have with EA funding.

This has been on myradar for a while now. Communication to the masses about AI seems like the most high value neglected thing to me. This is a fairly expensive project but I think can move this conversation that is typically held on LessWrong to the dinner table.

**Main points in favor of this grant**

• We just need to know this or have some idea of it (continuous work should be done here almost certainly). Hard to believe nobody has done this yet.
• Came strongly recommended from Peter Wildeford who I very much like. He preferred this to a different project I was also evaluating.
• Lots of spill on effects that will be valuable
• People involved seem qualified and have good track records doing this work

**Donor's main reservations**

• I don't know these people. It comes somewhat second hand due to a few recommendations from people I trust.
• I worry these results will be used too conclusively and not updated much. As in, people will have too much confidence in the results of this outcome
• Nothing could come of this but this isn't a big concern to me.

**Process for deciding amount**

I threw the rest of my balance into this. I didn't know exactly how to balance my Manifund budget over the few months. I didn't know how much more I would be getting if at all. This was a grant I was deciding on with a few other candidates. Once I decided this was the highest EV, I threw my entire remaining balance. Don't look into the weird number.

**Conflicts of interest**

Nothing to disclose.

Main points in favor of this grant

• Neel Nanda's top choice in the Nonlinear Network. Neel says many people want to hire him.
• Joseph is an official maintainer of TransformerLens (the top package for mech interp).
• Teaches at the ARENA program.
• Two really good posts on Decision Transformer Interpretability and Mech Interp Analysis of GridWorld Agent-Simulator.
• Work was listed by Anthropic May 2023 update
• Working on trajectory transfers is a natural progression from decision transformers

Donor's main reservations

I wonder if he is best to be hired by some other alignment team instead since I think he might be young as he might have better mentorship with others.

Process for deciding amount

This just should be fully funded, at least to $110,000. $25000 (but ideally $50000 would put him at ease for 6 months by which time he expects to have enough output to justify further funding. I'd give more but I have a limited budget. This is already half of my budget but I feel quite strongly about this.

Conflicts of interest

Nothing to disclose.

---

Joseph Bloom - Independent AI Safety Research, Jul 2023

I think with his track record so far and endorsements, he's earned the right to go the direction he thinks is best. Maybe it'd be better to have an org that "houses" a bunch of people that just want to work by themselves and the org just formally employs them and helps them raise funds for their project and maybe has some communal resources but I don't think I'd prefer to fund that org vs. fund someone who is just going to do good direct work.

---

Joseph Bloom - Independent AI Safety Research, Dec 2023

Want to update that Joseph has been crushing it. He's made good research progress, updates Dylan and I every month and receives good feedback. Super happy with this grant.

I have also let Joseph know that if he is in need for more funding, he should contact me and I will make sure it happens.

10/10

**Main points in favor of this grant**

• I think there is merit in developing a breadth of interpretability approaches. If Singular Learning Theory ends up having merit, there is a wealth of knowledge from physics and chemistry that directly applies.
• The bang for your buck with researchers in the project is really good compared to most.
• I am excited about bringing a talented researcher in Daniel Murfet into the field.
• I am excited about this project being a launching ground for a potential future for profit or non profit if this goes well
• I think Jesse is great at explaining his ideas and communicating his reasoning. I think Stan is very smart and worth supporting.

**Donor's main reservations**

• I don’t know about the merits of SLT and I’m not equipped to judge.
• I don’t know the direction they will take things if they are successful and it doesn’t seem like it was yet thought through.

**Process for deciding amount**

I need to save some money for 2 more grants I would like to make but I want to show significant support for this project and gets them across the $125k threshold.

**Conflicts of interest**

I might potentially hire Stan as a consultant for my business. Funding this probably works against me but I felt this was important to disclose.

---

Scoping Developmental Interpretability, Dec 2023

If I am not mistaken, Jesse + team have received additional funding from SFF now of $500k to expand. This puts it off the table for me for further Manifund grants for now but will check in with Jesse/Stan sometime in the new year for where they are at.

Main points in favor of this grant

When I talked with Lisa, she was clearly able to articulate why the project is a good idea. Often people struggle to do this

Lisa is smart and talented and wants to be expanding her impact by leading projects. This seems well worth supporting.

Davidad is fairly well-known to be very insightful and proposed the project before seeing the original results.

Reviewers from Nonlinear Network gave great feedback on funding Lisa for two projects she proposed. She was most excited about this one and, with rare exceptions, when a person has two potential projects, they should do the one they are most excited about.

I think we need to get more tools in our arsenal for attacking alignment. With some early promising results, it seems very good to build out activation vector steering.

Donor's main reservations

I don’t feel I’m a good judge of whether or not this is worth doing. I think I judge talent well, but I don't have nearly enough alignment background or neurotech background to judge this. This is far more of a bet on the people than on the project. I also don't think many people would be qualified to judge the project.

It's expensive.

I somewhat worry that Lisa won't be full-time on the project and/or that this might distract her from her other work. She did say she had broad support from her current workplace to pursue this in tandem.

Process for deciding amount

The project is in discussion with Foresight to see if it's possible to do a scaled-down version that isn't as expensive. My $15k should go towards getting the ball rolling with the expectation of a few more people to get this at least to the scaled-down stage but preferably the full proposed project.

Conflicts of interest

None

---

Activation vector steering with BCI, Dec 2023

I reached out to Lisa for a progress update on this:

-There really is an important minimum funding to make this project viable that hasn't been achieved yet and so they haven't started. Money is sitting in Lisa's account.

-She will talk with Davidad soon and decide if they are going to make a push for more funding, pivot to something else that could be lower cost or return the funds (expecting by end of March)

I like the honesty here and hope we can get Lisa funded to do a large project (whether this one of another one on areas she is interested in). I'm very "bullish" on Lisa still. I think she's someone that can/will just make something happen that should happen without needing too much permission. I think she also has a rather unique and needed blend of technical understanding + ability to do all the little organizing things that are needed for something to get off the ground.

About time I wrote up my reasoning for this grant:

Main points in favor of this grant

Kunvar has set out an ambitious research agenda for himself which would lead to some quite exciting papers. This donation also will get a lot of Mech Interp per dollar spent if successful (like 10x more per dollar than most other stuff I come across given the little compute and low overhead)

The main reason for funding this though is that I know Kunvar is very smart due to many interactions I have with him and I doubt this will be as apparent to other people so I am pretty uniquely able to make this grant.

**Donor's main reservations**

Independent alignment work without any mentorship doesn't have a fantastic track record. That said, I am expecting this to launch Kunvar into something like MATS or another program to get him some mentorship on this and he has mentioned collaborating on these projects with others.

**Process for deciding amount**

This was the amount needed to cover the compute and the minimum salary (which was super low). If more were needed, I could make that happen but this just gets the project off the ground.

**Conflicts of interest**

None.

This is very interesting. Want to set up a call and talk about it? Ill take notes (and maybe publish those with your permission)

---

Exploring novel research directions in prosaic AI alignment, Dec 2023

I really liked this when I saw it and I flagged it as something I needed to look back at. Now that I have looked into it more in depth and see Neel's endorsement (which has worked out fantastically for my grant to Joseph Bloom), as well as Evan's endorsement, I finished off the required funding. I also met Lawrence briefly and was impressed by him but this is minor.

I also think his reasoning for doing this work is great.

Main points in favor of this grant

Forecasting conditional policy effects and forecasting important questions are among the best use of forecasters in my opinion.

It's good to see forecasters work be used for something actionable.

I want to encourage forecasters to do more of this kind of work.

I'm not aware of anyone else trying to quantify impacts of different AI policies which seems important.

Tolga seems quite bright and is a top superforecaster.

The grant is just very cheap compared to what it could accomplish.

I am hopeful that if successful, when Tolga graduates, he will be doing a lot more of this kind of work. Making forecasts useful for AI Policy.

Donor's main reservations

I think there is an order of magnitude too much funding going towards forecasting. I am worried about contributing to this. However, this isn't the type of forecasting grants I am against (this is mainly just blind funding forecasting platforms).

I'm skeptical how much forecasting work can really be done in areas like this vs. just guessing and putting numbers on things to make them seem more credible when that number can just be fairly meaningless.

Process for deciding amount

The amount Tolga wanted for this was very reasonable. Everyone is working on below market rate salaries or for free. I didn't want there to be a tradeoff between compensating previous forecasting work vs. ensuring there was some funding for the next forecasting steps.

Conflicts of interest

None.

I think this is a decent giving opportunity. Will consider if I have left over. Would broadly support this over funding independent researchers. This $1400/month covers housing and work space and some basic services (and maybe food?) for someone in London? Seems very good.

Let me know how this goes and if there is plans for an add on

I think Apollo is going to be very hard to beat for a few reasons and I would have written up the grant if Marius didn't.

• They are a very very talented team. I think several people involved could be working at top AI labs.
• They are focused on a tractable and "most important problem" of deception in AI systems.
• They seem well positioned to be a potential centre for an AI safety org in Europe
• They could grow to absorb a lot more funding, effectively. A key barrier to EA funds right now is productively spending lots of money. Seems possible that they could absorb >20M/year in 4 years time
• I think the people here are very "alignment-pilled"
• I think I am a good judge of talent/character and Marius passes the bar.
• I have a preference that they become a non-profit vs for-profit company that sells auditing and red-teaming. I think funding in the first few years will be pivotal for this.

Reservation:

• They seem quite expensive on a per person basis compared to other projects Manifund has been funding. That said, there is going to be a lot of "bang per person here". Marius explained to me that they are competing with large AI labs and are already a significant salary cut. I would much rather see someone working within Apollo than doing independent research. I think we should be quite weary of funding independent researchers until orgs like Apollo are fully funded, with rare exceptions.
• I still don't get why SFF and OP can't just fully fund them. My best guess here is that they are terrified of seeding another "AI accelerator" or wanting to save their cash for other things and thus allow others to donate in their place.
• I am slightly worried that there isn't a good way to coordinate across the community of "how much they should receive". From the inside, Apollo wants to raise as much as possible. I don't think that is optimal at a movement level and this leads to them spending a lot of time fundraising so they can have more money. There probably is a funding level that Apollo shouldn't exceed in it's first year though I don't know what that number is.

I would bet that if we reviewed Apollo 1.5-2 years down the line, it will outperform a majority of grants on a per dollar basis (very hard to operationalize this though).

I saw this a couple weeks ago and thought it was a really good idea. Main reason I didn't fund it is because there are things I want to do more. Would be a real shame if the EA community couldn't step in and do this.