Thomas Larsen

I think this is very promising. This team seems to have some of the people who have the clearest thoughts in the world about acausal interactions. I've asked several people who I trust a lot in this space and gotten universally positive references about the team.

My main concern is that thinking about acausal interactions is extremely difficult (meaning that zero progress is somewhat likely) and sign uncertain (so, even if they did make progress, it's not clear this would be net helpful). Overall, my view is that it still seems good to have some people working on this, and I trust this team in particular to be thoughtful about the tradeoffs.

Also, this type of work doesn't get funded by OP.

---

Acausal research and interventions, Jan 2026

Fewer people have donated to this than I thought, so I am donating my remaining regranting budget here.

I'm regranting 40k to this on the basis that I think there are few people thinking seriously about how to solve AI alignment and Thane seems to be one of them, and I have been impressed by a bunch of Thane's LW posts.

I think that my median case view is that this doesn't end up mattering because this agenda seems very hard and unlikely to pan out. Nevertheless, I think it's big if true, and relatively cheap to fund, and pretty neglected (relative to empirical ML stuff) at the current margin. And I'd guess that this is the type of grant that's relatively less legible to most grantmakers, and so I wouldn't expect it to get funded via default sources (or for it to be dramatically underfunded).

I think it'd be pretty silly if Thane (and others like them) don't move to the US and collaborate in person with people that would speed them up a lot due to funding bottlenecks.

I'm donating at least $10k (via Manifund).

I think Lightcone has been hugely impactful (see the above, which I think is very compelling evidence of a large amount of impact), and can't get funded from many of the usual places.

COI Note: Lightcone contracted with AI Futures Project (the org I work for) on AI 2027, and is continuing to contract with us on followup work. This is a donation for general support, and not anything like payment for services received.

I'm donating $10k, and I think there's a good chance I'll come back and end up donating more.

I think that AI Digest has done great work and will be able to put marginal funding to good use. Stuff I'm particularly excited about in the past:

• The Agent Village. I think getting real world experience seeing how AI agents will interact with tasks outside the narrow confines of a benchmark suite is a neglected form of capability evaluation.
• Various explainers, especially the one of the METR graph.

Main reservations:

• Nothing jumps out to me.

Conflicts of Interest:

No COIs.

This is a team which started in SERI MATS applying for funding to continue their SERI MATS project on research checking for dishonesty in advanced AI systems.

My cruxes for this type of grant are:

(1) If done successfully, would this project help with alignment?

(2) How likely is this team to be successful?

My thoughts on (1):

This is meant to build upon Burns’ et al.'s Discovering Latent Knowledge paper (DLK), which finds a direction in activation space that is supposed to represent the 'truth' of a logical proposition.

I think that Eliciting Latent Knowledge (ELK) is an important subproblem of alignment, and I think it can be directly applied to combat deceptive alignment. My independent impression is that this specific direction towards solving ELK is not very useful towards a full alignment solution, but that it may lead to slightly better monitoring. (In particular, I think even in a good outcome, this will only lead to an average case solution to ELK, meaning that when we explicitly train against this detector, it will fail.) I expect that AGI projects will be in a position where it's obvious that the systems they are building are capable and dangerous, and it will be apparent that instrumental incentives kick in for e.g. powerseeking and deception. I think that this technique might help us detect this danger, but given that we can't train against it, it doesn't let us actually fix the underlying problem. Thus, the lab will be in the difficult position of continuing on, or having to train against their detection system. I still think that incremental progress on detecting deception is good, because it can help push for a stop in capabilities growth before prematurely continuing to AGI.

My thoughts on (2):

They produced reasonable output during SERI MATS, including the beginning of a replication of the DLK paper. They weren't that specific in their grant application, but they wrote a number of ideas for ways to extend the paper in the LW post. The two ideas that seem best to me are:

• Connecting DLK to mechanistic interpretability. This seems hard, but maybe tinkering around in activation space can be helpful.
• Creating a better confidence loss. In the original paper, only one statement was considered, and so the loss was coming from the constraint that P(q) + P(not q) = 1. They propose evaluating two propositions p & q, and getting more constraints from that.

These ideas don't seem amazing, but they seem like reasonable things to try. I expect that the majority of the benefit will come from staring at the model internals and the results of the techniques and then iterating. I hope that this process will churn out more and better ideas.

One reservation I have is that none of the applicants have an established research track record, though they have published several papers:

• Kaarel's Arxiv page

• Walter's Google Scholar Profile

• [Georgios's ORCID](https:// https://orcid.org/0000-0003-3854-3150)

This team did get strong references from Colin Burns and John Wentworth, which makes me a lot more excited about the project. All things considered, I'm excited about giving this team a chance to work on this project, and see how they are doing. I'm also generally enthusiastic about teams trying their hand at alignment research.

Joseph applied for independent research funding to continue his research into decision transformer interpretability. I'm happy about Joseph's initial result, which found circuits in a decision transformer in a simple RL environment. I thought the applicant's write up was solid and gave me some updates on what cognitive machinery I expect to be induced by RL. In particular, I was excited about the preference directions in embedding space that they constructed. This seems like a useful initial step for retargeting the search, though more understanding of the circuits that are doing the optimization seems critical for this approach.

I think interpretability on RL models is pretty neglected and very relevant for safety.

According to a reference, the applicant was also in the top 3 ARENA participants, and was very motivated and agentic.

The counterfactual is that Joseph tries to get funding elsewhere, and if that fails, getting a research engineer job at an AI safety org (e.g. Redwood, Conjecture, Ought, etc). I encouraged this person to apply to the AI safety orgs, as I think that working at an org is generally more productive than independent research. These jobs are quite competitive, so it's likely that Joseph won't get hired by any of them, and in this case, it seems great to pay him to do independent alignment research.

Overall, I think that Joseph is a promising researcher, and is working on a useful direction, so I feel excited about supporting this.

Since receiving this grant, Joseph has received some more funding (here), and was mentioned in the Anthropic May Update.