individual · LinkedIn · Manifund
Technical AI safety researchAI safety
| Long-Term Future Fund | $67,000 | 85% | 1 |
| Neel Nanda | $11,000 | 14% | 1 |
| BlueDot Impact | $1,000 | 1% | 1 |
Neel Nanda · Manifund · full post · Nov 19, 2024
Mechanistic Interpretability research for unfaithful chain-of-thought (1 month), Nov 2024
Read more ▾Show less ▴I'm fully funding this. I think that understanding, detecting and potentially mitigating chain of thought unfaithfulness is a very important problem, especially with the rise of o1 models. I think the approach taken here is reasonable. I think Arthur is fairly good at supervising projects, and that under him Jett and Ivan have a decent shot of making progress, and that enabling this to start a month earlier is clearly a good idea. I know Arthur well, and got a moderate amount of data on Jett and Ivan's ability to execute on this project in my MATS training program.
Conflicts of interest: Jett and Ivan were in my MATS training program (but no longer are, and I would not be involved in this project going forwards). I manage Arthur and he works on my interpretability team at GDM (but he is doing MATS in a purely personal capacity, and if anything I'm incentivised to *reduce *his amount of extra-curricular distractions :P ). Overall I don't feel particularly concerned about conflicts of interest here.
| Date ↓ | Funder | Via | Amount | Cause | Source | Purpose |
|---|---|---|---|---|---|---|
| Jun 2026 | BlueDot Impact | $1,000 | AI safety | bluedot | Directing an MSc thesis on how unverbalized biases enter LLM's chain-of-thoughts: finding the 'bias anchor' sentences and the attention path from the model's input to its output. | |
| Nov 19, 2024 | Neel Nanda | Manifund | $11,000 | Interpretability | manifund | Mechanistic Interpretability research for unfaithful chain-of-thought (1 month) |
| Jan 2024 | Long-Term Future Fund | EA Funds | $67,000 | Interpretability | ea_funds | 4-month stipend for MATS extension on mechanistic interpretability benchmark + 2-month stipend for career switch |