John Wentworth

• I have been consistently impressed with John Wentworth’s AI safety work, just as I have been when we funded him in the past. Though this grant is more open-ended than previous grants we’ve made to John, I think John is an experienced enough AI safety researcher that general, open-ended research is something I am absolutely excited about him doing.
• Note: We recommended this grant to a private funder, rather than funding it through LTFF donations. At the time, we believed that the general nature of the grant might include work outside of the scope of what we are able to fund as a charitable organization, but we intend to make similar grants through EA Funds going forward.

Developing tools to test the [natural abstraction hypothesis](https://www.lesswrong.com/posts/cy3BhHrGinZCp3LXE/testing-the-natural-abstraction-hypothesis-project-intro).

John Wentworth is an independent AI safety researcher who has published a large number of articles on the AI Alignment Forum, primarily focusing on agent foundations and specifically the problem of understanding abstractions.

John’s current work, which he applied for the grant to work on, is in testing what John calls the “natural abstraction hypothesis.” This work builds directly on my all-time favorite post of John’s, his “Alignment By Default,” which makes the case that there is a non-negligible chance that the abstractions/proxies that humans use are natural enough properties of the world that any trained model would likely use similar abstractions/proxies as well, making such a model aligned effectively “by default”.

According to my inside-view model of AI safety, I think this work is very exciting. I think that understanding abstractions in general is likely to be quite helpful for being able to better understand how models work internally. In particular, I think that the natural abstraction hypothesis is a very exciting thing to try and understand, in that I expect doing so to give us a good deal of information about how models are likely to use abstractions. Additionally, the truth or falsity of the general alignment by default scenario is highly relevant to general AI safety strategy. Though I don’t expect John’s analysis to actually update me that much on this specific question, I do think the relevance suggests that his work is pointed in the right direction.

Regardless, I would have supported funding John even if I didn’t believe his current work was very inside-view exciting, simply because I think John has done a lot of good work in the past—e.g. his original “Alignment By Default” post, or any number of other posts of his that I’ve read and thought were quite good—and I think it’s important to give researchers who’ve demonstrated the ability to do good work in the past a sort of tenure, so they feel comfortable working on things that they think are exciting even if others do not. Additionally, I think the outside-view case for John is quite strong, as all of me, Scott Garrabrant, and Abram Demski are all very positive on John’s work and excited about it continuing, with MIRI researchers seeming like a good reference class for determining the goodness of agent foundations work.

Building a theory of abstraction for embedded agency using real-world systems for a tight feedback loop.

John participated in the recent MIRI Summer Fellows Program, where he proposed some research directions to other MIRI researchers that they were excited about. As well as receiving multiple strong testimonials from various AI alignment researchers, he has also been very actively posting his ideas to the AI Alignment Forum, where he has received substantive engagement and positive comments from several top researchers; this is one of the main reasons for this grant.