Dan Hendrycks

I'll likely regrant to this project because I think CAIS is great, but I'll first look for projects where my grants funge less with Open Phil, SFF, Longview, LTFF, etc.

A competition for wellbeing trackers / hedonometers.

Update on this Grant (September 2022): Dan Hendrycks ended up not taking the grant to work on this project.

We’re funding Dan Hendrycks to run a competition to build a better hedonometer, in the spirit of projects like these two.

Current hedonometers are clearly quite bad—they track things like Twitter sentiment, which is a very weak proxy for actual general happiness. GDP is probably our best current proxy for the happiness of a population, but is also clearly flawed in many ways.

That being said, I think my inside-view case for this project is pretty weak, as I don’t fully understand what would make a better hedonometer useful. The best positive case I could come up with after talking with Dan was just that building a better hedonometer might teach us a lot about how to measure happiness, which could be useful at some point down the road for aligning AI systems or setting public policy.

However, I think the downside risk is pretty minimal; it mostly comes from the lost value of the time people spend on this competition rather than other projects. I hope that people won’t work on it unless they actually feel like they see the project as useful and think it’s worth their time. Overall, I’m willing to defer to Dan here—since I think he has a pretty solid track record—and fund this competition despite not having a strong positive case for it.

A human value modeling benchmark

Dan Hendrycks is a second-year AI PhD student at UC Berkeley, advised by Dawn Song and Jacob Steinhardt. This is a restricted grant to support creation of a benchmark for NLP model's predictive power for human models. In particular, it supports paying contractors via platforms such as Mechanical Turk to generate and validate question-answer pairs.

I am generally excited about benchmarks as a route for progress on AI safety, especially for focusing the attention of the AI research community. Their impact is heavy-tailed, with many benchmarks seeing little adoption while others being extremely influential, so this is definitely a "hits-based" grant. However, Dan does have a strong track record of creating several benchmarks in the field of robust machine learning which makes me optimistic.

The topic, testing whether NLP models implicitly capture notions of human morality, is very novel. Language is a natural channel by which to express preferences, especially over more abstract concepts, so it seems important that NLP models can represent preferences. It is common in the field to train unsupervised language models on large corpora and then fine-tune for specific applications. So testing whether preexisting models have already learned some information about preferences is a natural starting point.

A key concern regarding this project is that being able to predict human moral judgements in text does not directly translate to better aligned AI systems. We expect most of the project's impact to come from gaining a better qualitative understanding of language models deficiencies, and from increased attention on human values in general in the NLP community. However, there is some risk that the limitations of the benchmark are not sufficiently recognized, and the NLP community wrongly believes that value learning is "solved".

Main points in favor of this grant

• The premise and theory of change is strong. There’s a need for such a platform.
• The team has journalism and editorial experience.
• CAIS has strong networks with academics and researchers; the project has already seen interest from a variety of external contributors.

Donor's main reservations

• Sustaining a high rate of high-quality contributions may be challenging.
• May require significant marketing to build readership and visibility.

Process for deciding amount

I reviewed a budget.

Conflicts of interest

I will be advising this project.

Main points in favor of this grant

Removing hazardous capabilities from models would greatly help reduce AI x-risk from malicious use and unilateral actors. Alex is a researcher with a strong track record who is interested in AI safety and has done previous AI safety research. The timing is right; NIST has been tasked by the recent EO to help develop standards and regulations on “dual-use foundation models.” Research now has a much higher likelihood of helping shape regulation.

Donor's main reservations

This is a relatively complex project with many moving parts. It’s crucial that the project is executed well on a relatively short timeline.

Process for deciding amount

It was estimated by the researchers that this was the total amount needed for the dataset. I have reviewed the budget and approved.

Conflicts of interest

I will be helping advise this project.