Joshua Clymer

The Long-Term Future Fund provided a $1,500 grant to Joshua Clymer for compute funds to rent A100 GPUs for experiments on instruction-following generalization. Although Clymer had previously worked on AI safety field-building and communications, this was his first technical AI safety project.

The fund was interested in whether models trained within a specific distribution can generalize "correctly" out of distribution, such as on benchmarks like TruthfulQA, instead of learning the idiosyncrasies of human evaluators. More broadly, the fund wanted to ensure that models can faithfully follow human instructions even when operating in vastly different contexts from their training data. The grantee believed that training models on general instruction-following could be a plausible approach to aligning AI systems with insufficiently specified rewards.

The proposal employed a method called "sandwiching," where the model is trained on data from a non-expert who cannot evaluate the model's performance, but the later evaluation is conducted by an expert. We were excited about the grant as it was cheap and tractable, while addressing an obvious difficulty in AI alignment, making it an attractive funding opportunity.

*Outcomes:* The paper and benchmark have been published, with the authors finding that reward models do not inherently learn to evaluate 'instruction-following' and instead favor personas that resemble internet text. In other words, models trained to follow instructions on easy tasks do not naturally follow instructions on hard tasks, even when we are fairly confident the model has the knowledge to answer the questions correctly. You can also check out Joshua’s thoughts on the project on the alignment forum.