Trace

UK AI Security Institute

www.aisi.gov.uk · government · AI Safety Institute, UK AI Safety Institute

The UK government's AI Security Institute, which tests frontier models and does research on AI safety and security.

Leadership

  • Henry de Zoete · Director
  • Jade Leung · Chief Technology Officer
  • Nate Burnikell · Chief Strategy Officer
  • Chris Summerfield · Research Director

Team

100 people

source, Oct 2026

Total granted
$35,608,000
Grants
77
Average size
$468,526
Coverage
100%
itemized by recipient
Years active
2025–26

Grants over time

$0$10M$20M$30M$40M20252026
Technical AI safety researchAI safetyAI governance, policy, and advocacy

Biggest cause areas

Technical AI safety research$32,717,25092%56
AI safety$2,596,0007%20
AI governance, policy, and advocacy$294,7501%1

Biggest recipients

Anti Entropy$1,768,5005%3
Zhijing Jin$1,179,0003%2
Vinod Vaikuntanathan$1,179,0003%2
Kai Sandbrink$589,5002%1
Max Duchi$589,5002%1
Alexandros Psomas$589,5002%1
John Duchi$589,5002%1
Simplex$589,5002%1

Grants made

Date ↓RecipientViaAmountCauseSourcePurpose
Feb 19, 2026Kai SandbrinkUniversity College London, University of Oxford~$589,500*Technical AI safety researchuk_aisiAlignment Training Dynamics — led by Kai Sandbrink. — Alignment Project
Feb 19, 2026Max Duchi~$589,500*Technical AI safety researchuk_aisiDevelop and leverage training dynamics theory for AI alignment through early detection of phase transitions and stagewise learning dynamics — led by Max Duchi. — Alignment Project
Feb 19, 2026Alexandros PsomasPurdue University~$589,500*Technical AI safety researchuk_aisiBeyond Nash: Pseudo-Equilibria for Robust AI Alignment — led by Alexandros Psomas. — Alignment Project
Feb 19, 2026John DuchiStanford University~$589,500*Technical AI safety researchuk_aisi(Mis)aligned preferences: what data is needed for agents to reflect honest human preferences — led by John Duchi. — Alignment Project
Feb 19, 2026Anti Entropy~$589,500*Technical AI safety researchuk_aisiSynthetic benchmarks enabling controlled experiments on how argument types and evidence structures affect judge accuracy in debate-based AI oversight — led by Gabriel Recchia. — Alignment Project
Feb 19, 2026Simplex~$589,500*Interpretabilityuk_aisiSimplex: A Principled Science of Interpretability — led by Adam Shai. — Alignment Project
Feb 19, 2026ELLIS Institute Tübingen~$589,500*Technical AI safety researchuk_aisiFoundations of Situational Awareness Evaluation and Control — led by Sahar Abdelnabi. — Alignment Project
Feb 19, 2026Andrew DuncanImperial College London~$589,500*Technical AI safety researchuk_aisiPRISM: Probabilistic Rare-event Inference for Safety of Models — led by Andrew Duncan. — Alignment Project
Feb 19, 2026Bin YuUniversity of California, Berkeley~$589,500*Technical AI safety researchuk_aisiThe Geometry of Risk: Using Model Complexity to Improve Predictions of Catastrophic AI Failures — led by Bin Yu. — Alignment Project
Feb 19, 2026Pavel IzmailovNew York University~$589,500*Technical AI safety researchuk_aisiWeak-to-Strong Consistency Reinforcement Learning — led by Pavel Izmailov. — Alignment Project
Feb 19, 2026Rocco Servedio, Xi Chen, and Anindya DeColumbia University~$589,500*Technical AI safety researchuk_aisiDiscovering rare harmful behaviors exhibited by high-dimensional AI systems — led by Rocco Servedio, Xi Chen, Anindya De. — Alignment Project
Feb 19, 2026Yale University and University of Michigan~$589,500*Technical AI safety researchuk_aisiIs the AI Misaligned or Misinformed? Applying and further developing the strategic communication framework to study how a user can productively interact with an AI agent who may be misinformed or may have conflicting interests with the user — led by Yang Cai, Nima Haghpanah, Nicole Immorlica, Elliot Liponowski, and Doron Ravid. — Alignment Project
Feb 19, 2026Rico AngellNew York University~$589,500*Technical AI safety researchuk_aisiWhat Can Mr. Hyde Tell Us About Dr. Jekyll? Efficiently Evaluating and Red-teaming "Safe" Models with Unsafe Alter Egos — led by Rico Angell. — Alignment Project
Feb 19, 2026Yonatan KahnUniversity of Toronto~$589,500*Technical AI safety researchuk_aisiScaling laws, data distributions, and learning dynamics: simulated high-energy physics data as a benchmark for data in the wild — led by Yonatan Kahn, Noam Levi, Andrew Larkoski. — Alignment Project
Feb 19, 2026Ryan WilliamsMassachusetts Institute of Technology~$589,500*Technical AI safety researchuk_aisiExploring The Role of Relativization and Problem Extensions in AI Safety — led by Ryan Williams. — Alignment Project
Feb 19, 2026Rubi HudsonPrinciples of Intelligence~$589,500*Technical AI safety researchuk_aisiConstructing robust incentivizes for AI agents to act cautiously — led by Rubi Hudson. — Alignment Project
Feb 19, 2026Alignment Research Center~$589,500*Technical AI safety researchuk_aisiLow probability estimation: mechanistically estimating the probability of rare failures in machine learning systems — led by Jacob Hilton. — Alignment Project
Feb 19, 2026Aditi RaghunathanCarnegie Mellon University~$589,500*Technical AI safety researchuk_aisiEnhanced Cyber Control Evaluations: An Empirical and Learning-Theoretic Approach — led by Aditi Raghunathan. — Alignment Project
Feb 19, 2026Annie LiangNorthwestern University~$589,500*Technical AI safety researchuk_aisiFriend or Foe: Delegating to an AI whose Alignment is Unknown — led by Annie Liang. — Alignment Project
Feb 19, 2026Daniele CondorelliUniversity of Warwick~$589,500*Technical AI safety researchuk_aisiHow does AI learn to play games? Theory and experiments — led by Daniele Condorelli. — Alignment Project
Feb 19, 2026Dirk BergemannYale University~$589,500*Technical AI safety researchuk_aisiUnified framework for analyzing how AI systems might be jointly aligned and controlled via the language of mechanism and information design — led by Dirk Bergemann, Stephen Morris. — Alignment Project
Feb 19, 2026Flourishing Future Foundation~$589,500*Technical AI safety researchuk_aisiWhen Models Refuse to Be Steered: Understanding Self-Monitoring and Alignment Drives — led by Diogo De Lucena. — Alignment Project
Feb 19, 2026Alexander Gietelink Oldenziel~$589,500*Technical AI safety researchuk_aisiLoss Landscape and Training Dynamics for (ReLU) Neural Networks — led by Alexander Gietelink Oldenziel. — Alignment Project
Feb 19, 2026Anti Entropy~$589,500*Technical AI safety researchuk_aisiMeasuring how debate assistance, annotator expertise, and time constraints affect human feedback accuracy, including the effect of aggregation methods and additional judges — led by Gabriel Recchia. — Alignment Project
Feb 19, 2026Risi KondorUniversity of Chicago~$589,500*Technical AI safety researchuk_aisiProbabilistic Agentic Foundations for Mesa-Optimization and Emergent Misalignment: formalising how hidden objectives emerge within agents and designing strategies to mitigate them — led by Risi Kondor. — Alignment Project
Feb 19, 2026Rahul SanthanamUniversity of Oxford~$589,500*Technical AI safety researchuk_aisiA Theoretical Framework for Machine Understanding — led by Rahul Santhanam. — Alignment Project
Feb 19, 2026Susan WeiMonash University~$589,500*Interpretabilityuk_aisiProbabilistic Interpretability of Transformers — led by Susan Wei. — Alignment Project
Feb 19, 2026London School of Economics and Political Science~$589,500*Technical AI safety researchuk_aisiCompetition and the AI Control Problem: Mechanism Design with Opaque Agents — led by Matthew Levy. — Alignment Project
Feb 19, 2026Zhijing JinUniversity of Toronto~$589,500*Technical AI safety researchuk_aisiFormal verification and Empirical Testing of Translucency in Multi-Agent AI Systems — led by Zhijing Jin. — Alignment Project
Feb 19, 2026Xiaowei HuangUniversity of Liverpool~$589,500*Technical AI safety researchuk_aisiRare-Event Estimation in Large Language Models via Subset Simulation and Contrastive Unalignment — led by Xiaowei Huang. — Alignment Project
Feb 19, 2026Arcadia Impact~$589,500*Alignment methodsuk_aisiBuilding alignment research capacity in collaboration with AISI alignment team including initial focus areas: scalable oversight and learning theory — led by Erin Robertson. — Alignment Project
Feb 19, 2026Venkatesan GuruswamiUniversity of California, Berkeley~$589,500*Technical AI safety researchuk_aisiError-correction and Pseudorandomness in Alignment: Watermarks and Backdoors — led by Venkatesan Guruswami. — Alignment Project
Feb 19, 2026Carmine VentreKing's College London~$589,500*Technical AI safety researchuk_aisiThe Implementability-Capability Frontier: Mechanism Design for Robust AI Alignment — led by Carmine Ventre. — Alignment Project
Feb 19, 2026Center for AI Safety, Inc.~$589,500*Technical AI safety researchuk_aisiSteering Models Towards Safer Reasoning — led by Austin Meek. — Alignment Project
Feb 19, 2026LawZero~$589,500*Technical AI safety researchuk_aisiDesigning a safety case and latent variable model for the non-agentic Scientist AI — led by Yoshua Bengio. — Alignment Project
Feb 19, 2026Linglong KongUniversity of Alberta~$589,500*Technical AI safety researchuk_aisiSample-Efficient Online Fine-Tuning Against Resistant Behaviors: Statistical Foundations for Post-Training Alignment — led by Linglong Kong. — Alignment Project
Feb 19, 2026Center for Responsible, Decentralized IntelligenceUniversity of California, Berkeley~$589,500*Technical AI safety researchuk_aisiModel Organisms of Reward Hacking and Long-Horizon Deception for Testing RL Alignment — led by Dawn Song. — Alignment Project
Feb 19, 2026Cole WyethUniversity of Waterloo~$589,500*Foundationsuk_aisiSafe Superintelligence via AIXI Agent Foundations — led by Cole Wyeth. — Alignment Project
Feb 19, 2026Recursive Safeguarding Ltd~$589,500*Technical AI safety researchuk_aisiBayesian Scientist AI for LLM Agent Alignment: GFlowNet-sampled plan abstractions and white-Box monitors to mitigate drift and deception — led by Younesse Kaddar. — Alignment Project
Feb 19, 2026Andrew SaxeUniversity College London~$589,500*Technical AI safety researchuk_aisiTheoretical Model Organisms of Misalignment: Learning Dynamics of Misalignment and Theory-Grounded Mitigation — led by Andrew Saxe. — Alignment Project
Feb 19, 2026Bei JiangUniversity of Alberta~$589,500*Technical AI safety researchuk_aisiAdaptive Tilting to Quantify Tail Risks in Aligned Language Models — led by Bei Jiang. — Alignment Project
Feb 19, 2026Shirin Saeedi BidokhtiUniversity of Pennsylvania~$589,500*Technical AI safety research, AI governance, policy, and advocacyuk_aisiStrategic Behavior and Manipulation in Large Language Models: Experiments, Theory, and Regulatory Mechanisms — led by Shirin Saeedi Bidokhti. — Alignment Project
Feb 19, 2026Louis Thomson (Independent Researcher)~$589,500*Technical AI safety researchuk_aisiInvestigating Dynamics of Agentic Monitoring in AI Control — led by Louis Thomson. — Alignment Project
Feb 19, 2026Toulouse School of Economics~$589,500*Technical AI safety researchuk_aisiDesigning Human-AI Interactions: Misalignment, Robustness, and Trust — led by Alex Smolin, Piotr Dworczak. — Alignment Project
Feb 19, 2026Vinod VaikuntanathanMassachusetts Institute of Technology~$589,500*Technical AI safety researchuk_aisiVerification, Robustness and Alignment through a Cryptographic Lens — led by Vinod Vaikuntanathan. — Alignment Project
Feb 19, 2026Ari HoltzmanUniversity of Chicago~$589,500*Technical AI safety researchuk_aisiHyperstition in LLMs: Mapping how narratives become model policies, probing their persistence after removal, and building tamper-evident tripwires against self-fulfilling AI behaviour — led by Ari Holtzman. — Alignment Project
Feb 19, 2026Inbal Talgam-CohenTel Aviv University~$589,500*Technical AI safety researchuk_aisiMitigating Moral Hazard in AI using Contracts — led by Inbal Talgam-Cohen. — Alignment Project
Feb 19, 2026Or ZamirTel Aviv University~$589,500*Technical AI safety researchuk_aisiProvable and Rigorous Notions of Security for Machine Learning via Cryptography — led by Or Zamir. — Alignment Project
Feb 19, 2026Zhijing JinUniversity of Toronto~$589,500*Technical AI safety researchuk_aisiGame-Theoretic Safety Guarantees for Advanced AI Systems — led by Zhijing Jin. — Alignment Project
Feb 19, 2026Vinod VaikuntanathanMassachusetts Institute of Technology~$589,500*Technical AI safety researchuk_aisiBackdoors and Cryptographic Hardness in Neural Networks — led by Vinod Vaikuntanathan. — Alignment Project
Feb 19, 2026Eva SilversteinStanford University~$589,500*Technical AI safety researchuk_aisiPrecision optimization and symmetry breaking for improving AI predictability — led by Eva Silverstein. — Alignment Project
Feb 19, 2026Fernando Rosas~$589,500*Technical AI safety researchuk_aisiCybernetic Foundations for Aligned Agency: From Internal Models to Verifiable Control — led by Fernando Rosas. — Alignment Project
Feb 19, 2026Timaeus~$589,500*Interpretabilityuk_aisiPatterning: Singular Learning Theory for Alignment — led by Jesse Hoogland. — Alignment Project
Feb 19, 2026Joan VeljaUniversity of Oxford~$589,500*Technical AI safety researchuk_aisiAdversarial Elicitation Theory — led by Joan Velja. — Alignment Project
Feb 19, 2026Benjamin Van RoyStanford University~$589,500*Technical AI safety researchuk_aisiA Mathematical Model of Misalignment: a general mathematical model of how a training process can give rise to a misaligned AI system that poses catastrophic risk — led by Benjamin Van Roy. — Alignment Project
Feb 19, 2026Anti Entropy~$589,500*Technical AI safety researchuk_aisiInvestigating the potential for AI debate to autonomously provide meaningful oversight on difficult-to-evaluate safety-relevant tasks — led by Jason Brown. — Alignment Project
2026METR—Evalsorg_reportedListed as a supporter in METR’s August 2026 funding update; amount not disclosed.
Apr 3, 2025Gina NeffQueen Mary University of London~$129,800†AI safetyuk_aisiWork for AI Safety (WAIS): Sociotechnical capabilities in the workplace to counter systemic risks — led by Gina Neff. Seed grant of up to £200,000. — Systemic Safety Grants (Phase 1)
Apr 3, 2025Newtons Tree~$129,800†AI safetyuk_aisiCLIO: Clinical LLM Integration and Oversight — led by Haris Shuaib. Seed grant of up to £200,000. — Systemic Safety Grants (Phase 1)
Apr 3, 2025Farhana Ferdousi LizaUniversity of East Anglia, University of Sussex~$129,800†AI safetyuk_aisiBRA(AI)N: Building Resilience and Accountability in Artificial Intelligence Navigation — led by Farhana Ferdousi Liza, Shoaib Ahmed, Katherine Deane. Seed grant of up to £200,000. — Systemic Safety Grants (Phase 1)
Apr 3, 2025Adel BibiUniversity of Oxford~$129,800†AI safetyuk_aisiThe Safety of Operating Systems AI Agents: Formulations, Evaluations, and Certification — led by Adel Bibi, Philip Torr, Adam Mahdi. Seed grant of up to £200,000. — Systemic Safety Grants (Phase 1)
Apr 3, 2025Michael BronsteinMcGill University, Mila, University of Oxford~$129,800†AI safetyuk_aisiTowards Trustworthy AI Agents for Information Veracity — led by Michael Bronstein, Reihaneh Rabbany, Shenyang Huang. Seed grant of up to £200,000. — Systemic Safety Grants (Phase 1)
Apr 3, 2025Carl MacraeUniversity of Nottingham~$129,800†AI safetyuk_aisiBuilding a learning infrastructure for systemic AI safety: developing processes for investigating and learning from systemic AI safety incidents — led by Carl Macrae. Seed grant of up to £200,000. — Systemic Safety Grants (Phase 1)
Apr 3, 2025Shoaib EhsanQueensland University of Technology, University College London, University of Southampton~$129,800†AI safetyuk_aisiPRIV-LOC: Assessing and Mitigating Privacy Risks of Vision-Language Models in Image-based Geolocation Systems — led by Shoaib Ehsan, Jack Stilgoe, Michael Milford. Seed grant of up to £200,000. — Systemic Safety Grants (Phase 1)
Apr 3, 2025Mark McGillUniversity of Edinburgh, University of Glasgow~$129,800†AI safetyuk_aisiWearAI: Examining the Societal Vulnerabilities Exposed by AI Embedded in Context-Aware Smart Wearables — led by Mark McGill, Richard Jones, Tanaya Guha. Seed grant of up to £200,000. — Systemic Safety Grants (Phase 1)
Apr 3, 2025Qian LuCoventry University~$129,800†AI safetyuk_aisiSystemic Risk of Using AI-Generated Synthetic Data in Autonomous Vehicle Development — led by Qian Lu, Vasile Palade, Huw Davis. Seed grant of up to £200,000. — Systemic Safety Grants (Phase 1)
Apr 3, 2025The Alan Turing Institute~$129,800†AI safetyuk_aisiAI-enabled disinformation during security incidents: mitigating the risk of public disorder — led by Alexander Babuta, Ardi Janjeva, Sam Stockwell. Seed grant of up to £200,000. — Systemic Safety Grants (Phase 1)
Apr 3, 2025Marios KogiasImperial College London~$129,800†AI safetyuk_aisiC3Infer: A Framework for Compartmentalized, Confidential, and Certified AI Inference — led by Marios Kogias, Hamed Haddadi. Seed grant of up to £200,000. — Systemic Safety Grants (Phase 1)
Apr 3, 2025Yali DuKing's College London~$129,800†AI safetyuk_aisiEvaluating the Cooperative Behaviour of Systems of Generative Agents — led by Yali Du. Seed grant of up to £200,000. — Systemic Safety Grants (Phase 1)
Apr 3, 2025Nick HawesUniversity of Oxford~$129,800†AI safetyuk_aisiAI and Hard Choices: The Parity Model — led by Nick Hawes, Ruth Chang. Seed grant of up to £200,000. — Systemic Safety Grants (Phase 1)
Apr 3, 2025Brian SheilAnglia Ruskin University, University of Cambridge~$129,800†AI safetyuk_aisiGuiding the boots on the ground: Advancing ethically informed socio-technical safety of AI systems in the public sector — led by Brian Sheil, Jennifer Schooling, Maya Indira Ganesh. Seed grant of up to £200,000. — Systemic Safety Grants (Phase 1)
Apr 3, 2025Caitlin BentleyKing's College London~$129,800†AI safetyuk_aisiEvolving Human-AI Competencies: Workforce Development for Building Systemically Safe Cyber-physical Systems — led by Caitlin Bentley, Gordon Meadow, David Wavell. Seed grant of up to £200,000. — Systemic Safety Grants (Phase 1)
Apr 3, 2025Alexander SerbUniversity of Edinburgh~$129,800†AI safetyuk_aisiArtificial intelligence Enabled Guide for Investment Strategies (AEGIS) — led by Alexander Serb, Themis Prodromakis. Seed grant of up to £200,000. — Systemic Safety Grants (Phase 1)
Apr 3, 2025Health and Safety Executive Science and Research Centre, Emlyn Square~$129,800†AI safetyuk_aisiSafe Artificial Intelligence for High Hazard Environments (SAFEHAZ) — led by Martin Thomson, Richard Waine. Seed grant of up to £200,000. — Systemic Safety Grants (Phase 1)
Apr 3, 2025Antonio Valerio Miceli-BaroneUniversity of Edinburgh~$129,800†AI safetyuk_aisiUnderstanding and Improving the Behaviour of AI Agents in Competitive and Cooperative Games — led by Antonio Valerio Miceli-Barone, Vaishak Belle, Shay B. Cohen. Seed grant of up to £200,000. — Systemic Safety Grants (Phase 1)
Apr 3, 2025Centre for Legal TechnologyUlster University~$129,800†AI safetyuk_aisiThe use of Agentic AI in Judicial Decision-Making — led by John Keers, Jun Liu, Niall McCarroll. Seed grant of up to £200,000. — Systemic Safety Grants (Phase 1)
Apr 3, 2025Syed A. R. ZaidiUniversity of Leeds~$129,800†AI safetyuk_aisiStrategic Decision-Making and Cooperation among AI Agents: Exploring Safety and Governance in Telecom — led by Syed A. R. Zaidi, Zeinab Nizami, Maryam Hafeez. Seed grant of up to £200,000. — Systemic Safety Grants (Phase 1)

* UK AISI reported a total of 27m GBP over 60 projects, so estimating each of them at the average grant size.

† UK AISI awarded grants of up to 200k GBP for this round, so estimating each of them at 100k.