Overview
HWREBench, short for Hardware Reverse Engineering Bench, examines the ability of AI agents to perform hardware hacking against embedded devices, peripherals, and smart systems. It was created to make the emerging capabilities of AI-assisted reverse engineering more visible and to track how those capabilities change over time.
The project focuses on devices whose firmware may receive few updates and whose security has often depended partly on obscurity. It is presented as an evolving, subjective evaluation rather than a polished universal score for comparing every model.
Evaluation Approach
Testing is performed one device and one AI model at a time. Depending on the target, a device is connected to a dedicated Wi-Fi network or computer, while the researcher watches the model’s reasoning and tool calls for potentially unsafe directions.
The evaluation may also include physical access to hardware through tools such as a serial adapter or Bus Pirate. This makes it possible to inspect flash memory and assess whether sensitive device data can be recovered. Exploit solutions are not published because of the security implications.
Findings So Far
- Many tested devices contained security gaps that open-source models could identify and exploit with limited steering.
- Most devices included some defensive measures, but only a small number exposed no apparent gaps during the work.
- Closed models sometimes issued warnings, while open models generally attempted the requested evaluation tasks.
- Physical access substantially expanded the available attack surface, including opportunities to dump and inspect flash memory.
Research Context
The benchmark grew out of an AI-assisted effort to reverse engineer a DVD drive and gain control of its laser and movement mechanisms for a microscopy project. That work involved undocumented device commands, firmware validation, encryption, and obfuscation, illustrating how newer models could help solve hardware problems that had previously resisted the researcher’s attempts.
HWREBench is a solo project by Jonathan Whitaker with funding support from Blue Dot Rapid Grants. Its purpose is to demonstrate current AI-agent hardware-security capabilities and highlight why assumptions about the safety of overlooked embedded systems may need reconsideration.