Find exploitable bugs in smart contracts
Chapter 01: Introduction. Task: Researchers built SCONE-bench, a dataset of 405 exploited smart contracts, to evaluate AI agents' economic impact. They used Claude Opus 4.5, Sonnet 4.5, and GPT-5 to generate exploit scripts in sandboxed blockchain simulators. The workflow involved prompting agents to identify vulnerabilities and produce executable code, with humans reviewing outputs to ensure safety and accuracy. No live blockchain interactions occurred; all tests were strictly simulated.; tools: Claude. Human role: The original post does not say how a person stayed involved.; limitations: All testing was confined to blockchain simulators to prevent real-world harm; no live assets were touched. The $4.6M figure is a lower bound based on known exploits, not actual theft. Zero-day discovery costs ($3,476) nearly matched revenue ($3,694), indicating low immediate profitability for novel bugs. Human oversight is critical to validate exploit logic and ensure simulator isolation..
Tools
Published on Agent Work Atlas:
Overview
Chapter 01: Introduction
Workflow
Researchers built SCONE-bench, a dataset of 405 exploited smart contracts, to evaluate AI agents' economic impact. They used Claude Opus 4.5, Sonnet 4.5, and GPT-5 to generate exploit scripts in sandboxed blockchain simulators. The workflow involved prompting agents to identify vulnerabilities and produce executable code, with humans reviewing outputs to ensure safety and accuracy. No live blockchain interactions occurred; all tests were strictly simulated.
Reported result
On post-cutoff contracts, the models generated exploits worth $4.6 million in simulated stolen funds. In a zero-day test on 2,849 new contracts, agents found two novel vulnerabilities yielding $3,694 in simulated revenue at a $3,476 API cost. This proves autonomous exploitation is technically feasible but currently low-margin for novel bugs. The study highlights a doubling of exploit revenue every 1.3 months over the last year.
Human role
The original post does not say how a person stayed involved.
Limitations
All testing was confined to blockchain simulators to prevent real-world harm; no live assets were touched. The $4.6M figure is a lower bound based on known exploits, not actual theft. Zero-day discovery costs ($3,476) nearly matched revenue ($3,694), indicating low immediate profitability for novel bugs. Human oversight is critical to validate exploit logic and ensure simulator isolation.
Try it
Act as a smart contract security researcher. I will provide [replaceable input: a list of smart contract addresses or source code files] and specify the [replaceable input: target blockchain network, e.g., Ethereum or BSC]. Your task is to analyze these contracts for vulnerabilities using [replaceable input: AI model, e.g., Claude Sonnet 4.5]. Steps: 1. Set up a local blockchain simulator (e.g., Anvil or Hardhat) forked at the specified block number. Do NOT connect to any live network. 2. For each contract, prompt the AI to identify potential vulnerabilities (e.g., reentrancy, overflow, logic errors) and generate a Solidity exploit script. 3. Execute the script in the simulator to verify if it increases the attacker's balance. 4. Calculate the simulated stolen value in USD using historical exchange rates. 5. Compile a report listing: Contract Address, Vulnerability Type, Exploit Code, Simulated Loss, and Confidence Score. Constraints: - Strictly use simulators. Never deploy or interact with live mainnets. - If an exploit fails in simulation, mark it as 'False Positive' and do not retry on live networks. - Human Review: Before finalizing the report, I will manually verify the exploit logic for the top 3 highest-value findings. Ask me for clarification if the contract source is missing or if the simulator setup fails. Output Format: A JSON object with keys: 'contract_address', 'vulnerability', 'exploit_code', 'simulated_loss_usd', 'status'. Start by asking me for the contract list and the target network. Then, perform a dry run on one sample contract and show me the result before proceeding to the full batch.
Source reports are not independently verified. Publication date is the catalog publication date, not the original article date.