Let’s begin with understanding how hacks occur. In most cases, security breaches in software are the result of edge cases a developer didn’t anticipate during the program’s unit testing and, therefore didn’t write tests for. But what if there was a way to address these unpredictable scenarios with a single method that can stress-test nearly every possibility?
What is fuzz testing, and how can it be applied to smart contracts in Solidity?
We’ll start with understanding fuzz testing itself and, by the end of this article, connect the dots to Solidity smart contracts. Now, if you’re here for the formal textbook definition of fuzz testing, I’ll save you the trouble because that’s not what you’re looking for, right? 🙂
Let me simplify it for you. When writing tests, the goal is often to achieve 100% code coverage. However, even with perfect coverage, you can’t guarantee that there are no bugs lurking in the code. This is where fuzzers come into play. Fuzzers generate a range of inputs that test the boundaries you may have missed in your unit tests.
To define it formally:
“Fuzz testing (or fuzzing) involves feeding random data to a simulation of the system in an attempt to break it.”
The effectiveness of fuzzing, however, relies on the quality of the fuzz tests you write. Fuzzers, as software tools, are inherently unintelligent. They operate within computational boundaries and lack context or decision-making ability.
For instance, in a system with multiple possible actions, a fuzzer may randomly choose an inappropriate one. Take the case of an onlyOwner function—if the fuzzer uses an invalid address, it will naturally trigger a reversion. These kinds of situations are predictable and trivial, often classified as low-hanging fruit. Since you expect these failures, they should already be covered in unit tests.
To avoid wasting valuable fuzzing resources on such irrelevant scenarios, it’s crucial to bypass obvious failures and focus on edge cases that matter. Writing effective fuzz tests, therefore, is about squeezing the most value out of the process.
Introducing Invariants: The Core Properties That Must Always Hold
Now let’s talk about invariants also called properties. This is where things get slightly more nuanced, but not as complicated as it might seem. Simply put, an invariant is a property or rule that you assert must always remain true.
Unlike unit tests, where you provide a single input and validate an expected result, invariant testing verifies that a particular property holds true across many random inputs. Fuzzers repeatedly test these properties with a wide range of values, ensuring the system behaves as expected under all scenarios.
Let me simplify further. In the world of DeFi, invariants are the foundational rules that protocols depend on to maintain stability and fairness. These are the “laws” that must never be broken, no matter what actions are performed.
Lending Market Invariant
In lending protocols like Compound or Aave, one key rule is that a user’s borrowed value can never exceed their collateral. To explain this further, when you borrow assets, you must deposit collateral worth more than the borrowed value. It’s similar to a mortgage where you can’t borrow more than the value of your house. The protocol prevents any actions that would put accounts in this unsafe state or worsen an already risky situation.
AMM Invariant
Automated Market Makers like Uniswap or SushiSwap rely on a mathematical invariant to maintain liquidity pool balance. This rule is expressed as x * y = k, where x and y represent token amounts, and k is a constant. If someone buys more of one token, the price increases proportionally to preserve the invariant.
Staking/Liquidity Mining Invariant
In staking protocols, there’s a simple but critical rule: a user can only withdraw the same number of tokens they originally deposited. For example, if you stake 10 tokens, you can withdraw exactly 10 tokens back. While rewards are earned for staking, the principle amount remains constant.
Example time?
pragma solidity ^0.8.0;
contract UniswapInvariantCheck {
uint256 public reserveX; // Reserve for token X
uint256 public reserveY; // Reserve for token Y
constructor(uint256 _initialX, uint256 _initialY) {
reserveX = _initialX; // Initialize reserves
reserveY = _initialY;
}
// Function to simulate a token swap
function swap(uint256 inputX, uint256 inputY) public {
require(inputX == 0 || inputY == 0, "Only one token can be swapped"); // Ensure only one token is swapped
if (inputX > 0) {
// Token X is being swapped into the pool
uint256 newReserveX = reserveX + inputX; // Update reserve for X
uint256 newReserveY = (reserveX * reserveY) / newReserveX; // Calculate new reserve for Y using invariant
reserveX = newReserveX; // Update state
reserveY = newReserveY;
} else if (inputY > 0) {
// Token Y is being swapped into the pool
uint256 newReserveY = reserveY + inputY; // Update reserve for Y
uint256 newReserveX = (reserveX * reserveY) / newReserveY; // Calculate new reserve for X using invariant
reserveX = newReserveX; // Update state
reserveY = newReserveY;
}
}
// Function to check if the invariant holds
function invariantHolds() public view returns (bool) {
uint256 k = reserveX * reserveY; // Calculate the constant k
return k == reserveX * reserveY; // Verify if x * y = k
}
}
|
- The Invariant Rule (x * y = k):
- In Uniswap V3, reserveX and reserveY represent the quantities of two tokens in the liquidity pool.
- The invariant x * y = k ensures that the product of these reserves remains constant during swaps. This rule is fundamental for maintaining the balance of token prices.
- How the Swap Function Works:
- Input Validation: The swap function ensures that only one token (either X or Y) is swapped at a time (require(inputX == 0 || inputY == 0)).
- Updating Reserves: If token X is swapped, the new reserve for Y is calculated using the invariant formula:
newReserveY = (reserveX * reserveY) / newReserveX
- Similarly, if token Y is swapped, the new reserve for X is computed:
newReserveX = (reserveX * reserveY) / newReserveY
- Invariant Verification:
The invariantHolds function checks if the product of reserveX and reserveY remains consistent. If any action disrupts this balance, the invariant will not hold, indicating an issue in the implementation or logic.
Stateless vs. Stateful Fuzzing: Two Approaches to Breaking Systems
To understand fuzzing techniques, let’s use the analogy of a fragile glass.
Stateless fuzzing tests each scenario independently. Imagine testing a glass by:
- Tapping it with a spoon.
- Dropping a pebble into it.
- Throwing it to the ground.
In each case, you use a fresh glass. Past actions don’t influence the next test. While this method is fast, it overlooks how earlier actions could impact the outcome.
Stateful fuzzing, on the other hand, uses the same glass across all tests. If you tap the glass in the first step, drop a pebble in the second, and throw it in the third, you observe the cumulative effects. This approach mirrors real-world systems where previous actions shape future behavior, uncovering deeper bugs.
Bounded Model Checking (BMC): Setting Limits for Effective Testing
Bounded Model Checking (BMC) improves fuzz testing by limiting the steps taken to identify bugs. Instead of endlessly exploring inputs, BMC sets logical “bounds.”
For instance, depositing zero tokens into an AMM might trigger a MIN_INITIAL_SHARES error. Since this is a predictable failure, you guide the fuzzer to avoid such inputs, focusing instead on meaningful edge cases.
Think of BMC as navigating a maze. You decide to only check paths that take 10 steps or fewer. If a bug exists within those bounds, you’ll find it. If not, the system remains stable within that range.
End-to-End Testing Meets Fuzzing: The Perfect Combination
End-to-End (E2E) testing simulates real-world user actions, ensuring that a system behaves as expected. For example, an E2E test for a signup form would validate various inputs: blank fields, invalid emails, and valid credentials.
When combined with fuzz testing, E2E tests become even more powerful. While E2E testing checks normal workflows, fuzzing introduces unpredictability to test how the system responds under stress. Together, they provide a robust framework for validating both intended and unexpected behavior.
Conclusion:
Fuzz testing is a game-changer for finding hidden vulnerabilities that traditional testing might miss. By combining fuzzing with invariant testing, we go beyond just testing individual functions and focus on the system’s overall behavior and stability. Methods like stateful fuzzing, bounded model checking, and end-to-end testing work together to uncover edge cases and make smart contracts more secure.
