Randomized testing: Gotta Catch 'Em All
Abstract
Chance does things well.
If we apply this idea to unit tests or integration tests, we can make our tests much more unpredictable — and as a result, uncover issues that our minds would never have dared to imagine! For example, I recently discovered a bug
in a configuration management library that occurs when the Locale is set to AZ. 🤦🏼♂️
Another, even simpler, example:
int input = generateInteger(Integer.MIN_VALUE, Integer.MAX_VALUE);
int output = Math.abs(input);
This can generate -2147483648… which is quite unexpected for an absolute value! 😉
Randomized tests can uncover these twisted edge cases… That’s what the Elasticsearch team has been doing for years using the RandomizedTesting
framework to test all their Java code.
Add to that real integration tests using TestContainers , and you’ll have a complete approach to tests that regularly fail!
After this talk, you’ll never look at the random() function the same way again — and you’ll discover how (bad) luck can actually help you! 🍀
Resources
The following resources were mentioned during the presentation or are useful additional information.
Demo project
The source code for the demo project
RandomizedTesting framework
The framework provided by Carrot Search
Ariane 5 launch video
The Ariane 5 first launch on June 4th, 1996
FSCrawler project
FSCrawler is running tests with RandomizedTesting framework
A weird CI error in FSCrawler
The error I could not reproduce locally
Gestalt issue 242
The issue I opened after my findings