property-based-testing-benchmark

Property-based testing benchmark

Description

This project aims to provide a comprehensive benchmark analysis of the performance and effectiveness of different property-based testing tools against real-world DeFi projects.

• Halmos
• Foundry
• Echidna
• Medusa

Usage

Launch the benchmark infrastructure with Terraform:

cd infrastructure
terraform init
terraform apply

Run the benchmark

python3 -m benchmark producer --full-benchmark

Analyse the results

python3 -m benchmark analyser

Project structure

• Dockerfile: Install all referenced tools and benchmark scripts
• infrastructure: Set of Terraform configuration files that will launch
  • SQS: queue of job messages
  • S3: bucket of job outputs
  • ECS: cluster of consumer components, responsible for reading the SQS queue and launching runner tasks that execute the benchmark and upload results to S3
  • ECR: registry that hosts the Docker image
  • IAM: access controls for Docker containers to manage the referenced resources
  • VPC: networking required for ECS
• benchmark: Benchmark scripts
  • producer: Push job messages to the queue
  • consumer: Pull messages from the queue and launch a new task to run the benchmark
  • runner: Run benchmark jobs, and analyse benchmark results
  • analyser: Analyse benchmark results

Contribute

Contributions are welcome. Feel free to open an issue or pull request.