Note
Made with ❤️ by @clflushopt, @alamb and @kevinjqliu
Originally clflushopt/tpchgen-rs, the project is now maintained at
datafusion-contrib/tpcgen-rs
Modern, blazing fast and easy to use TPC-H and TPC-DS benchmark data generators.
Democratize the comparison of analytical systems by making it easy to efficiently generate benchmark data in modern formats such as Apache Parquet, Apache Arrow, and CSV.
- Blazing Speed 🚀
- Easy to Use: CLI or embeddable libraries
- Obsessively Tested: byte-for-byte identical with the reference generators 📋
- Resource Efficient: multi-core and constant memory use 🧠
tpcgen-cli is distributed as a Python package. Run it with uvx:
uvx tpcgen-cli tpch parquet -s 1 --output-dir /tmp/tpch
uvx tpcgen-cli tpcds parquet -s 1 --output-dir /tmp/tpcdsOr install it with pip:
python -m pip install tpcgen-cli
tpcgen-cli tpch parquet -s 1 --output-dir /tmp/tpchSee the tpcgen-cli README for more examples.
Time to generate TPC-H Scale Factor 100 data in Parquet format. See the performance spreadsheet for details and more comparisons.
| Project | Description |
|---|---|
tpcgen-cli |
Command line tool to generate TPC-H and TPC-DS data in multiple formats |
tpchgen-cli |
Command line tool to generate only TPC-H data in multiple formats (kept for backwards compatibility) |
tpchgen |
Rust library to generate TPC-H data (zero dependencies) |
tpchgen-arrow |
Rust library to generate TPC-H data in Apache Arrow format |
tpcdsgen |
Rust library to generate TPC-DS data (zero dependencies) |
tpcdsgen-arrow |
Rust library to generate TPC-DS data in Apache Arrow format |
We go to great lengths to ensure our data generators produce the same exact bytes as the reference implementations, both for single and multi-part output. Our multi-level strategy for ensuring correctness is detailed in TESTING.md and includes comparing the output of our generators to the original reference implementations as part of CI.
Pull requests are welcome. For major changes, please open an issue first for discussion. See our contributors guide for more details.
Please see architecture guide for details on how the code is structured.
The project is licensed under the Apache License 2.0.