Add a stored procedure mode to the TPC-C benchmark - #638
Open
akorotkov wants to merge 1 commit into
Open
Conversation
TPC-C as implemented here spends most of its wall clock in the network.
A new order is roughly 25 client/server round trips, so on any setup
where the database is not the bottleneck the measurement is dominated by
the round trip time rather than by the database. Other TPC-C harnesses
(HammerDB, for one) run the transactions as stored procedures for this
reason, which makes the numbers hard to compare against BenchBase.
Add an opt-in mode that keeps the workload identical and only collapses
the round trips. With <useStoredProcedures>true</useStoredProcedures>:
* --create=true additionally installs procedures-postgres.sql, five
PL/pgSQL functions that issue exactly the statement sequence of the
matching Java procedure, in the same order, including the FOR UPDATE
on district and stock and the 1% new order rollback;
* each terminal calls one function per transaction instead of stepping
through the statements itself.
The 1% of new orders that reference an unused item id raise SQLSTATE
TPCC1, which NewOrder turns back into a UserAbortException, so they are
still counted as aborted transactions and not as SQL errors.
Such a call is already a complete transaction, so the commit the worker
issues afterwards buys nothing and costs a full round trip. Measured
against a PostgreSQL 18 instance on a second host in the same
availability zone, one round trip on that link is 46 us and a SELECT 1
costs 46 us with autocommit against 88 us with an explicit commit, so
the commit is a whole extra trip. BenchmarkModule therefore gains
usesAutoCommit(), false everywhere except TPC-C in this mode; workers
open their connection with that setting and skip commit and rollback
when it is on. The server has already ended the transaction either way,
including for the new orders that roll back, and the configured
isolation level still reaches the server, which applies it to the
implicit transactions.
ScriptRunner splits scripts on lines ending in the delimiter, which cuts
a PL/pgSQL body in half at its first internal semicolon. Teach it to
track open $tag$ quotes and to keep the line breaks inside them, with
unit tests for the scanner.
The mode is off by default and nothing changes for existing configs. It
is currently implemented for PostgreSQL only; asking for it with another
database type fails at create time with a clear message rather than
silently running the statement path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a stored procedure mode to the TPC-C benchmark
It's currently hard to utilize a large machine with BenchBase TPC-C. BenchBase is too resource-consuming to run it on the same machine as the DB. At the same time, if you run it on a different machine, most of the time is spent on network latency. This PR is written by Claude and implements procedural mode similar to HammerDB. The rest of the PR description is written by Claude. I read both the PR description and the commit; it looks OK to me.
Why
A BenchBase TPC-C transaction is a sequence of individually prepared statements:
a New-Order alone is about 25 client/server round trips. On any setup where the
database is not the bottleneck, what the benchmark measures is largely the round
trip time between the driver and the server, not the database.
That makes BenchBase numbers hard to place next to other TPC-C harnesses.
HammerDB, for instance, runs each transaction as a single stored procedure call,
and the gap this opens is not small. Measuring the same workload against the same
PostgreSQL 18 instance on the same hardware, only changing how the transactions
are issued:
Same schema, same data, same transaction mix, same isolation level. The whole
difference is that a transaction now costs one round trip instead of roughly 26.
The point of this PR is not that stored procedures are faster; it is that a user
comparing systems, or comparing BenchBase against another harness, should be able
to separate the driver's round trips from the database's work. Today BenchBase
can only measure the two together.
What this adds
Setting
in the workload configuration:
--create=trueinstallprocedures-postgres.sqlnext to the schema —five PL/pgSQL functions, one per TPC-C transaction;
the statements itself.
Each function issues exactly the statement sequence of the matching Java
procedure, in the same order, with the same
FOR UPDATEondistrictandstock, the same customer-by-last-name selection (the middle row ordered byc_first, chosen for 60% of Payments and Order-Statuses), the sames_quantityrestock rule, the samec_datarewrite for bad-credit customers,and the same 1% of New-Orders that reference an unused item id and must roll
back. Order-Status still returns its order lines to the client, so that result
set still crosses the wire.
The rolled-back New-Orders raise SQLSTATE
TPCC1, whichNewOrdertranslatesback into a
UserAbortException. They are therefore still counted as abortedtransactions rather than as SQL errors, exactly as in the statement path.
The mode is off by default; existing configurations are unaffected. It is
implemented for PostgreSQL only — asking for it with another database type fails
at create time with a clear message instead of silently falling back.
ScriptRunner
ScriptRunnersplits a script on lines that end with the delimiter, which cutsa PL/pgSQL body in half at its first internal semicolon; today no
$$-quotedbody can be loaded through it at all. This PR teaches it to track open
$tag$quotes, ignore the delimiter inside them, and keep the line breaks (so that a
--comment inside a body does not comment out the rest of the function). Thetag scanner has unit tests. Scripts without dollar quotes take exactly the same
path as before.
New hook in BenchmarkModule
createDatabasegains one call to an overridablegetPostDDLScriptPath(dbType),which returns
nullfor every benchmark but TPC-C in stored procedure mode. Thisis the general "run this after the DDL" hook that stored procedures, triggers, or
extension setup would all need.
Skipping the redundant commit
A procedure call is already a complete transaction, so the
conn.commit()theworker issues afterwards buys nothing and costs a full round trip.
BenchmarkModulegainsusesAutoCommit(), which is false for every benchmarkexcept TPC-C in this mode;
Workeropens its connection with that setting andskips the commit and the rollbacks when it is on. The server has already ended
the transaction in both the success and the abort case, and the configured
isolation level still reaches the server, which applies it to the implicit
transactions (verified for
TRANSACTION_SERIALIZABLE).The round trip is real and measurable. On this pair of hosts a
SELECT 1costs46 us with autocommit and 88 us with an explicit commit, so the commit is a whole
extra trip. In TPC-C at four terminals, where the round trips are visible above
the queueing, dropping it is worth 5,751 -> 6,224 tps with the median transaction
falling from 639 us to 582 us. At 32 terminals it disappears into the run to run
spread, which is +-17% at that concurrency; the three-repeat means there were
31,642 tps with the commit and 32,300 without.
Without this the mode still works, it just pays one more round trip per
transaction.
Testing
mvn test— 251 tests, all passing, including 4 new ones for the dollar quotescanner.
heap-table PostgreSQL 18 and an OrioleDB one,
--create --load --execute:no unexpected SQL errors, and all four TPC-C consistency conditions of
clause 3.3.2 hold afterwards in every combination:
w_ytd = sum(d_ytd)per warehoused_next_o_id - 1 = max(o_id) = max(no_o_id)per districtnew_orderids of a district are contiguoussum(o_ol_cnt) = count(order_line)per districtmodes and an empty "Unexpected SQL Errors" section.
transactions with statements, 30 in 2.1M with procedures, at scale factor 10
and 32 terminals) — a property of TPC-C at low warehouse counts rather than of
this change.
Not included
separate script per dialect behind the same flag.