Skip to content

Add a stored procedure mode to the TPC-C benchmark - #638

Open
akorotkov wants to merge 1 commit into
cmu-db:mainfrom
akorotkov:tpcc-stored-procedures
Open

Add a stored procedure mode to the TPC-C benchmark#638
akorotkov wants to merge 1 commit into
cmu-db:mainfrom
akorotkov:tpcc-stored-procedures

Conversation

@akorotkov

Copy link
Copy Markdown

Add a stored procedure mode to the TPC-C benchmark

It's currently hard to utilize a large machine with BenchBase TPC-C. BenchBase is too resource-consuming to run it on the same machine as the DB. At the same time, if you run it on a different machine, most of the time is spent on network latency. This PR is written by Claude and implements procedural mode similar to HammerDB. The rest of the PR description is written by Claude. I read both the PR description and the commit; it looks OK to me.

Why

A BenchBase TPC-C transaction is a sequence of individually prepared statements:
a New-Order alone is about 25 client/server round trips. On any setup where the
database is not the bottleneck, what the benchmark measures is largely the round
trip time between the driver and the server, not the database.

That makes BenchBase numbers hard to place next to other TPC-C harnesses.
HammerDB, for instance, runs each transaction as a single stored procedure call,
and the gap this opens is not small. Measuring the same workload against the same
PostgreSQL 18 instance on the same hardware, only changing how the transactions
are issued:

driver placement terminals statements stored procedures ratio
client on the DB host 8 9,070 tps 13,008 tps 1.43x
client on a second host in the same AZ 4 2,650 tps 6,224 tps 2.35x
client on a second host in the same AZ 32 19,952 tps 35,600 tps 1.78x

Same schema, same data, same transaction mix, same isolation level. The whole
difference is that a transaction now costs one round trip instead of roughly 26.

The point of this PR is not that stored procedures are faster; it is that a user
comparing systems, or comparing BenchBase against another harness, should be able
to separate the driver's round trips from the database's work. Today BenchBase
can only measure the two together.

What this adds

Setting

<useStoredProcedures>true</useStoredProcedures>

in the workload configuration:

  • makes --create=true install procedures-postgres.sql next to the schema —
    five PL/pgSQL functions, one per TPC-C transaction;
  • makes each terminal issue one call per transaction instead of stepping through
    the statements itself.

Each function issues exactly the statement sequence of the matching Java
procedure, in the same order, with the same FOR UPDATE on district and
stock, the same customer-by-last-name selection (the middle row ordered by
c_first, chosen for 60% of Payments and Order-Statuses), the same
s_quantity restock rule, the same c_data rewrite for bad-credit customers,
and the same 1% of New-Orders that reference an unused item id and must roll
back. Order-Status still returns its order lines to the client, so that result
set still crosses the wire.

The rolled-back New-Orders raise SQLSTATE TPCC1, which NewOrder translates
back into a UserAbortException. They are therefore still counted as aborted
transactions rather than as SQL errors, exactly as in the statement path.

The mode is off by default; existing configurations are unaffected. It is
implemented for PostgreSQL only — asking for it with another database type fails
at create time with a clear message instead of silently falling back.

ScriptRunner

ScriptRunner splits a script on lines that end with the delimiter, which cuts
a PL/pgSQL body in half at its first internal semicolon; today no $$-quoted
body can be loaded through it at all. This PR teaches it to track open $tag$
quotes, ignore the delimiter inside them, and keep the line breaks (so that a
-- comment inside a body does not comment out the rest of the function). The
tag scanner has unit tests. Scripts without dollar quotes take exactly the same
path as before.

New hook in BenchmarkModule

createDatabase gains one call to an overridable getPostDDLScriptPath(dbType),
which returns null for every benchmark but TPC-C in stored procedure mode. This
is the general "run this after the DDL" hook that stored procedures, triggers, or
extension setup would all need.

Skipping the redundant commit

A procedure call is already a complete transaction, so the conn.commit() the
worker issues afterwards buys nothing and costs a full round trip.
BenchmarkModule gains usesAutoCommit(), which is false for every benchmark
except TPC-C in this mode; Worker opens its connection with that setting and
skips the commit and the rollbacks when it is on. The server has already ended
the transaction in both the success and the abort case, and the configured
isolation level still reaches the server, which applies it to the implicit
transactions (verified for TRANSACTION_SERIALIZABLE).

The round trip is real and measurable. On this pair of hosts a SELECT 1 costs
46 us with autocommit and 88 us with an explicit commit, so the commit is a whole
extra trip. In TPC-C at four terminals, where the round trips are visible above
the queueing, dropping it is worth 5,751 -> 6,224 tps with the median transaction
falling from 639 us to 582 us. At 32 terminals it disappears into the run to run
spread, which is +-17% at that concurrency; the three-repeat means there were
31,642 tps with the commit and 32,300 without.

Without this the mode still works, it just pays one more round trip per
transaction.

Testing

  • mvn test — 251 tests, all passing, including 4 new ones for the dollar quote
    scanner.
  • End-to-end at scale factor 10, 8 terminals, 60 s, both modes, on both a
    heap-table PostgreSQL 18 and an OrioleDB one, --create --load --execute:
    no unexpected SQL errors, and all four TPC-C consistency conditions of
    clause 3.3.2 hold afterwards in every combination:
    • w_ytd = sum(d_ytd) per warehouse
    • d_next_o_id - 1 = max(o_id) = max(no_o_id) per district
    • the new_order ids of a district are contiguous
    • sum(o_ol_cnt) = count(order_line) per district
  • The aborted-transaction histogram shows the expected ~1% of New-Orders in both
    modes and an empty "Unexpected SQL Errors" section.
  • Deadlocks appear at the same order of magnitude in both modes (8 in 1.2M
    transactions with statements, 30 in 2.1M with procedures, at scale factor 10
    and 32 terminals) — a property of TPC-C at low warehouse counts rather than of
    this change.

Not included

  • Only PostgreSQL. The same treatment for MySQL, Oracle, or SQL Server would be a
    separate script per dialect behind the same flag.

TPC-C as implemented here spends most of its wall clock in the network.
A new order is roughly 25 client/server round trips, so on any setup
where the database is not the bottleneck the measurement is dominated by
the round trip time rather than by the database. Other TPC-C harnesses
(HammerDB, for one) run the transactions as stored procedures for this
reason, which makes the numbers hard to compare against BenchBase.

Add an opt-in mode that keeps the workload identical and only collapses
the round trips. With <useStoredProcedures>true</useStoredProcedures>:

  * --create=true additionally installs procedures-postgres.sql, five
    PL/pgSQL functions that issue exactly the statement sequence of the
    matching Java procedure, in the same order, including the FOR UPDATE
    on district and stock and the 1% new order rollback;
  * each terminal calls one function per transaction instead of stepping
    through the statements itself.

The 1% of new orders that reference an unused item id raise SQLSTATE
TPCC1, which NewOrder turns back into a UserAbortException, so they are
still counted as aborted transactions and not as SQL errors.

Such a call is already a complete transaction, so the commit the worker
issues afterwards buys nothing and costs a full round trip. Measured
against a PostgreSQL 18 instance on a second host in the same
availability zone, one round trip on that link is 46 us and a SELECT 1
costs 46 us with autocommit against 88 us with an explicit commit, so
the commit is a whole extra trip. BenchmarkModule therefore gains
usesAutoCommit(), false everywhere except TPC-C in this mode; workers
open their connection with that setting and skip commit and rollback
when it is on. The server has already ended the transaction either way,
including for the new orders that roll back, and the configured
isolation level still reaches the server, which applies it to the
implicit transactions.

ScriptRunner splits scripts on lines ending in the delimiter, which cuts
a PL/pgSQL body in half at its first internal semicolon. Teach it to
track open $tag$ quotes and to keep the line breaks inside them, with
unit tests for the scanner.

The mode is off by default and nothing changes for existing configs. It
is currently implemented for PostgreSQL only; asking for it with another
database type fails at create time with a clear message rather than
silently running the statement path.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant