tulip: optional admission-gate for real financial actions (pay_order, accept_dispute_claim, cancel_subscription, cancel_sent_invoice)

[fede-kamel]

Who we are, up front

We're Tulip (tulip-agents on PyPI) — an open-source agentic AI SDK, Apache-2.0. Pre-GA (currently v2.4.0, actively developed, not a mature/enterprise-proven product) — flagging that plainly. Everything below is working code, run for real, with results reported honestly including where verification fell short. Genuinely open to "not interested" or "needs changes."

What we noticed

Every one of this toolkit's four framework adapters (langchain, openai, crewai, bedrock) ultimately calls through exactly one method to actually execute a real PayPal API request: PayPalAPI.run(method, params) (shared/api.py). As far as we can tell, the only gate on that today is shared/configuration.py's is_tool_allowed() — a static, developer-configured allow-list set once at Configuration construction, with no visibility into a specific call's actual arguments. It can't distinguish "capture $12" from "capture $50,000," has no escalation path, and keeps no audit trail. If that's an inaccurate read of where things stand, genuinely want to know.

What this PR adds

paypal_agent_toolkit/tulip/ — an optional admission layer:

• GovernedPayPalAPI (governance.py) — a drop-in subclass of PayPalAPI. Duck-type compatible: anything that accepts a PayPalAPI instance (this toolkit's own PayPalTool factories) accepts this instead, with no other code changes. Overrides run(method, params) to route every real call through tulip.control.admit() before PayPalAPI.run() (and the real PayPal request it triggers) ever happens. A denied or held call never reaches PayPal.
• classify(method, params) — a pure function returning a tulip.control.Action (name, asset, blast_radius, environment, kind, tags). Deterministic, no model call, no network call — a static lookup against a fixed method-name set, same shape as is_tool_allowed() in spirit but per-call rather than per-configuration.
• A tamper-evident, hash-chained AuditTrail — every decision (allowed and held), independent of PayPal's own transaction logs. GovernedPayPalAPI.audit_trail() exposes .records() / .verify() / .export_jsonl().
• A sync/async bridge (_run_admit_sync) handling the real mismatch between PayPalAPI.run()'s sync signature (every existing tool wrapper calls it synchronously) and admit()'s async one — including the specific failure mode this toolkit's own openai/tool.py triggers (on_invoke_tool is a coroutine calling .run() synchronously from inside an already-running event loop, where a naive asyncio.run() would raise). Falls back to a fresh loop on a separate thread only when a loop is already running on the calling thread.

benign read/draft (e.g. get_order_details)         -> executes normally
real financial action (e.g. pay_order)              -> held for a human, never executes

Nothing changes for anyone who doesn't opt in: tulip-agents isn't added to the top-level pyproject.toml/requirements.txt — only to examples/tulip/requirements.txt, specifically to avoid bloating every installer's dependency tree the way the existing framework packages (langchain, crewai-tools, openai-agents, boto3) currently do as hard core deps.

The exact policy, and the reasoning behind every line of it

Of this toolkit's 31 real tools in shared/tools.py, 9 have a genuine, hard-to-undo financial, liability, or real-external-party consequence and are held for a human by default:

Method	Why
pay_order	Captures/moves real money.
accept_dispute_claim	Accepts real financial liability on a dispute.
cancel_subscription	Real, ongoing revenue impact.
cancel_sent_invoice	A real, customer-facing cancellation.
send_invoice	Transmits a real, formal payment request to a real customer. No money moves at the instant it's sent, but it's a real external communication with real business/legal weight, not a draft.
create_subscription	Starts a real, ongoing recurring-billing commitment.
create_subscription_plan	Defines the terms of a real recurring-billing commitment.
create_recurring_series	Starts a real recurring invoice series.
activate_recurring_series	Activates a real recurring invoice series.

The last 5 were added after a real methodology step, not guessed at from the start: we ran an independent, model-based classifier (separate from the rule-based classify() this PR ships) against this same 31-tool dataset. It agreed completely on the original 4 flagship methods — zero missed money-movement/liability cases — and additionally flagged 7 more. We reviewed every disagreement by hand rather than accepting either side automatically:

• 5 were real gaps, added above. The original policy held cancel_subscription to a high bar but treated starting the same recurring-billing relationship as low-risk — a real asymmetry, not a considered choice. Same logic for send_invoice: a real transmission to a real customer, not a draft.
• 2 we disagreed with on purpose, and kept low-risk: create_order and create_invoice both create a real record, but neither notifies an external party nor starts a recurring commitment — a materially different risk shape from an actual transmission or an actual recurring commitment.

Everything else — every list_/get_/show_ read, draft creation that doesn't notify anyone or start a recurring commitment, shipment tracking, merchant insights — auto-allows. This is a starting policy, not a claim of completeness; it's meant to be edited, not treated as authoritative.

Also disclosed rather than hidden: pay_order's own parameters (OrderIdParameters) don't carry a dollar amount — the amount was fixed earlier, at create_order time. Real amount-aware escalation (e.g. auto-allow a capture under $50, hold anything over) would need a pre-fetch of the order before classifying the capture — a real, useful enhancement this module doesn't attempt, to keep the change small and auditable on its own. classify()'s signature already accepts params, not just method, specifically so a caller can layer that on top without changing the call site.

A real, disclosed finding in this repo's own docs, unrelated to this change

The top-level README lists create_refund/get_refund tools. We grepped the whole shared/ tree — neither exists in shared/tools.py as of this writing. Not fixed here (out of scope), flagging so it isn't confused with something this PR broke, and so this policy wasn't accidentally built against a tool that doesn't exist.

Honest scope limits

• Each existing framework Toolkit.__init__ constructs its own internal PayPalAPI rather than accepting one as a constructor argument, so GovernedPayPalAPI can't drop into openai.toolkit.PayPalToolkit/etc. as-is today. examples/tulip/app_agent.py demonstrates the real path instead: calling this toolkit's own unmodified openai.tool.PayPalTool() factory directly against a GovernedPayPalAPI instance. Making paypal_api an optional constructor arg on the existing toolkits would let this plug into all four uniformly — a real, small, complementary change we didn't make here since it touches files this PR didn't otherwise need to.
• Of the 9 high-risk methods, only pay_order is verified against a real live PayPal sandbox account (see below). The other 8 remain verified only via the mocked full-catalog sweep — exercising most of them for real needs pre-existing sandbox state (an approved subscription, a filed dispute) that itself requires a real buyer-approval redirect flow, out of scope for this pass.
• The adversarial dataset (below) tests near-miss method names, not adversarial params content. method is a closed, fixed dispatch string chosen by the calling framework's own tool definitions, not attacker-controlled free text — PayPalAPI.run() itself already rejects any string not in the real 31-tool catalog, so there's no equivalent evasion surface to the kind a free-text query language would have.

Dataset validation — the full real catalog, not a handful of examples

examples/tulip/datasets/ — three standalone scripts, not shipped as part of the installable package:

python datasets/full_catalog.py    # all 31 real tools, hand-reviewed ground truth
python datasets/full_run.py        # all 31 run end-to-end (mocked execution)
python datasets/adversarial.py     # near-miss method-name variants

full_catalog.py — every one of the 31 real tools classified and checked against an independently-written ground-truth expectation, reviewed one at a time, not re-derived from HIGH_RISK_METHODS itself. 0 mismatches.

full_run.py — all 31 run end-to-end through the real GovernedPayPalAPI.run(), not just classified in isolation:

Tools that actually executed: 21
Tools held (require_human): 9
No gate failures -- every high-risk method was blocked from executing, every low-risk method actually ran.
audit trail: 31 decisions across all 31 calls, chain intact: True

Real, disclosed finding along the way: PayPalAPI.run() itself refuses get_merchant_insights outright in sandbox mode, for its own reasons, unrelated to this gate. This gate correctly allowed it (it's a read); PayPal's own code raised independently — confirming control genuinely reaches real PayPal logic on allow rather than being intercepted by the mock.

adversarial.py — 61 cases total: 5 near-miss method-name variants (case, hyphenation, leading/trailing whitespace, no-underscore) for each of the 9 real high-risk method strings (54 cases), plus 7 real low-risk methods that share words with a high-risk one, including the two sharpest real near-misses in the catalog: send_invoice_reminder literally starts with the string send_invoice, and list_subscription_plans shares two whole words with create_subscription_plan. 0 false positives, 0 false negatives.

Verified live against a real PayPal sandbox account — no mocks

examples/tulip/datasets/live_sandbox.py (credential-gated via .env, not run in CI). Real output:

1. create_order (real, low-risk, should auto-allow)
  EXECUTED -> {"id": "97913981JL2462612", "status": "PAYER_ACTION_REQUIRED", ...}

2. get_order_details (real, low-risk, should auto-allow)
  EXECUTED -> {"message": "...", "status": "PAYER_ACTION_REQUIRED", ...}

3. pay_order (real, HIGH-RISK, must be held, must never reach PayPal)
  REQUIRE_HUMAN -> blast radius 5 exceeds the maximum 1; labels ['high-risk'] require human approval

4. Same pay_order, explicit allow-everything policy override
   (expect PayPal's own real ORDER_NOT_APPROVED rejection)
  ALLOWED, THEN REAL PAYPAL API ERROR -> HTTPError: 422 Client Error: ... /capture

5. list_products / list_disputes -> EXECUTED, real empty catalogs
   list_transactions -> ALLOWED, THEN REAL PAYPAL 403 (see below)
   get_merchant_insights -> ALLOWED, THEN REAL sandbox-mode refusal (see above)

primary audit trail: 7 decisions, chain intact: True
override-policy audit trail: 1 decisions, chain intact: True

Step 4 is the most important line in this whole PR: a GovernedPayPalAPI built with an explicit allow-everything ControlPolicy override genuinely reached PayPal's real API and got PayPal's own real ORDER_NOT_APPROVED business rejection back (no real buyer ever approved the order through PayPal's own checkout flow, since nothing here scripts a real browser/login approval). That proves the override isn't a stub, and that this gate and PayPal's own business validation are two independent, composable layers — not overlapping or redundant ones.

Two real findings from that live run, neither a bug in this gate:

• get_order_details's own response message (shared/orders/tool_handlers.py) always says "has been successfully captured" regardless of the order's real status — looks copy-pasted from capture_order's handler. Worth knowing if anything reads that message field instead of the real status field, since it would be misleading.
• list_transactions returned a real 403 Forbidden from PayPal's Transaction Search API on this sandbox app — looks like a scope/permission a default sandbox app doesn't have enabled by default, not something this gate controls. Correctly allowed by this gate (it's a read); the 403 is PayPal's own real API declining it independently.

One real bug this live run caught in this PR's own examples, fixed: app_agent.py and the dataset scripts originally used the wrong request field name (id instead of the real order_id, per shared/orders/parameters.py's OrderIdParameters/CaptureOrderParameters). Invisible under full mocking, which replaces execute() wholesale and never validates params against the real schema — caught immediately once real schema validation was in the loop. Fixed everywhere it appeared.

Tests

examples/tulip/test_governance.py — 6 real tests against GovernedPayPalAPI/classify() (monkeypatched execution, no PayPal call): all 9 high-risk methods flagged, reads/drafts stay low-risk, a low-risk call genuinely executes and lands an allow audit record, a high-risk call is genuinely held and its underlying execute function genuinely never runs, the audit trail survives mixed decisions and verifies, and — the one specifically worth calling out — a test reproducing this toolkit's own on_invoke_tool call shape (a coroutine calling .run() synchronously from inside an already-running event loop), confirming the sync/async bridge holds up under the real call pattern the openai adapter actually uses, not a simplified one.

What's in this PR

paypal_agent_toolkit/tulip/
├── __init__.py
└── governance.py              # GovernedPayPalAPI, classify(), the sync/async bridge

examples/tulip/
├── README.md
├── requirements.txt            # tulip-agents as a dependency of this example only
├── .env.sample
├── app_agent.py                 # runnable demo, no credentials required (stubbed)
├── test_governance.py           # 6 real tests, no PayPal call
└── datasets/
    ├── full_catalog.py          # all 31 real tools vs. hand-reviewed ground truth
    ├── full_run.py               # all 31 run end-to-end through the real gate
    ├── adversarial.py            # near-miss method-name robustness
    └── live_sandbox.py           # real PayPal sandbox account, no mocks

The ask

Whether this is a real gap from where you sit, whether these 9 high-risk methods (and the 2 we deliberately excluded) are the right line, and whether an optional dependency like this is wanted at all — happy to iterate, narrow scope, or hear it's not a fit.

[fede-kamel]

Update: validated against 3 real datasets, not the 2-tool demo in isolation

Same "sweep the whole real thing, don't cherry-pick" methodology used earlier on a similar admission-gate PR against Velociraptor's real artifact catalog. Added examples/tulip/datasets/:

• full_catalog.py — all 31 real tools in shared/tools.py, classified and checked against an independently hand-reviewed ground truth (not re-derived from the code itself). 0 mismatches.
• full_run.py — all 31 run end-to-end through the real GovernedPayPalAPI.run() (mocked PayPal execution, per the earlier caveat). All 4 high-risk tools provably never execute; every low-risk tool that doesn't hit an unrelated real constraint actually does. Real, disclosed find along the way: PayPalAPI.run() itself refuses get_merchant_insights in sandbox mode — nothing to do with this gate, correctly passed through once allowed. Audit trail intact across all 31 decisions.
• adversarial.py — 29 near-miss method-name variants (case, hyphenation, whitespace, no-underscore) against the 4 high-risk methods, plus 5 real low-risk methods that share a word with a high-risk one (get_order_details vs pay_order, list_disputes vs accept_dispute_claim, etc.). 0 false positives, 0 false negatives.

Also caught and fixed a real, small error the sweep itself surfaced: earlier text here (and in the code/README) said "30 real tools" — the actual verified count is 31. Fixed everywhere.

[fede-kamel]

Update: verified live against a real PayPal sandbox account -- no mocks

Earlier in this PR I said no PayPal sandbox credentials were available. That's resolved -- ran examples/tulip/datasets/live_sandbox.py (new, credential-gated, not run in CI) against a real sandbox account:

1. create_order (real, low-risk, should auto-allow)
  EXECUTED -> {"id": "97913981JL2462612", "status": "PAYER_ACTION_REQUIRED", ...}

3. pay_order (real, HIGH-RISK, must be held, must never reach PayPal)
  REQUIRE_HUMAN -> blast radius 5 exceeds the maximum 1; labels ['high-risk'] require human approval

4. Same pay_order, explicit allow-everything policy override
   (expect PayPal's own real ORDER_NOT_APPROVED rejection)
  ALLOWED, THEN REAL PAYPAL API ERROR -> HTTPError: 422 Client Error: ... /capture

primary audit trail: 7 decisions, chain intact: True
override-policy audit trail: 1 decisions, chain intact: True

The forced-allow override in step 4 genuinely reaches PayPal's real API and gets PayPal's own real business rejection back (no real buyer ever approved the order through PayPal's own checkout flow) -- proves the override isn't a stub, and that this gate and PayPal's own business validation are two independent, composable layers, not overlapping ones.

Two real findings, neither a bug in this gate:

• get_order_details's response message always says "has been successfully captured" regardless of the order's real status — looks copy-pasted from capture_order's handler in shared/orders/tool_handlers.py. Worth knowing if anything reads that message field instead of the real status field.
• list_transactions got a real 403 Forbidden from PayPal's Transaction Search API — looks like a scope this particular sandbox app doesn't have enabled, not something this gate controls.

One real bug this caught in this PR's own examples: app_agent.py and the dataset scripts used the wrong request field name (id instead of the real order_id, per shared/orders/parameters.py's OrderIdParameters/CaptureOrderParameters). Invisible under full mocking, which replaces execute() wholesale and never validates params against the real schema — caught immediately once real schema validation entered the loop. Fixed everywhere.

Still honestly not covered live: accept_dispute_claim, cancel_subscription, cancel_sent_invoice remain verified only via the mocked full-catalog sweep — exercising them for real needs pre-existing sandbox state (an approved subscription, a filed dispute) that itself requires a real buyer-approval redirect flow, out of scope for this pass.

[fede-kamel]

Update: widened the high-risk policy from 4 to 9 methods after independent-classifier review

Tested an independent, model-based classifier against this PR's own 31-tool ground truth (separate from the shipped rule-based classify()) — it agreed completely on the original 4 flagship methods (zero missed money-movement/liability cases), and flagged 7 more. Re-reviewed each disagreement by hand rather than taking either side at face value.

5 of the 7 were real gaps, now added:

• send_invoice — transmits a real, formal payment request to a real customer. No money moves the instant it's sent, but it's a real external communication, not a draft.
• create_subscription, create_subscription_plan, create_recurring_series, activate_recurring_series — each starts a real recurring-billing commitment. The original policy held cancel_subscription to a high bar but treated starting the same relationship as low-risk — a real asymmetry, not a considered choice.

2 of the 7 stay low-risk, disagreeing on purpose: create_order and create_invoice both create a real record but notify no external party and start no recurring commitment — a materially different risk shape from an actual transmission or an actual recurring commitment.

HIGH_RISK_METHODS: 4 → 9. Updated everywhere the old count appeared — module docstring, README, the full-catalog ground truth (with per-method reasoning, not just a flag flip), the adversarial dataset (widened to all 9, including two of the sharpest real near-misses in the catalog: send_invoice_reminder literally starts with the string send_invoice, and list_subscription_plans shares two whole words with create_subscription_plan), and the unit tests.

Re-verified after every change: 6/6 tests, 31/31 full-catalog classification (0 mismatches against the updated ground truth), 31/31 full end-to-end run (9 held, 21 executed, 0 gate failures, audit trail intact), 31 adversarial cases (0 false positives/negatives).

[fede-kamel]

Second widening: four tools that notify a customer or take a payment

Re-audited the policy by asking what it covers as a category rather than checking the listed methods, and found the same class of gap the first widening found — this time on the outbound-communication side.

send_invoice is held on the reasoning that it is "a real external communication with real business/legal weight, not a draft." That reasoning was never carried to its neighbours. Worse, the ground-truth file filed them under a comment that read "...drafts/creates-without-committing-funds or without-notifying-anyone, listings, reminders, QR generation..." — listing reminders among the things that notify nobody.

Four methods move to high-risk:

generate_invoice_qr_code — the strongest case, and a gate bypass rather than a judgment call. PayPal's own description is "a QR code for an invoice, which can be used to pay the invoice using a mobile device or scanning app." That is a payable surface, and it can be generated for an invoice that was never sent. An agent held at send_invoice could produce a scannable payment artifact and distribute it out of band, reaching exactly the outcome the gate just refused. An unheld path to a held outcome is worth more to an attacker than the held path itself.

setup_invoice_auto_reminders and update_invoice_auto_reminder — "sets up automatic reminders for unpaid invoices, for BEFORE_DUE and/or AFTER_DUE." Account-level configuration committing the merchant to a standing schedule of future automated messages to customers. That is precisely the "commits to a real, ongoing relationship" shape used to justify holding create_subscription_plan, and with a wider blast radius: it applies to every unpaid invoice on the account, not to one customer.

send_invoice_reminder — the weakest of the four, and labelled that way in the docstring rather than presented as equal. The invoice is already sent, so the customer isn't hearing from the merchant for the first time. It's included because it's still unsolicited outbound under the merchant's name and an agent in a retry loop can send many. If you disagree with one of these four, it should be this one — drop it and keep the other three; that's why it's listed last.

Revalidated, all green

Same three datasets, re-run:

check	result
all 31 real tools vs hand-reviewed ground truth	0 mismatches
all 31 end-to-end through the real gate	13 high-risk provably never executed, low-risk ran
audit chain across 31 decisions	intact
86 adversarial cases	0 false positives, 0 false negatives

One honest consequence worth flagging

The adversarial suite used send_invoice_reminder as its canary for "prefix matching must not sweep up a legitimate method" — it literally starts with send_invoice. Now that it's high-risk, it can't serve as a low-risk canary. And after this widening, no real low-risk method contains a high-risk method name as a substring at all, so that property has no real-tool canary left in the catalog.

Rather than quietly drop the property, it's probed with two explicitly synthetic names (send_invoice_extended, generate_invoice_qr_code_v2), marked as synthetic in the case description so nobody mistakes them for tools this toolkit ships. Losing a test because the thing it tested stopped existing is fine; losing it silently is not.

Everything else is unchanged — create_order and create_invoice still stay low-risk on purpose (a draft with no external effect is a different risk shape), the create_refund/get_refund README gap is still flagged and still not fixed here, and pay_order's missing amount parameter is still disclosed rather than worked around.

Same ask as the original post: whether this is a gap worth closing in-tree, and whether the line is drawn where you'd draw it. The four above are the ones I'd most expect a PayPal reviewer to have an opinion on.