Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,12 @@ This changelog documents all the different updates that occur for this framework
### Added

- **The RAG Builder offers Weaviate.** A Weaviate setup takes `WEAVIATE_RAG_USER`/`_PW`, `WEAVIATE_HOST` and `WEAVIATE_PORT` from the credential domain and needs no database. The ingestion job, the Studio flow and the retrieval model it manifests are set up for Weaviate. The new option *RAG: offer Weaviate* (`enable_weaviate`) withholds it.
- **Gemini Decision 1 as a decision model.** `gemini_decision_1` calls Google's Decisions API with the `Google` key entry. The new `dec_gemini` score template translates inside the SCR container (state as `input`, key in `x-goog-api-key`, Google's token counts), so its callers keep the System One contract. `mdb add google gemini-decision-1 --kind decision` finds it in Google's model list; OpenRouter's snapshot lists `google/gemini-decision-1` too.

### Fixed

- **A decision model's `answer` and `confidence` answer the first question.** They took the first answer in the provider's order, and Gemini Decision 1 returns the answers in another order than the questions were asked. Re-register and republish a decision model to bring its scorer up to date.
- **Models start on the current SCR base image.** Their images pinned `numpy==1.26.4`, which breaks the base image's pandas 3.1 (Python 3.11): the container exited at start with "C extension: None not built". API models no longer install numpy; open-weight models, and the hand-kept requirements of Llama 3.2, Mistral Nemo and Phi-3/3.5, keep the base image's numpy. Run `mdb register --update` before you republish one.

## [2.5.0] - 2026-11-10

Expand Down
4 changes: 3 additions & 1 deletion Decision-Definitions/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ A decision model (a *System One* model, after TypeSafe's Jev) does not generate
| `noul` | Does this condition hold? | the probability that it does |
| `score` | Where on this ordered scale? | the probability-weighted position, a probability per level, a confidence |

That is the shape a SAS Intelligent Decisioning branch node wants: a value to branch on and a confidence to gate it with. Every provider in this family speaks the same wire format (`POST .../v1/systemone`), so one score template covers TypeSafe's own API, OpenRouter (which hosts Jev) and self-hosted reimplementations; Von additionally runs inside the container with its checkpoint baked into the image.
That is the shape a SAS Intelligent Decisioning branch node wants: a value to branch on and a confidence to gate it with. Every provider in this family speaks the same wire format (`POST .../v1/systemone`), so one score template covers TypeSafe's own API, OpenRouter (which hosts Jev) and self-hosted reimplementations; Google's Decisions API (Gemini Decision 1) asks the same questions in a slightly different request, which its own template translates inside the container; Von additionally runs inside the container with its checkpoint baked into the image.

Each subfolder here contains the definition for one decision model - the name of the folder is the model id. A folder is generated from its `definition.yaml` by the [Model Definition Builder](../Model-Definition-Builder/README.md); the other files are derived from it and are not edited by hand.

Expand All @@ -38,6 +38,7 @@ A call through the SCR endpoint, the same envelope every other container of the
mdb add openrouter typesafe/jev-1.13 --kind decision --id jev_1_13 --yes # Jev through OpenRouter
mdb add openrouter cloudflare/clef-flash --kind decision --id clef_flash --yes # any other decision model OpenRouter hosts
mdb add typesafe jev-latest --id jev_latest --yes # Jev through TypeSafe's own API
mdb add google gemini-decision-1 --kind decision --id gemini_decision_1 --yes # Gemini Decision 1 through Google's API
mdb add von --id von_1_2 --yes # Von, self-hosted in the container
```

Expand All @@ -51,6 +52,7 @@ OpenRouter's public model listing omits its decision models, so they come from t
| `clef_flash` | Clef-flash by Cloudflare (Apache-2.0) - the most accurate measured on many-option intent sets | OpenRouter | the `OpenRouter` entry |
| `liquid_d1` | Liquid D1 by Liquid AI - the cheapest hosted decision model measured | OpenRouter | the `OpenRouter` entry |
| `pplx_decider_27b` | Perplexity Decider 27B - a 256k context | OpenRouter | the `OpenRouter` entry |
| `gemini_decision_1` | Gemini Decision 1 by Google - not measured yet | Google | the `Google` entry |
| `mercury_decide` | Mercury Decide by Inception - not measured yet | OpenRouter | the `OpenRouter` entry |
| `von_1_2` | Von 1.2 (Apache-2.0, ModernBERT-large) | inside the container | none |

Expand Down
6 changes: 3 additions & 3 deletions Decision-Definitions/clef_flash/.mdb-lock.json
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
{
"generator": "mdb 2.3.1",
"generator": "mdb 2.5.0",
"manifest_sha256": "e67f05e9236d202dfa6705707bed86cbd53c3ad3912bed2246dc7a982efb9942",
"files": {
"Model-Card.md": "da6280c86f0c53f9c8ecb1ef286e8b5560defb71052aede1b066352833e1680a",
"README.md": "f7b45b0845876dfb97e4efb5a9ca4b2c12feb793a0f4602187baf4b7f2229151",
"clefFlashScore.py": "439fae68dd2c38554cdeef2137cbbbf1e420210ff1836093f3cc0edb57f9a152",
"clefFlashScore.py": "1fdca2f7248997832eecf516203923e7b6144e257e58f611ad775c3223b3f768",
"inputVar.json": "de2cdec7ea81011e78fd0f553b1e2d060a135b1d3911a8b3f83909a9db4f96f3",
"modelConfiguration.json": "d7d010a2226c098db910248ed9ef37087897c64413a42c7b7ccfc3926ec41a7c",
"options.json": "a96a32ee784e40271b6561fecd5076064dfc155550872ebd4f1d6f92b8e21303",
"outputVar.json": "44d1ba094471b76a2af5ad0b5106e11cc97111005d4f27211a772217222a3412",
"requirements.json": "23644c62b8ebaa9a08d4b6df1bb422a841ba0fa3da256b9e2fe01acd8d4ca483"
"requirements.json": "9a33900057c3c903a9be658218e8b3f5e51d84055153fd81a6548705f1e3a988"
}
}
14 changes: 11 additions & 3 deletions Decision-Definitions/clef_flash/clefFlashScore.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# GENERATED by mdb 2.3.1 from definition.yaml - edit definition.yaml and run 'mdb generate clef_flash' instead of editing this file.
# GENERATED by mdb 2.5.0 from definition.yaml - edit definition.yaml and run 'mdb generate clef_flash' instead of editing this file.
import time
import json
import logging
Expand Down Expand Up @@ -113,6 +113,15 @@ def _headline(answer):
return str(answer.get("choice", "")), float(answer.get("confidence", 0.0))


def _first_answer(answers, questions):
# The answer to the first question as the caller asked it - a provider may
# return the answers in another order (Gemini Decision 1 does).
if not isinstance(answers, dict):
return {}
first = next(iter(questions), None) if isinstance(questions, dict) else None
return answers.get(first) or next(iter(answers.values()), {})


def scoreModel(state, questions, options):
"Output: answers, answer, confidence, run_time, prompt_length, output_length"
started_timestamp = time.time()
Expand Down Expand Up @@ -151,8 +160,7 @@ def scoreModel(state, questions, options):
logger.error(f"HTTP {responseObject.status_code}: {json.dumps(responseJson)}")
raise RuntimeError(f"Request failed: {responseObject.status_code}")
answersObject = responseJson.get('answers') or {}
firstAnswer = next(iter(answersObject.values()), {}) if isinstance(answersObject, dict) else {}
answer, confidence = _headline(firstAnswer)
answer, confidence = _headline(_first_answer(answersObject, payload["questions"]))
answers = json.dumps(answersObject)
usage = responseJson.get('usage') or {}
prompt_length = usage.get('input_tokens', 0)
Expand Down
2 changes: 1 addition & 1 deletion Decision-Definitions/clef_flash/requirements.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,6 @@
},
{
"step": "install common packages",
"command": "pip3 -q install requests numpy==1.26.4"
"command": "pip3 -q install requests"
}
]
1 change: 1 addition & 0 deletions Decision-Definitions/decision_fact_sheet.csv
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
model_id,model,provider,description,release_date,size,deployment_type,license,cost_type,input_token_price,second_cost,context_length,max_options,question_types
"clef_flash","Clef-flash","OpenRouter","Clef-flash by Cloudflare - an open (Apache-2.0) System One decision model built on Qwen that answers typed questions about a state with calibrated probabilities; the most accurate hosted choice for many-option intent sets, served through the OpenRouter API.",2026-10-01,.,"API","Open-Source","Tokens",0.00000009,.,65536,255,"choice|noul|score"
"gemini_decision_1","Gemini Decision 1","Google","Gemini Decision 1 by Google - a decision (System One) model that answers typed questions about a state with calibrated probabilities instead of text, served directly through Google's Decisions API.",2026-11-11,.,"API","Proprietary","Tokens",0.0000001,.,262144,255,"choice|noul|score"
"jev_1_13","Jev 1.13","OpenRouter","Jev 1.13 by TypeSafe - a decision (System One) model that answers typed questions about a state with calibrated probabilities instead of text, served through the OpenRouter API.",2026-09-15,.,"API","Proprietary","Tokens",0.000000042,.,32000,255,"choice|noul|score"
"liquid_d1","Liquid D1","OpenRouter","Liquid D1 by Liquid AI - a System One decision model that answers typed questions about a state with calibrated probabilities; the cheapest hosted decision model per call, served through the OpenRouter API.",.,.,"API","Proprietary","Tokens",0.00000004,.,65536,255,"choice|noul|score"
"mercury_decide","Mercury Decide","OpenRouter","Mercury Decide by Inception - a System One decision model that answers typed questions about a state with calibrated probabilities, served through the OpenRouter API.",2026-09-30,.,"API","Proprietary","Tokens",0.00000002,.,65536,255,"choice|noul|score"
Expand Down
14 changes: 14 additions & 0 deletions Decision-Definitions/gemini_decision_1/.mdb-lock.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"generator": "mdb 2.5.0",
"manifest_sha256": "8358d6f0938065fd29abf3ba1088aca57fecdc39c44c0523ed7c51c517ad183f",
"files": {
"Model-Card.md": "e31029a657a233ba995ebdb724835c60d8caaddf1356bfb13bbbfeee4b5ad3c7",
"README.md": "acb3698a0c81e5541cfc21fbea9ed3a048ec96088cc4e13db686dfcf0e029607",
"geminiDecision1Score.py": "996e76665252b9c4b1bbc30bfea8013866891d83251ed75f5036c3d5b5fe6e89",
"inputVar.json": "de2cdec7ea81011e78fd0f553b1e2d060a135b1d3911a8b3f83909a9db4f96f3",
"modelConfiguration.json": "c9c5650f7688e070b16507106deb713cd9737e6714281ed15a641d11ffb58760",
"options.json": "6138c58b7157bd5e89a2ac03cf0b4ad86b5127d8b9e9f064a3989aeb87ebf830",
"outputVar.json": "44d1ba094471b76a2af5ad0b5106e11cc97111005d4f27211a772217222a3412",
"requirements.json": "9a33900057c3c903a9be658218e8b3f5e51d84055153fd81a6548705f1e3a988"
}
}
37 changes: 37 additions & 0 deletions Decision-Definitions/gemini_decision_1/Model-Card.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Model Card: Gemini Decision 1

## Model Details

- **Model:** Gemini Decision 1 (`gemini-decision-1`)
- **Provider:** Google
- **Release date:** 2026-11-11
- **License:** Proprietary
- **Deployment type:** API
- **Context length:** 262144
- **Knowledge cutoff:** unknown

## Description

Gemini Decision 1 by Google - a decision (System One) model that answers typed questions about a state with calibrated probabilities instead of text, served directly through Google's Decisions API.

## Pricing

Input: $0.1 per 1M tokens - Output: $0 per 1M tokens

## Intended Use

This model is registered in SAS Model Manager as part of the SAS Agentic AI Accelerator and is intended to be used through the framework's scoring contract (`state`, `questions`, `options` → `answers`, `answer`, `confidence`, `run_time`, `prompt_length`, `output_length`), for example from SAS Intelligent Decisioning (a branch node reads `answer` and `confidence`) or any program that posts a state and typed questions to the SCR endpoint.

## Deployment

The model runs as a SAS Container Runtime (SCR) image published from SAS Model Manager and is called through its SCR endpoint. What the container needs from its environment:

- The key arrives per call in the `API_KEY` option.
- The container keeps the System One contract for its callers and translates it for Google's Decisions API: the state is sent as the `input` text, the questions unchanged, the key in the `x-goog-api-key` header. It needs network access to `generativelanguage.googleapis.com`.

## Limitations and Responsible Use

Refer to the `modelConfiguration.json` of this definition for the framework's documented purpose, intended use, expected benefits, out-of-scope use cases and limitations, and consult the provider's own model documentation for model-specific evaluations and safety characteristics.

---
*Generated by the Model Definition Builder from `definition.yaml`; metadata prefilled from google (live, enriched from static snapshot 2026-11-12).*
40 changes: 40 additions & 0 deletions Decision-Definitions/gemini_decision_1/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Gemini Decision 1

Gemini Decision 1 by Google - a decision (System One) model that answers typed questions about a state with calibrated probabilities instead of text, served directly through Google's Decisions API.

This definition was generated by the [Model Definition Builder](../../Model-Definition-Builder/README.md) from `definition.yaml` — edit that file and run `mdb generate gemini_decision_1` instead of editing the generated files by hand.

## Model facts

| Attribute | Value |
| --------- | ----- |
| Provider | Google |
| Provider model | `gemini-decision-1` |
| Deployment type | API |
| License | Proprietary |
| Context length | 262144 |
| Knowledge cutoff | unknown |

## Configuration options

The following options can be supplied at scoring time through the `options` input (see the [LLM-Definitions README](../README.md) for the format):

| Option | Default | Range | Description |
| ------ | ------- | ----- | ----------- |
| `API_KEY` | Google | sk-**** | This is the Google key that is used to make the actual request. |

## Deployment

What the SCR container needs from its environment (the caller-side options are in the table above):

- The key arrives per call in the `API_KEY` option.
- The container keeps the System One contract for its callers and translates it for Google's Decisions API: the state is sent as the `input` text, the questions unchanged, the key in the `x-goog-api-key` header. It needs network access to `generativelanguage.googleapis.com`.

## Register and publish

```bash
mdb register gemini_decision_1
mdb publish gemini_decision_1 -d <your-destination>
```

Or do both in one step with `mdb ship gemini_decision_1`. The publishing destination can also come from `SAS_PUBLISH_DESTINATION` in your `.env`.
39 changes: 39 additions & 0 deletions Decision-Definitions/gemini_decision_1/definition.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
schema: 1
kind: decision
model_id: gemini_decision_1
display_name: Gemini Decision 1
provider:
adapter: google
model_version: gemini-decision-1
endpoint: https://generativelanguage.googleapis.com/v1beta/decisions
params: {}
auth:
mode: api_key
key_name: Google
runtime:
template: dec_gemini
requirements_profile: api-wrapper
timeout_s: 60
weights_source: baked
options: {}
tags:
size_class: Decision
license_class: Proprietary
provider_tag: Google
scr_sizing: small
extra: []
metadata:
description: Gemini Decision 1 by Google - a decision (System One) model that answers typed questions
about a state with calibrated probabilities instead of text, served directly through Google's Decisions
API.
release_date: '2026-11-11'
context_length: 262144
deployment_type: API
pricing:
cost_type: Tokens
input_token_price: 1.0e-07
output_token_price: 0.0
modeler: SAS Agentic AI Accelerator
generation:
overrides: []
catalog_provenance: google (live, enriched from static snapshot 2026-11-12)
Loading
Loading