Skip to content

feat: audio support for google models with integration into playground #2213

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
steebchen merged 15 commits into from
May 12, 2026
Merged

feat: audio support for google models with integration into playground #2213

Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
2dc042a
feat: support Google Gemini audio input
RATCHAW May 8, 2026
9484c9d
fix(gateway): audio input routing and token tracking
RATCHAW May 9, 2026
beecbd3
Merge branch 'main' into feat/google-audio-input
RATCHAW May 9, 2026
6a67b37
feat(api): add audio input cost tracking to activity and logs
RATCHAW May 9, 2026
7eafe27
fix(ui): add audioInputCost to activity types
RATCHAW May 9, 2026
916cf91
Merge branch 'main' into feat/google-audio-input
smakosh May 9, 2026
1b7756b
feat(audio): enhance audio support for playground
RATCHAW May 11, 2026
1a01806
Merge branch 'main' of https://github.com/theopenco/llmgateway into f…
RATCHAW May 11, 2026
b52f7a4
feat(audio): add audio support to chat messages
RATCHAW May 11, 2026
2e2a144
feat(prompt-input): add audio attachment UI with Music2Icon
RATCHAW May 11, 2026
e29b066
chore(api): regenerate typed clients with audios field
RATCHAW May 11, 2026
ad9de7d
chore(db): squash audio columns into single migration
RATCHAW May 11, 2026
fada058
Merge branch 'main' into feat/google-audio-input
RATCHAW May 11, 2026
421ca93
chore(db): align migrations with origin/main
steebchen May 12, 2026
085b587
Merge remote-tracking branch 'origin/main' into feat/google-audio-input
steebchen May 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions apps/api/src/routes/activity.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ const dailyActivitySchema = z.object({
requestCost: z.number(),
dataStorageCost: z.number(),
imageInputCost: z.number(),
audioInputCost: z.number(),
imageOutputCost: z.number(),
videoOutputCost: z.number(),
cachedInputCost: z.number(),
Expand Down Expand Up @@ -252,6 +253,10 @@ activity.openapi(getActivity, async (c) => {
sql<number>`COALESCE(SUM(${apiKeyHourlyStats.imageInputCost}), 0)`.as(
"imageInputCost",
),
audioInputCost:
sql<number>`COALESCE(SUM(${apiKeyHourlyStats.audioInputCost}), 0)`.as(
"audioInputCost",
),
imageOutputCost:
sql<number>`COALESCE(SUM(${apiKeyHourlyStats.imageOutputCost}), 0)`.as(
"imageOutputCost",
Expand Down Expand Up @@ -409,6 +414,7 @@ activity.openapi(getActivity, async (c) => {
const cacheCount = Number(day.cacheCount);
const discountSavings = Number(day.discountSavings);
const imageInputCost = Number(day.imageInputCost);
const audioInputCost = Number(day.audioInputCost);
const imageOutputCost = Number(day.imageOutputCost);
const videoOutputCost = Number(day.videoOutputCost);
const cachedInputCost = Number(day.cachedInputCost);
Expand Down Expand Up @@ -440,6 +446,7 @@ activity.openapi(getActivity, async (c) => {
requestCost,
dataStorageCost,
imageInputCost,
audioInputCost,
imageOutputCost,
videoOutputCost,
cachedInputCost,
Expand Down Expand Up @@ -521,6 +528,10 @@ activity.openapi(getActivity, async (c) => {
sql<number>`COALESCE(SUM(${projectHourlyStats.imageInputCost}), 0)`.as(
"imageInputCost",
),
audioInputCost:
sql<number>`COALESCE(SUM(${projectHourlyStats.audioInputCost}), 0)`.as(
"audioInputCost",
),
imageOutputCost:
sql<number>`COALESCE(SUM(${projectHourlyStats.imageOutputCost}), 0)`.as(
"imageOutputCost",
Expand Down Expand Up @@ -679,6 +690,7 @@ activity.openapi(getActivity, async (c) => {
const requestCost = Number(day.requestCost);
const dataStorageCost = Number(day.dataStorageCost);
const imageInputCost = Number(day.imageInputCost);
const audioInputCost = Number(day.audioInputCost);
const imageOutputCost = Number(day.imageOutputCost);
const videoOutputCost = Number(day.videoOutputCost);
const cachedInputCost = Number(day.cachedInputCost);
Expand Down Expand Up @@ -711,6 +723,7 @@ activity.openapi(getActivity, async (c) => {
requestCost,
dataStorageCost,
imageInputCost,
audioInputCost,
imageOutputCost,
videoOutputCost,
cachedInputCost,
Expand Down
18 changes: 16 additions & 2 deletions apps/api/src/routes/chats.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ const messageSchema = z.object({
role: z.enum(["user", "assistant", "system"]),
content: z.string().nullable(),
images: z.string().nullable(), // JSON string
audios: z.string().nullable(), // JSON string of audio attachments
reasoning: z.string().nullable(), // Reasoning content
tools: z.string().nullable(), // JSON string of tool parts
sequence: z.number(),
Expand All @@ -57,6 +58,7 @@ const sharedMessageSnapshotSchema = z.array(
role: z.enum(["user", "assistant", "system"]),
content: z.string().nullable(),
images: z.string().nullable(),
audios: z.string().nullable().optional(),
reasoning: z.string().nullable(),
tools: z.string().nullable(),
sequence: z.number(),
Expand All @@ -80,13 +82,19 @@ const createMessageSchema = z
role: z.enum(["user", "assistant", "system"]),
content: z.string().optional(),
images: z.string().optional(), // JSON string
audios: z.string().optional(), // JSON string of audio attachments
reasoning: z.string().optional(), // Reasoning content
tools: z.string().optional(), // Tool parts JSON
})
.refine(
(data) => data.content ?? data.images ?? data.reasoning ?? data.tools,
(data) =>
data.content ??
data.images ??
data.audios ??
data.reasoning ??
data.tools,
{
message: "Either content or images must be provided",
message: "Either content, images, or audios must be provided",
Comment on lines +93 to +100
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot May 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Update error message to match validation logic.

The refinement checks for content, images, audios, reasoning, or tools, but the error message only mentions content, images, or audios.

📝 Suggested fix 
 	.refine(
 		(data) =>
 			data.content ??
 			data.images ??
 			data.audios ??
 			data.reasoning ??
 			data.tools,
 		{
-			message: "Either content, images, or audios must be provided",
+			message: "At least one of content, images, audios, reasoning, or tools must be provided",
 		},
 	);
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
(data) =>
data.content ??
data.images ??
data.audios ??
data.reasoning ??
data.tools,
{
message: "Either content or images must be provided",
message: "Either content, images, or audios must be provided",
(data) =>
data.content ??
data.images ??
data.audios ??
data.reasoning ??
data.tools,
{
message: "At least one of content, images, audios, reasoning, or tools must be provided",
},
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/api/src/routes/chats.ts` around lines 93 - 100, The validation
refinement checks data.content ?? data.images ?? data.audios ?? data.reasoning
?? data.tools but the error text only mentions content, images, or audios;
update the error message used where this refinement is defined (the object with
message: "Either content, images, or audios must be provided") to list all
checked fields (e.g., include "reasoning" and "tools") so the message matches
the validation logic in that refinement.

},
);

Expand Down Expand Up @@ -484,6 +492,7 @@ chats.openapi(getChat, async (c) => {
role: message.role as "user" | "assistant" | "system",
content: message.content,
images: message.images,
audios: (message as any).audios ?? null,
reasoning: message.reasoning,
tools: (message as any).tools ?? null,
sequence: message.sequence,
Expand Down Expand Up @@ -657,6 +666,7 @@ chats.openapi(shareChat, async (c) => {
role: tables.message.role,
content: tables.message.content,
images: tables.message.images,
audios: tables.message.audios,
reasoning: tables.message.reasoning,
tools: tables.message.tools,
sequence: tables.message.sequence,
Expand All @@ -678,6 +688,7 @@ chats.openapi(shareChat, async (c) => {
role: message.role,
content: message.content,
images: message.images,
audios: message.audios,
reasoning: message.reasoning,
tools: message.tools,
sequence: message.sequence,
Expand Down Expand Up @@ -886,6 +897,7 @@ chats.openapi(forkSharedChat, async (c) => {
role: message.role,
content: message.content,
images: message.images,
audios: message.audios ?? null,
reasoning: message.reasoning,
tools: message.tools,
sequence: message.sequence,
Expand Down Expand Up @@ -1052,6 +1064,7 @@ chats.openapi(addMessage, async (c) => {
role: body.role,
content: body.content ?? null,
images: body.images ?? null,
audios: body.audios ?? null,
reasoning: body.reasoning ?? null,
tools: body.tools ?? null,
sequence: nextSequence,
Expand All @@ -1071,6 +1084,7 @@ chats.openapi(addMessage, async (c) => {
role: newMessage.role as "user" | "assistant" | "system",
content: newMessage.content,
images: newMessage.images,
audios: (newMessage as any).audios ?? null,
reasoning: newMessage.reasoning,
tools: (newMessage as any).tools ?? null,
sequence: newMessage.sequence,
Expand Down
2 changes: 2 additions & 0 deletions apps/api/src/routes/internal-models.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ const modelProviderMappingSchema = z.object({
maxOutput: z.number().nullable(),
streaming: z.boolean(),
vision: z.boolean().nullable(),
audio: z.boolean().nullable(),
reasoning: z.boolean().nullable(),
reasoningOutput: z.string().nullable(),
tools: z.boolean().nullable(),
Expand Down Expand Up @@ -219,6 +220,7 @@ internalModels.openapi(getModelsRoute, async (c) => {
return {
...mapping,
discount: effectiveDiscount,
audio: sharedMapping?.audio ?? null,
imageOutputPrice:
sharedMapping?.imageOutputPrice !== undefined
? String(sharedMapping.imageOutputPrice)
Expand Down
2 changes: 2 additions & 0 deletions apps/api/src/routes/logs.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,8 +136,10 @@ const logSchema = z.object({
cacheWriteInputCost: z.number().nullable().optional(),
webSearchCost: z.number().nullable().optional(),
imageInputTokens: z.string().nullable(),
audioInputTokens: z.string().nullable(),
imageOutputTokens: z.string().nullable(),
imageInputCost: z.number().nullable(),
audioInputCost: z.number().nullable(),
imageOutputCost: z.number().nullable(),
videoOutputCost: z.number().nullable(),
videoDownloadCount: z.number().nullable(),
Expand Down
1 change: 1 addition & 0 deletions apps/api/src/routes/public-chat-shares.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ const sharedMessageSchema = z.object({
role: z.enum(["user", "assistant", "system"]),
content: z.string().nullable(),
images: z.string().nullable(),
audios: z.string().nullable().optional(),
reasoning: z.string().nullable(),
tools: z.string().nullable(),
sequence: z.number(),
Expand Down
10 changes: 10 additions & 0 deletions apps/code/src/lib/api/v1.d.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -527,6 +527,7 @@ export interface paths {
role: "user" | "assistant" | "system";
content: string | null;
images: string | null;
audios?: string | null;
reasoning: string | null;
tools: string | null;
sequence: number;
Expand Down Expand Up @@ -1083,8 +1084,10 @@ export interface paths {
cacheWriteInputCost?: number | null;
webSearchCost?: number | null;
imageInputTokens: string | null;
audioInputTokens: string | null;
imageOutputTokens: string | null;
imageInputCost: number | null;
audioInputCost: number | null;
imageOutputCost: number | null;
videoOutputCost: number | null;
videoDownloadCount: number | null;
Expand Down Expand Up @@ -1334,8 +1337,10 @@ export interface paths {
cacheWriteInputCost?: number | null;
webSearchCost?: number | null;
imageInputTokens: string | null;
audioInputTokens: string | null;
imageOutputTokens: string | null;
imageInputCost: number | null;
audioInputCost: number | null;
imageOutputCost: number | null;
videoOutputCost: number | null;
videoDownloadCount: number | null;
Expand Down Expand Up @@ -1469,6 +1474,7 @@ export interface paths {
requestCost: number;
dataStorageCost: number;
imageInputCost: number;
audioInputCost: number;
imageOutputCost: number;
videoOutputCost: number;
cachedInputCost: number;
Expand Down Expand Up @@ -7403,6 +7409,7 @@ export interface paths {
role: "user" | "assistant" | "system";
content: string | null;
images: string | null;
audios: string | null;
reasoning: string | null;
tools: string | null;
sequence: number;
Expand Down Expand Up @@ -7657,6 +7664,7 @@ export interface paths {
role: "user" | "assistant" | "system";
content?: string;
images?: string;
audios?: string;
reasoning?: string;
tools?: string;
};
Expand All @@ -7676,6 +7684,7 @@ export interface paths {
role: "user" | "assistant" | "system";
content: string | null;
images: string | null;
audios: string | null;
reasoning: string | null;
tools: string | null;
sequence: number;
Expand Down Expand Up @@ -9324,6 +9333,7 @@ export interface operations {
maxOutput: number | null;
streaming: boolean;
vision: boolean | null;
audio: boolean | null;
reasoning: boolean | null;
reasoningOutput: string | null;
tools: boolean | null;
Expand Down
17 changes: 17 additions & 0 deletions apps/gateway/src/app.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import { cors } from "hono/cors";
import { HTTPException } from "hono/http-exception";
import { z } from "zod";

import { UnsupportedAudioFormatError } from "@llmgateway/actions";
import { redisClient } from "@llmgateway/cache";
import { db } from "@llmgateway/db";
import {
Expand Down Expand Up @@ -112,6 +113,22 @@ app.use("*", async (c, next) => {
});

app.onError((error, c) => {
if (error instanceof UnsupportedAudioFormatError) {
logger.warn("Unsupported audio format", {
message: error.message,
format: error.format,
providerTarget: error.providerTarget,
});
return c.json(
{
error: true,
status: 400,
message: error.message,
},
400,
);
}

if (error instanceof HTTPException) {
const status = error.status;

Expand Down
Loading
Loading