Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions apps/web/src/components/chat/message-activity-model.ts
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,8 @@ const TOOL_VERBS: Record<string, ToolVerbSpec> = {
docs_generate_pdf: { verb: "Generated a PDF" },
docs_generate_slides: { verb: "Generated slides" },
docs_generate_xlsx: { verb: "Generated a spreadsheet" },
generate_or_edit_image: { verb: "Generated or edited an image" },
generate_or_extend_video: { verb: "Generated or extended a video" },
search_scrape: { verb: "Scraped", argKeys: ["url"] },
search_web_content: { verb: "Searched the web", argKeys: ["query", "q"] },
search_extract: { verb: "Extracted", argKeys: ["url"] },
Expand Down
7 changes: 5 additions & 2 deletions packages/agent-core/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,11 @@ Sandbox and artifact capabilities cross tool-domain boundaries only through
Single-consumer data, document, and media implementations live under
`src/tools/`. Data tools profile and normalize bounded tabular inputs and render
deterministic SVG/Recharts output. Document tools generate sandbox-side PPTX,
DOCX, XLSX, and PDF source against `/opt/cheatcode-doc-runtime`. Media and Google-backed
browser tools resolve the user's Google AI BYOK key lazily when invoked. All three receive
DOCX, XLSX, and PDF source against `/opt/cheatcode-doc-runtime`; routine XLSX generation
supports bounded scalar and workbook-local formula cells with cached results and standard number
formats, while complex models remain on the custom Office path. Image and video generation use
separate strict contracts so one mode cannot acquire the other mode's arguments. Media and
Google-backed browser tools resolve the user's Google AI BYOK key lazily when invoked. All three receive
sandbox and R2 artifact capabilities through request-scoped contracts; they do
not read environment variables, persist credentials, or log keys.
Published media is rendered automatically in the chat artifact card and is available in Files;
Expand Down
5 changes: 4 additions & 1 deletion packages/agent-core/src/mastra/agent-tool-policy.ts
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,10 @@ const DOCUMENT_TOOLS = [
"docs_generate_xlsx",
] as const satisfies readonly ToolCapabilityName[];

const MEDIA_TOOLS = ["generate_or_edit_media"] as const satisfies readonly ToolCapabilityName[];
const MEDIA_TOOLS = [
"generate_or_edit_image",
"generate_or_extend_video",
] as const satisfies readonly ToolCapabilityName[];

const RESEARCH_TOOLS = [
"search_extract",
Expand Down
6 changes: 3 additions & 3 deletions packages/agent-core/src/mastra/system-prompt.ts
Original file line number Diff line number Diff line change
Expand Up @@ -322,15 +322,15 @@ This project is scaffolded at the workspace root and its dev server + live previ

const DOCS_MODULE = `## Building documents & slides

Build decks and docs from scratch with the preinstalled libraries — pptxgenjs for .pptx, docx, @react-pdf/renderer, exceljs — when the request needs custom layout control, or use docs_generate_slides / docs_generate_docx / docs_generate_pdf / docs_generate_xlsx for a fast structured deliverable. Preserve exact requested counts and scope. The structured generators validate their bounded layouts, publish the file, return its canonical read-only project path, and report the authoritative slide, section, or sheet count. For routine structured output, confirm that returned count and finish; do not search for the file, convert it, start a local server, or take screenshots. Use the custom path and inspect real renders when the user explicitly requests visual QA, the layout is custom or template-driven, or the document is high-stakes. In that path, check for faint text, overflow, placeholder text, and unfilled sections; re-render only affected pages after a fix. The file lands in the Deliverables automatically — refer to it naturally ("your deck is ready below"), don't paste a download link.`;
Build decks and docs from scratch with the preinstalled libraries — pptxgenjs for .pptx, docx, @react-pdf/renderer, exceljs — when the request needs custom layout control, or use docs_generate_slides / docs_generate_docx / docs_generate_pdf / docs_generate_xlsx for a fast structured deliverable. Preserve exact requested counts and scope. The XLSX generator accepts scalar cells plus typed formula cells with cached results and number formats. The structured generators validate their bounded layouts, publish the file, return its canonical read-only project path, and report the authoritative slide, section, or sheet count. For routine structured output, confirm that returned count and finish; do not search for the file, convert it, start a local server, or take screenshots. Use the custom path and inspect real renders when the user explicitly requests visual QA, the layout is custom or template-driven, or the document is high-stakes. In that path, check for faint text, overflow, placeholder text, and unfilled sections; re-render only affected pages after a fix. The file lands in the Deliverables automatically — refer to it naturally ("your deck is ready below"), don't paste a download link.`;

const DATA_MODULE = `## Data & analysis

For a quick question, compute the answer and just tell the user — a small code_run or data_analyze_csv is enough; don't build a chart or open a browser unless asked. For a real analysis, profile the data (data_analyze_csv, or pandas / Node in the sandbox), surface the key findings, and build charts (data_chart) only when they add insight. Verify: open the produced file, confirm the numbers reconcile and formulas evaluate (no #REF! / #DIV/0!), and sanity-check every chart against the data.`;

const MEDIA_MODULE = `## Images & video

Load the generate-media skill before creating or editing an image or generating or extending a video. Use generate_or_edit_media for the final asset, choose the requested aspect ratio intentionally, preserve stated invariants when editing, and refer to the resulting Deliverable naturally without pasting its download URL. A successful result is rendered automatically in the chat and can be opened in Files; do not open its sandbox path in the browser or claim that the user cannot preview it.`;
Load the generate-media skill before creating or editing an image or generating or extending a video. Use generate_or_edit_image for an image and generate_or_extend_video for a video. Call the selected tool once, choose the requested aspect ratio intentionally, preserve stated invariants when editing, and refer to the resulting Deliverable naturally without pasting its download URL. A successful result is rendered automatically in the chat and can be opened in Files; do not open its sandbox path in the browser or claim that the user cannot preview it. If the provider rejects or fails the request, report that failure instead of retrying the same request or substituting a hand-authored asset.`;

const RESEARCH_MODULE = `## Research

Expand All @@ -343,7 +343,7 @@ Pick the path that fits and load the matching skill (skill_invoke) for its full
- Web or mobile app → build it in the sandbox (React / Next.js, or Expo for mobile), start the dev server, and verify it in the browser; the running app shows in the Computer panel.
- Slides or documents → use the fast structured generator when it fits, or the document libraries for custom control; preserve exact counts and verify proportionately before finishing. The file lands in the Deliverables.
- Data → profile it (data_analyze_csv, or pandas / Node) and chart it (data_chart) when it adds insight; verify the numbers.
- Image or video → load generate-media, then use generate_or_edit_media; the asset lands in the project and Deliverables and renders automatically in chat and Files.
- Image or video → load generate-media, then use generate_or_edit_image for an image or generate_or_extend_video for a video; the asset lands in the project and Deliverables and renders automatically in chat and Files.
- Research → gather and cross-check real sources (search_web / firecrawl_* / research_deep); cite everything, and use the deep-research workflow when the user asks for a report so its PDF is delivered automatically.
- Acting in the user's connected apps → composio_list_tools then composio_execute, only when they ask.`;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ export const mastraDocsGenerateDocx = createTool({
export const mastraDocsGenerateXlsx = createTool({
id: "docs_generate_xlsx",
description:
"Generate an XLSX workbook from sheets, columns, and rows. Publishes it as a Deliverable and returns its exact project filePath.",
"Generate an XLSX workbook from sheets, columns, scalar cells, and typed formula cells with cached results and number formats. Publishes it as a Deliverable and returns its exact project filePath; routine workbooks need no custom script or recalculation pass.",
inputSchema: GenerateSpreadsheetInputSchema,
outputSchema: GenerateXlsxOutputSchema,
execute: async (input, context) =>
Expand Down
45 changes: 34 additions & 11 deletions packages/agent-core/src/mastra/tool-defs/media-tools.ts
Original file line number Diff line number Diff line change
@@ -1,24 +1,47 @@
import { createTool } from "@mastra/core/tools";
import { executeGenerateOrEditMedia } from "../../tools/media/execute";
import {
GenerateOrEditMediaInputSchema,
GenerateOrEditMediaOutputSchema,
executeGenerateOrEditImage,
executeGenerateOrExtendVideo,
} from "../../tools/media/execute";
import {
GenerateImageOutputSchema,
GenerateOrEditImageInputSchema,
GenerateOrExtendVideoInputSchema,
GenerateVideoOutputSchema,
} from "../../tools/media/schemas";
import { resolveGoogleToolApiKey } from "./request-context";
import { requestContextFromToolContext, workspaceRuntimeFromContext } from "./tool-runtime-context";

/** Generates or edits media after lazily resolving the user's Google AI tool key. */
export const mastraGenerateOrEditMedia = createTool({
id: "generate_or_edit_media",
/** Generates or edits one image through a contract that cannot accept video-only fields. */
export const mastraGenerateOrEditImage = createTool({
id: "generate_or_edit_image",
description:
"Generate a new image or edit/reference provided images with Google AI. Publishes the result to the project and Deliverables in one bounded call.",
inputSchema: GenerateOrEditImageInputSchema,
outputSchema: GenerateImageOutputSchema,
execute: async (input, context) => {
const requestContext = requestContextFromToolContext(context);
const googleApiKey = await resolveGoogleToolApiKey(requestContext);
return executeGenerateOrEditImage(
GenerateOrEditImageInputSchema.parse(input),
await workspaceRuntimeFromContext(context),
googleApiKey ?? "",
);
},
});

/** Generates or extends one video through a contract that cannot accept image-edit fields. */
export const mastraGenerateOrExtendVideo = createTool({
id: "generate_or_extend_video",
description:
"Generate or edit an image, or generate/extend a video, using Google media models. Stores the result in the project and Deliverables.",
inputSchema: GenerateOrEditMediaInputSchema,
outputSchema: GenerateOrEditMediaOutputSchema,
"Generate a new video or extend a provided video with Google AI. Publishes the result to the project and Deliverables in one bounded call.",
inputSchema: GenerateOrExtendVideoInputSchema,
outputSchema: GenerateVideoOutputSchema,
execute: async (input, context) => {
const requestContext = requestContextFromToolContext(context);
const googleApiKey = await resolveGoogleToolApiKey(requestContext);
return executeGenerateOrEditMedia(
GenerateOrEditMediaInputSchema.parse(input),
return executeGenerateOrExtendVideo(
GenerateOrExtendVideoInputSchema.parse(input),
await workspaceRuntimeFromContext(context),
googleApiKey ?? "",
);
Expand Down
2 changes: 1 addition & 1 deletion packages/agent-core/src/mastra/tool-defs/registry.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ export {
mastraDocsGenerateSlides,
mastraDocsGenerateXlsx,
} from "./document-tools";
export { mastraGenerateOrEditMedia } from "./media-tools";
export { mastraGenerateOrEditImage, mastraGenerateOrExtendVideo } from "./media-tools";
export {
mastraDeepResearch,
mastraFirecrawlExtract,
Expand Down
6 changes: 4 additions & 2 deletions packages/agent-core/src/mastra/tool-defs/tool-set.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@ import {
mastraFsRead,
mastraFsSearch,
mastraFsWrite,
mastraGenerateOrEditMedia,
mastraGenerateOrEditImage,
mastraGenerateOrExtendVideo,
mastraGitClone,
mastraGitCommit,
mastraGitPush,
Expand Down Expand Up @@ -74,7 +75,8 @@ export const cheatcodeTools = {
fs_read: mastraFsRead,
fs_search: mastraFsSearch,
fs_write: mastraFsWrite,
generate_or_edit_media: mastraGenerateOrEditMedia,
generate_or_edit_image: mastraGenerateOrEditImage,
generate_or_extend_video: mastraGenerateOrExtendVideo,
git_clone: mastraGitClone,
git_commit: mastraGitCommit,
git_push: mastraGitPush,
Expand Down
27 changes: 26 additions & 1 deletion packages/agent-core/src/tools/docs/schemas.ts
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,26 @@ export const GenerateMarkdownPdfInputSchema = z.strictObject({
title: TextValueSchema.optional(),
});

const SpreadsheetCellSchema = z.union([z.string(), z.number(), z.boolean(), z.null()]);
const SpreadsheetScalarSchema = z.union([
z.string().max(10_000),
z.number(),
z.boolean(),
z.null(),
]);

const SpreadsheetFormulaSchema = z.strictObject({
formula: z
.string()
.trim()
.min(1)
.max(1_000)
.refine(isSafeSpreadsheetFormula, "Formula must use workbook-local values and functions."),
numberFormat: z.enum(["general", "integer", "decimal", "percent", "currency"]).optional(),
result: z.union([z.string().max(10_000), z.number(), z.boolean()]),
type: z.literal("formula"),
});

const SpreadsheetCellSchema = z.union([SpreadsheetScalarSchema, SpreadsheetFormulaSchema]);
const SpreadsheetRowSchema = z.record(z.string().min(1).max(80), SpreadsheetCellSchema);

const SpreadsheetSheetSchema = z.strictObject({
Expand Down Expand Up @@ -107,3 +126,9 @@ export type GenerateSlidesInput = z.input<typeof GenerateSlidesInputSchema>;
export type GenerateSlidesOutput = z.output<typeof GenerateSlidesOutputSchema>;
export type GenerateSpreadsheetInput = z.input<typeof GenerateSpreadsheetInputSchema>;
export type GenerateXlsxOutput = z.output<typeof GenerateXlsxOutputSchema>;

function isSafeSpreadsheetFormula(value: string): boolean {
const formula = value.startsWith("=") ? value.slice(1) : value;
const blockedFunction = /\b(?:CALL|DDE|EXEC|FILTERXML|HYPERLINK|RTD|WEBSERVICE)\s*\(/i;
return formula.length > 0 && !/[[\]"'|@]/.test(formula) && !blockedFunction.test(formula);
}
22 changes: 21 additions & 1 deletion packages/agent-core/src/tools/docs/scripts.ts
Original file line number Diff line number Diff line change
Expand Up @@ -107,14 +107,34 @@ export function buildXlsxScript(inputPath: string, outputPath: string, filename:
'workbook.creator = "Cheatcode";',
"workbook.created = new Date();",
"workbook.title = input.title;",
"const numberFormats = {",
' general: "General",',
' integer: "#,##0",',
' decimal: "#,##0.00",',
' percent: "0.0%",',
' currency: "$#,##0.00;($#,##0.00);-",',
"};",
"function cellValue(value) {",
' if (!value || typeof value !== "object" || value.type !== "formula") return value;',
' return { formula: value.formula.replace(/^=/, ""), result: value.result };',
"}",
"for (const sheet of input.sheets) {",
" const worksheet = workbook.addWorksheet(sheet.name);",
" worksheet.columns = sheet.columns.map((column) => ({",
" header: column,",
" key: column,",
" width: Math.max(12, Math.min(36, column.length + 8)),",
" }));",
" worksheet.addRows(sheet.rows);",
" for (const inputRow of sheet.rows) {",
" const values = Object.fromEntries(sheet.columns.map((column) => [column, cellValue(inputRow[column])]));",
" const row = worksheet.addRow(values);",
" sheet.columns.forEach((column, columnIndex) => {",
" const value = inputRow[column];",
' if (value && typeof value === "object" && value.type === "formula" && value.numberFormat) {',
" row.getCell(columnIndex + 1).numFmt = numberFormats[value.numberFormat];",
" }",
" });",
" }",
" worksheet.getRow(1).font = { bold: true };",
" worksheet.views = [{ state: 'frozen', ySplit: 1 }];",
"}",
Expand Down
78 changes: 49 additions & 29 deletions packages/agent-core/src/tools/media/execute.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,14 @@ import type { ArtifactUploadResult, CodeRuntimeContext } from "@cheatcode/sandbo
import { GoogleGenAI, type Image, type Video, VideoGenerationReferenceType } from "@google/genai";
import { z } from "zod";
import {
type GenerateOrEditMediaInput,
GenerateOrEditMediaInputSchema,
type GenerateOrEditMediaOutput,
GenerateOrEditMediaOutputSchema,
type GenerateImageOutput,
GenerateImageOutputSchema,
type GenerateOrEditImageInput,
GenerateOrEditImageInputSchema,
type GenerateOrExtendVideoInput,
GenerateOrExtendVideoInputSchema,
type GenerateVideoOutput,
GenerateVideoOutputSchema,
} from "./schemas";

const IMAGE_MODEL = "gemini-3.1-flash-image";
Expand Down Expand Up @@ -43,37 +47,53 @@ const InteractionStepsSchema = z.array(
}),
);

export async function executeGenerateOrEditMedia(
input: GenerateOrEditMediaInput,
export async function executeGenerateOrEditImage(
input: GenerateOrEditImageInput,
runtimeContext: CodeRuntimeContext,
googleApiKey: string,
): Promise<GenerateOrEditMediaOutput> {
const parsed = GenerateOrEditMediaInputSchema.parse(input);
): Promise<GenerateImageOutput> {
const parsed = GenerateOrEditImageInputSchema.parse(input);
const apiKey = requiredApiKey(googleApiKey);
const client = new GoogleGenAI({ apiKey });
const media =
parsed.type === "image"
? await generateImage(client, parsed, runtimeContext)
: await generateVideo(client, apiKey, parsed, runtimeContext);
const filename = generatedFilename(parsed.prompt, media.extension);
const sandboxPath = await writeMediaToWorkspace(
runtimeContext,
parsed.type,
filename,
media.bytes,
);
const artifact = await storeArtifact(runtimeContext, parsed.type, filename, media);
return GenerateOrEditMediaOutputSchema.parse({
artifact,
model: media.model,
sandboxPath,
type: parsed.type,
const media = await generateImage(client, parsed, runtimeContext);
const result = await persistGeneratedMedia(runtimeContext, "image", parsed.prompt, media);
return GenerateImageOutputSchema.parse({
...result,
type: "image",
});
}

export async function executeGenerateOrExtendVideo(
input: GenerateOrExtendVideoInput,
runtimeContext: CodeRuntimeContext,
googleApiKey: string,
): Promise<GenerateVideoOutput> {
const parsed = GenerateOrExtendVideoInputSchema.parse(input);
const apiKey = requiredApiKey(googleApiKey);
const client = new GoogleGenAI({ apiKey });
const media = await generateVideo(client, apiKey, parsed, runtimeContext);
const result = await persistGeneratedMedia(runtimeContext, "video", parsed.prompt, media);
return GenerateVideoOutputSchema.parse({
...result,
type: "video",
});
}

async function persistGeneratedMedia(
runtime: CodeRuntimeContext,
type: "image" | "video",
prompt: string,
media: GeneratedMedia,
) {
const filename = generatedFilename(prompt, media.extension);
const sandboxPath = await writeMediaToWorkspace(runtime, type, filename, media.bytes);
const artifact = await storeArtifact(runtime, type, filename, media);
return { artifact, model: media.model, sandboxPath };
}

async function generateImage(
client: GoogleGenAI,
input: GenerateOrEditMediaInput,
input: GenerateOrEditImageInput,
runtime: CodeRuntimeContext,
): Promise<GeneratedMedia> {
const references = await loadReferences(input.reference_images ?? [], runtime);
Expand Down Expand Up @@ -110,7 +130,7 @@ function imageInteractionInput(prompt: string, references: MediaReference[]) {
];
}

function imagePrompt(input: GenerateOrEditMediaInput): string {
function imagePrompt(input: GenerateOrEditImageInput): string {
if (!input.reference_images?.length) {
return input.prompt;
}
Expand Down Expand Up @@ -155,7 +175,7 @@ function extractInteractionImage(steps: unknown): MediaReference | null {
async function generateVideo(
client: GoogleGenAI,
apiKey: string,
input: GenerateOrEditMediaInput,
input: GenerateOrExtendVideoInput,
runtime: CodeRuntimeContext,
): Promise<GeneratedMedia> {
const references = await loadReferences(input.reference_images ?? [], runtime);
Expand Down Expand Up @@ -186,7 +206,7 @@ async function generateVideo(
return generatedMedia(bytesToBase64(bytes), generated.mimeType ?? "video/mp4", VIDEO_MODEL);
}

function videoConfig(input: GenerateOrEditMediaInput, references: MediaReference[]) {
function videoConfig(input: GenerateOrExtendVideoInput, references: MediaReference[]) {
return {
aspectRatio: input.aspect_ratio ?? "16:9",
durationSeconds: input.duration ?? 8,
Expand Down
Loading