feat(mcp): Add MCP servers for Babylon.js graph editors#18344
Conversation
Add Model Context Protocol (MCP) servers that expose Babylon.js authoring workflows to MCP-compatible clients (VS Code Copilot, Claude, etc.). Servers included: - mcp-server-core: shared helpers for validation, response shaping, schemas - nme-mcp-server: Node Material Editor graph authoring with live session bridge - nge-mcp-server: Node Geometry Editor graph authoring - npe-mcp-server: Node Particle Editor graph authoring - nrge-mcp-server: Node Render Graph Editor graph authoring - flow-graph-mcp-server: Flow Graph visual scripting - gui-mcp-server: Babylon.js GUI layout authoring - gltf-mcp-server: glTF/glb asset authoring and validation - smart-filters-mcp-server: Smart Filters post-processing chains Each server includes README, unit tests, and example outputs. Also adds MCP session UI components to the Node Editor and Playground.
|
Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s). |
|
Reviewer - this PR has made changes to one or more package.json files. |
|
Building or testing the playground has failed. If the tests failed, results can be found here: |
|
Snapshot stored with reference name: Test environment: To test a playground add it to the URL, for example: https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/18344/merge/index.html#WGZLGJ#4600 Links to test your changes to core in the published versions of the Babylon tools (does not contain changes you made to the tools themselves): https://playground.babylonjs.com/?snapshot=refs/pull/18344/merge To test the snapshot in the playground with a playground ID add it after the snapshot query string: https://playground.babylonjs.com/?snapshot=refs/pull/18344/merge#BCU1XR#0 If you made changes to the sandbox or playground in this PR, additional comments will be generated soon containing links to the dev versions of those tools. |
|
Smart Filters Editor is available to test at: |
|
You have changed file(s) that made possible changes to the sandbox. https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/SANDBOX/refs/pull/18344/merge/ |
- Change mcp-server-core imports from dist/index.js to src/index.js so vitest can resolve TypeScript source directly without a build step - Add babylonjs-gltf2interface stub alias to vitest.config.mts (matching master) so glTF loader tests can resolve the types-only package
- Replace 20 barrel (directory) imports with direct file imports - Rename type params to StrictPascalCase (TSchema→Schema, etc.) - Rename IJsonObject→JsonObject, _parseJsonObject→ParseJsonObject - Add Async suffix: resolveExternalBuffers→resolveExternalBuffersAsync - Rename _getNumComponents→_GetNumComponents (static private) - Add missing JSDoc @param/@returns annotations - Replace Promise.then with async IIFE in signal handlers - Merge duplicate core/Misc/gradients imports
|
Reviewer - this PR has made changes to one or more package.json files. |
|
Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s). |
|
Reviewer - this PR has made changes to one or more package.json files. |
- Fix className casing to match engine RegisterClass names: Csm→Cascaded, Fxaa→FXAA, Ssao2→SSAO2, Ssr→SSR, Taa→TAA - Add missing blocks: ComputeShaderBlock, IblShadowsRendererBlock, PassCubePostProcessBlock - Update documentation strings with corrected block names
|
Building or testing the playground has failed. If the tests failed, results can be found here: |
|
Snapshot stored with reference name: Test environment: To test a playground add it to the URL, for example: https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/18344/merge/index.html#WGZLGJ#4600 Links to test your changes to core in the published versions of the Babylon tools (does not contain changes you made to the tools themselves): https://playground.babylonjs.com/?snapshot=refs/pull/18344/merge To test the snapshot in the playground with a playground ID add it after the snapshot query string: https://playground.babylonjs.com/?snapshot=refs/pull/18344/merge#BCU1XR#0 If you made changes to the sandbox or playground in this PR, additional comments will be generated soon containing links to the dev versions of those tools. |
|
Smart Filters Editor is available to test at: |
|
You have changed file(s) that made possible changes to the sandbox. https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/SANDBOX/refs/pull/18344/merge/ |
Flow Graph: add pointer events (Down/Up/Move), physics collision event, audio sound ended event, physics execution blocks (ApplyForce, ApplyImpulse, SetLinearVelocity, SetAngularVelocity, SetMotionType), physics data blocks (GetLinearVelocity, GetAngularVelocity, GetMassProperties), audio execution blocks (PlaySound, StopSound, PauseSound, SetVolume), audio data blocks (GetVolume, IsSoundPlaying), matrix combine/extract blocks (Matrix2D, Matrix3D), and DebugBlock. Remove stale 'NOT yet implemented' comments for blocks now in core. NGE: add ExtrudeGeometryBlock to registry.
|
Reviewer - this PR has made changes to one or more package.json files. |
|
Building or testing the playground has failed. If the tests failed, results can be found here: |
|
Smart Filters Editor is available to test at: |
|
You have changed file(s) that made possible changes to the sandbox. https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/SANDBOX/refs/pull/18344/merge/ |
⚡ Performance Test Results🟢 All performance tests passed — no regressions detected. |
|
WebGL2 visualization test reporter: |
|
Visualization tests for WebGPU |
🟢 Memory Leak Test Results13 passed, 0 leaked out of 13 scenarios 🟢 All memory leak tests passed — no leaks detected. Passed Scenarios (13)
|
|
Reviewer - this PR has made changes to one or more package.json files. |
|
Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s). |
|
Reviewer - this PR has made changes to one or more package.json files. |
|
Snapshot stored with reference name: Test environment: To test a playground add it to the URL, for example: https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/18344/merge/index.html#WGZLGJ#4600 Links to test your changes to core in the published versions of the Babylon tools (does not contain changes you made to the tools themselves): https://playground.babylonjs.com/?snapshot=refs/pull/18344/merge To test the snapshot in the playground with a playground ID add it after the snapshot query string: https://playground.babylonjs.com/?snapshot=refs/pull/18344/merge#BCU1XR#0 If you made changes to the sandbox or playground in this PR, additional comments will be generated soon containing links to the dev versions of those tools. |
|
Smart Filters Editor is available to test at: |
|
You have changed file(s) that made possible changes to the sandbox. https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/SANDBOX/refs/pull/18344/merge/ |
|
Visualization tests for WebGPU |
|
WebGL2 visualization test reporter: |
⚡ Performance Test Results🟢 All performance tests passed — no regressions detected. |
|
You have made possible changes to the playground. https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/PLAYGROUND/refs/pull/18344/merge/ The snapshot playground with the CDN snapshot (only when available): Note that neither Babylon scenes nor textures are uploaded to the snapshot directory, so some playgrounds won't work correctly. |
🟢 Memory Leak Test Results13 passed, 0 leaked out of 13 scenarios 🟢 All memory leak tests passed — no leaks detected. Passed Scenarios (13)
|
Summary
Adds a full Model Context Protocol (MCP) authoring layer for Babylon.js graph-based tools, plus a shared live-session architecture that lets MCP clients create or modify graphs and hand them directly to the matching web editor.
The branch now covers the MCP servers themselves, shared server utilities, live editor session infrastructure, editor-side session panels/connectors, packaging for public consumption, coverage/registry maintenance, and focused validation.
MCP servers
This PR adds private tool workspaces for the Babylon.js authoring surfaces that can be driven through MCP-compatible clients such as VS Code Copilot, Claude Desktop, or other MCP hosts.
@tools/mcp-server-core@tools/nme-mcp-server@tools/nge-mcp-server@tools/npe-mcp-server@tools/nrge-mcp-server@tools/flow-graph-mcp-server@tools/gui-mcp-server@tools/smart-filters-mcp-serverEach server includes a block/control registry or catalog, graph manager, MCP tools, README coverage, examples, and unit tests. The servers build through a shared Rollup MCP config and emit executable ESM bundles with shebangs.
Shared live editor sessions
Adds a reusable live-session server in
@tools/mcp-server-coreso every graph MCP server uses the same MCP-to-editor workflow instead of maintaining per-server session code.The shared session architecture provides:
McpEditorSessionControllerused by all graph/editor MCP servers.Editor integrations
Adds MCP session UI integration to the graph editors so a user can connect an editor to a session started by an MCP server and load the agent-generated graph.
Integrated editor surfaces:
The editor side shares a browser connector in
packages/dev/sharedUiComponents/src/mcp/mcpEditorSessionConnection.tsand each tool adds an MCP session component wired into its property tab/global state flow.The earlier Playground MCP session UI and local
.vscode/mcp.jsontest config were removed from the PR so the final branch stays focused on the graph tools and MCP packages.Public package
Adds a public package under
packages/public/@babylonjs/mcp-serversthat bundles all supported MCP servers into one installable npm package.The package provides:
babylonjs-nme-mcp-server,babylonjs-nge-mcp-server,babylonjs-nrge-mcp-server,babylonjs-npe-mcp-server,babylonjs-flow-graph-mcp-server,babylonjs-gui-mcp-server, andbabylonjs-smart-filters-mcp-server.babylonjs-mcp-servers, with aliases such asnme,nge,npe,nrge,flow-graph,gui, andsmart-filters.npxusage and MCP client configuration.build:es6:toolsnow includes@babylonjs/mcp-serversso the public package participates in the ES6/public-package build flow.Registry/catalog coverage
The branch updates MCP registries and catalogs so the servers expose the current set of relevant authoring blocks and controls.
Recent coverage work includes:
FlowGraphKeyDownEventBlock,FlowGraphKeyUpEventBlock, andFlowGraphIsKeyPressedBlock.BevelBlock.ScrollbarandMultiLine.Cleanup and maintenance
Also included:
Validation
Focused validation performed on the latest branch state:
249 passedacross the graph/server test files.npm run build -w @babylonjs/mcp-serverspasses and rebuilds/copies all bundled MCP servers.git diff --checkpasses.