AI-Native Development

IWSDK is built from the ground up for AI-assisted immersive web development. AI agents can see, interact with, and debug your WebXR experience through 32 MCP (Model Context Protocol) tools — screenshots, controller input, scene inspection, ECS (Entity-Component-System) debugging, and more.

How It Works

When you enable AI in your Vite config, the plugin sets up three things automatically:

Playwright Browser — A managed Chromium instance that loads your app and provides screenshots and console capture for the AI agent.
MCP Server — A local server that exposes 32 tools for controlling the emulated XR runtime, inspecting the scene, and debugging ECS state.
MCP Config Files — Auto-generated config files (e.g., .mcp.json for Claude) so your AI tool discovers the server on startup.

text

┌──────────────────────┐
│  AI Tool             │
│  (Claude, Cursor...) │
└──────────┬───────────┘
           │ MCP protocol (stdio)
┌──────────▼───────────┐
│  MCP Server          │◄── screenshots, console logs
└──────────┬───────────┘
           │ WebSocket
┌──────────▼───────────┐     ┌──────────────────────┐
│  Vite Dev Server     │────►│  Normal Browser      │
│                      │     │  (developer)         │
└──────────┬───────────┘     └──────────────────────┘
           │
┌──────────▼───────────┐
│  Playwright Browser  │
│  (managed)           │
└──────────────────────┘

The AI agent communicates with the MCP server over stdio. The server relays commands to the Playwright browser via WebSocket, where the IWER runtime processes them (move controllers, trigger inputs, query state). Screenshots and console logs are captured server-side through Playwright's CDP integration — no browser round-trip needed.

Your normal browser runs independently with its own XR session, so you can develop and test manually while the agent works in the background.

Additional MCP Servers

The plugin can auto-register additional MCP servers alongside iwsdk-dev-mcp:

iwsdk-rag-local — If @felixtz/iwsdk-rag-mcp is installed, a local RAG server is registered that provides semantic code search and IWSDK API knowledge. The embedding model is downloaded and cached on first startup.
hzdb — If @meta-quest/hzdb is installed, the hzdb MCP server is registered. This provides Meta Quest device management, 3D asset search from Meta's asset library, and IWSDK documentation lookup.

These appear automatically in the generated MCP config files when the corresponding packages are present in node_modules.

Three Modes

IWSDK supports three usage modes, each optimized for a different workflow:

Mode	Description	Playwright	DevUI	Browser
Agent	AI works autonomously in the background	Headless, fixed viewport	Off	Normal browser opens
Oversight	You watch the AI work in real time	Visible, resizable	Off	Playwright is the browser
Collaborate	You and the AI share the same session	Visible, resizable	On	Playwright is the browser

Agent mode is the default — the AI operates in a headless browser optimized for screenshots while you develop in your normal browser. Switch to oversight or collaborate when you need visibility or hands-on interaction with the agent's session.

See Modes for the full deep dive.

What Can the Agent Do?

The MCP server exposes tools across several categories:

Session — Accept, monitor, and end XR sessions
Transforms — Position and orient the headset, controllers, and hands
Input — Trigger selects, manipulate gamepad buttons and axes, switch input modes
Browser — Take screenshots, read console logs, reload the page
Scene — Inspect the Three.js scene hierarchy and object transforms
ECS — Pause/step the simulation, query entities, diff state snapshots

See MCP Tools Reference for the complete list.

Next Steps

Getting Started — Set up AI in 5 minutes
Modes — Understand agent, oversight, and collaborate
MCP Tools Reference — All 32 tools documented
Workflows — Practical agent workflow patterns

AI-Native Development ​

How It Works ​

Additional MCP Servers ​

Three Modes ​

What Can the Agent Do? ​

Next Steps ​