Live Canvas

Live Canvas is an agent-driven visual workspace that allows CoWork OS agents to create, display, and interact with dynamic HTML/CSS/JavaScript content in real-time.

Overview

Live Canvas enables agents to:

• Render interactive visualizations, dashboards, and forms
• Display data analysis results with charts and graphs
• Create prototypes and mockups
• Build interactive tools for user feedback
• Execute JavaScript in the canvas context and retrieve results

Each canvas session opens in a dedicated Electron BrowserWindow and is isolated per task.

In-App Preview

When an agent pushes content to a canvas, a live preview appears directly in the task view:

Preview Modes

The preview supports two modes, toggled with the I key or the pointer button:

Interactive Mode (Default)

• Full browser experience: Interact with canvas content directly in the preview
• Click buttons, fill forms, scroll: All standard browser interactions work
• No external window needed: Stay in the main app while using the canvas
• Snapshot paused: Auto-refresh is disabled to avoid interrupting your interaction

Snapshot Mode

• Static screenshot: Shows a captured image of the canvas content
• Auto-refresh: Automatically updates every 2 seconds while active
• Click to expand: Click the preview image to open the full canvas window
• Lower resource usage: Good for monitoring when not actively interacting

Features

• Resizable preview: Drag the bottom edge to make the preview taller or shorter
• Minimize/Expand: Toggle button to collapse or expand the preview without closing the session
• Status indicator: Shows current session status:
  • 🟢 Live - Session is active and auto-refreshing
  • 🟡 Paused - Session is paused
  • 🔴 Closed - Session has been closed
• Dimensions display: Shows the current canvas resolution (e.g., "1800 x 1336")

Preview Controls

Note: Keyboard shortcuts work when the preview is focused (click on it first).

Export Options

The export menu (E key or export button) provides three options:

• Export HTML: Download the canvas as a standalone HTML file
• Open in Browser: Open the canvas in your default system browser
• Show in Finder: Open the canvas session folder in Finder

Additional Features

• Snapshot history: Browse through previous snapshots with the history panel
• Console viewer: View console logs from the canvas (when available)
• Copy to clipboard: Quickly copy the current snapshot for pasting into documents or chat
• Save as PNG: Download the snapshot with an auto-generated filename
• Error details: When errors occur, see specific details and a "Try Again" button
• Memory efficient: Previous images are cleared before loading new ones to reduce memory usage
• Summary-mode friendliness: repeated screenshot updates from visual refinement loops are collapsed more compactly in task summaries so the feed stays readable while the canvas is iterating

The in-app preview allows you to interact with canvas content directly without switching windows. Use interactive mode (default) for full browser-like interaction, or switch to snapshot mode when you just want to monitor changes.

Relationship To Web Page Artifacts

Live Canvas is an active visual workspace. Web page artifacts are durable local output files.

Use Live Canvas when an agent is actively iterating on an HTML/CSS/JavaScript experience through canvas sessions, snapshots, console output, checkpoints, and canvas-specific export controls.

Use Web Page Artifacts when a task creates a local .html / .htm file or built React output such as dist/index.html, build/index.html, or out/index.html. Those files render as compact task-feed artifact cards and open in the shared artifact sidebar/fullscreen viewer with a sandboxed iframe, browser/folder/copy actions, and follow-up composer context. The artifact viewer does not auto-start React, Vite, or Next dev servers; source projects need built HTML output before they can preview in this path.

Build Mode

Build Mode is a dedicated "idea → working prototype" workflow layered on top of Live Canvas. It guides the agent through four structured phases, each producing a named checkpoint that you can revert to or diff against.

Phases

Using Build Mode

Build Mode is available as a built-in skill (build-mode). You can activate it by:

1. Selecting the Build Mode skill from the skill picker
2. Providing your idea and optionally a tech stack preference
3. The agent walks through each phase, creating canvas checkpoints at each stage

Phase Checkpoints

Each phase creates a named checkpoint (e.g., build:concept, build:plan, build:scaffold, build:iterate). You can:

• View timeline: See all phases with their checkpoint IDs and timestamps
• Revert to phase: Roll back to any previous phase's checkpoint
• Diff between phases: Compare file-level changes between any two phase checkpoints (added, removed, modified files)

Implementation Files

• src/electron/canvas/build-mode-orchestrator.ts — Phase management, checkpoint mapping, revert/diff logic
• resources/skills/build-mode.json — Skill definition with phase prompts and parameters

---

Named Checkpoints

Canvas sessions support named checkpoints for saving, restoring, and comparing canvas states.

Checkpoint Tools

Advanced Checkpoint Methods

---

Architecture

┌─────────────────────────────────────────────────┐
│              Agent (Task Executor)               │
│  Uses canvas_* tools to create visual content    │
└─────────────────────────────────────────────────┘
                      ↓
┌─────────────────────────────────────────────────┐
│              Canvas Manager                      │
│  - Session lifecycle management                  │
│  - BrowserWindow creation (hidden for snapshots) │
│  - File watching (chokidar)                      │
│  - Event broadcasting                            │
│  - Snapshot capture for in-app preview           │
└─────────────────────────────────────────────────┘
                      ↓
┌─────────────────────────────────────────────────┐
│           Canvas BrowserWindow                   │
│  - Loads content via canvas:// protocol          │
│  - Isolated session directory                    │
│  - A2UI bridge for user interactions             │
│  - Hidden by default, shown on demand            │
└─────────────────────────────────────────────────┘
          ↓                           ↓
┌─────────────────────────┐   ┌─────────────────────────┐
│     In-App Preview      │   │   Full Canvas Window    │
│  - Interactive mode     │   │   - Full interactivity  │
│    (embedded webview)   │   │   - Form submissions    │
│  - Snapshot mode        │   │   - A2UI communication  │
│  - Resizable            │   │   - Separate window     │
│  - Export options       │   │                         │
└─────────────────────────┘   └─────────────────────────┘
                      ↓
┌─────────────────────────────────────────────────┐
│           canvas:// Protocol Handler             │
│  - Secure file serving from session directory    │
│  - MIME type detection                           │
│  - Path traversal protection                     │
└─────────────────────────────────────────────────┘

Agent Tools

The following tools are available to agents for canvas operations:

canvas_create

Creates a new canvas session.

Input:

Output:

{
  "sessionId": "abc123-def456",
  "sessionDir": "/path/to/session/directory"
}

canvas_push

Pushes content to the canvas session.

Input:

Output:

{
  "success": true
}

canvas_show

Shows the canvas window and brings it to focus.

Input:

Output:

{
  "success": true
}

canvas_hide

Hides the canvas window without closing the session.

Input:

Output:

{
  "success": true
}

canvas_close

Closes the canvas session and its window.

Input:

Output:

{
  "success": true
}

canvas_eval

Executes JavaScript in the canvas context.

Input:

Output:

{
  "result": <any>
}

canvas_snapshot

Takes a screenshot of the canvas content.

Input:

Output:

{
  "imageBase64": "iVBORw0KGgo...",
  "width": 800,
  "height": 600
}

canvas_list

Lists all canvas sessions for the current task.

Input: None

Output:

{
  "sessions": [
    {
      "id": "abc123",
      "title": "My Canvas",
      "status": "active",
      "createdAt": 1706789012345
    }
  ]
}

Example Usage

Creating a Data Visualization

// Agent creates a canvas session
const { sessionId } = await canvas_create({ title: "Sales Dashboard" });

// Push HTML content with Chart.js
await canvas_push({
  session_id: sessionId,
  content: `
    <!DOCTYPE html>
    <html>
    <head>
      <script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
      <style>
        body { font-family: system-ui; padding: 20px; }
        canvas { max-width: 600px; }
      </style>
    </head>
    <body>
      <h1>Q4 Sales by Region</h1>
      <canvas id="chart"></canvas>
      <script>
        new Chart(document.getElementById('chart'), {
          type: 'bar',
          data: {
            labels: ['North', 'South', 'East', 'West'],
            datasets: [{
              label: 'Sales ($M)',
              data: [12, 19, 8, 15],
              backgroundColor: ['#4CAF50', '#2196F3', '#FFC107', '#9C27B0']
            }]
          }
        });
      </script>
    </body>
    </html>
  `
});

// The in-app preview automatically shows the content!
// No need to call canvas_show unless user needs full interactivity

// Optional: Take a snapshot programmatically
const { imageBase64 } = await canvas_snapshot({ session_id: sessionId });

Interactive Form

// Create canvas with a form for user input
const { sessionId } = await canvas_create({ title: "Configuration" });

await canvas_push({
  session_id: sessionId,
  content: `
    <!DOCTYPE html>
    <html>
    <head>
      <style>
        body { font-family: system-ui; padding: 20px; }
        input, select { margin: 10px 0; padding: 8px; width: 200px; }
        button { background: #007AFF; color: white; padding: 10px 20px; border: none; cursor: pointer; }
      </style>
    </head>
    <body>
      <h2>Project Settings</h2>
      <form id="config">
        <div>
          <label>Project Name:</label><br>
          <input type="text" id="name" value="my-project">
        </div>
        <div>
          <label>Framework:</label><br>
          <select id="framework">
            <option value="react">React</option>
            <option value="vue">Vue</option>
            <option value="svelte">Svelte</option>
          </select>
        </div>
        <button type="button" onclick="window.coworkCanvas.sendA2UIAction('submit', 'config', getFormData())">
          Apply Settings
        </button>
      </form>
      <script>
        function getFormData() {
          return {
            name: document.getElementById('name').value,
            framework: document.getElementById('framework').value
          };
        }
      </script>
    </body>
    </html>
  `
});

// For interactive forms, show the canvas window so user can interact
// The in-app preview shows a snapshot, but for clicking buttons/filling forms,
// the user needs the full window
await canvas_show({ session_id: sessionId });

A2UI (Agent-to-UI) Communication

Canvas windows can send actions back to the agent using the A2UI bridge. This enables interactive workflows where user actions in the canvas trigger agent responses.

Sending Actions from Canvas

The canvas preload script exposes window.coworkCanvas:

// Send an action to the agent
window.coworkCanvas.sendA2UIAction(
  'button_click',           // Action name
  'submit-button',          // Component ID
  { formData: {...} }       // Context object
);

Receiving Actions in Agent

When a user interacts with the canvas, the agent receives a formatted message:

[Canvas Interaction]
Action: button_click
Component: submit-button
Context: { "formData": { ... } }

The user interacted with the canvas. Please respond appropriately based on this action.

Security

Path Traversal Protection

The canvas:// protocol implements multiple layers of security:

1. Double dot check: Paths containing .. are rejected
2. Double slash check: Paths containing // are rejected
3. Path containment: Resolved paths must be within the session directory

Session Isolation

• Each canvas session has its own directory
• Sessions are scoped to specific tasks
• File operations are sandboxed to the session directory

URL Format

canvas://{sessionId}/{filename}

Example: canvas://abc123-def456/index.html

Configuration

Live Canvas uses the following directory for session storage:

~/Library/Application Support/cowork-os/canvas/{sessionId}/

Sessions are automatically cleaned up when:

• The session is explicitly closed via canvas_close
• The application exits
• The parent task is deleted

Supported Content

File Types

External Resources

Canvas content can load external resources via CDN:

• Chart.js, D3.js for visualizations
• Tailwind CSS, Bootstrap for styling
• Any other libraries via script/link tags

Events

The Canvas Manager emits events that can be observed in the main process:

Troubleshooting

In-App Preview Not Appearing

1. The preview only appears after content is pushed (not when session is created)
2. Wait for the initial load - the preview retries up to 3 times before showing an error
3. Ensure the canvas session is still active (not closed)

Preview Shows "Failed to capture canvas"

1. The hidden browser window may not have loaded the content yet
2. Try clicking the refresh button to force a new snapshot
3. Check that the HTML content is valid and doesn't have JavaScript errors

Preview Not Updating

1. Auto-refresh only works when the session status is "active"
2. Make sure the preview is not minimized (minimized previews don't auto-refresh)
3. The refresh interval is 2 seconds - be patient for updates

Canvas Window Not Showing

1. Ensure the session exists with canvas_list
2. Check that canvas_show was called after canvas_push
3. Verify the content has valid HTML structure

Content Not Updating

1. The canvas auto-reloads on file changes
2. Ensure you're pushing to the correct session ID
3. Check browser console for JavaScript errors

A2UI Actions Not Received

1. Verify window.coworkCanvas is available in the canvas context
2. Check that the action name is descriptive
3. Ensure the canvas window is still open

API Reference

See the following files for implementation details:

• src/electron/canvas/canvas-manager.ts - Session management
• src/electron/canvas/canvas-protocol.ts - URL protocol handler
• src/electron/agent/tools/canvas-tools.ts - Agent tool definitions
• src/electron/ipc/canvas-handlers.ts - IPC handlers
• src/electron/canvas/build-mode-orchestrator.ts - Build Mode phase management
• src/renderer/components/CanvasPreview.tsx - In-app preview component
• src/renderer/components/MainContent.tsx - Canvas session integration