# Desktop & Browser Automation

Orb and its `sandboxed.sh` backend support headless GUI and browser automation inside workspaces so agents can launch Chromium, inspect accessibility trees, capture screenshots, and record video.

## Wayland-First Architecture (Default)

Desktop sessions are **Wayland-first**:
- Each desktop session runs a headless **Sway** compositor whose Wayland socket lives under the workspace at `.sandboxed-sh/wayland/<display>/` (`src/api/desktop.rs`).
- Companion tools include `grim` (screenshots), `wtype` / `wlrctl` (keyboard and pointer input), `wf-recorder` (screen recording), **Chromium**, **AT-SPI2** (accessibility tree), and **Tesseract OCR**.
- **Xvfb + i3** (`:99`+) is retained only as a legacy X11 fallback when Wayland is unavailable.
- Harnesses and MCPs do not receive a display by default unless the mission requests desktop tools on a workspace with an active desktop session.

## Docker vs Native Setup

- **Docker**: Pre-installed out of the box (Sway, `grim`, `wtype`, `wlrctl`, `wf-recorder`, Chromium, ImageMagick, Tesseract).
- **Native Ubuntu 24.04**: Install the Wayland/Sway stack (or Xvfb/i3 fallback) as documented in [`docs/install-native.md`](https://github.com/Th0rgal/orb/blob/master/docs/install-native.md):

```bash
apt update && apt install -y \
  sway grim wtype wlrctl wf-recorder \
  chromium chromium-sandbox \
  at-spi2-core libatspi2.0-0 python3-gi gir1.2-atspi-2.0 \
  tesseract-ocr imagemagick fonts-liberation fonts-dejavu-core
```

## Available Desktop MCP Tools

When the `desktop` MCP server is enabled for a workspace, the agent can call:

| Tool | Description |
| --- | --- |
| `desktop_start_session` | Start an isolated headless compositor session + optional Chromium instance |
| `desktop_stop_session` | Stop the desktop session and clean up child processes |
| `desktop_screenshot` | Capture a screenshot of the current display |
| `desktop_type` | Send keyboard text or key chords |
| `desktop_click` | Click at screen coordinates |
| `desktop_mouse_move` | Move the pointer |
| `desktop_scroll` | Scroll the mouse wheel |
| `desktop_get_text` | Extract visible text via AT-SPI2 accessibility tree or Tesseract OCR fallback |
| `desktop_i3_command` | Send window-manager layout commands (`split`, `focus`, `fullscreen`, `kill`) |