Skip to content

feat: outline and label the window the agent is using - #43

Merged
iamnbutler merged 2 commits into
mainfrom
feat/agent-window
Oct 17, 2026
Merged

iamnbutler merged 2 commits into
mainfrom
feat/agent-window

Conversation

@iamnbutler

Copy link
Copy Markdown
Contributor

Follows #40 (agent cursor), using the same click-through overlay approach and the same kind of switch.

What

When the host turns it on, the window an inspection or action targets gets an orange outline and a label such as "Claude is using TextEdit".

  • When it shows: from the moment an authorized request for one exact window reaches the host. That covers a desktop_inspect of a window, every snapshot action (through its mutation lease or literal insert), and window-scoped mutations such as move, resize and drag. It moves to the new window when the target changes and fades about 4 s after the last request. Requests without one exact window (desktop_screen, apps, menus, clipboard) don't show it.
  • Tracking: CGWindowList polling at 60 Hz while the outline is up, then none. That's about 2% CPU in the app while visible and 0 after the fade. The outline follows moves and resizes, hides while the window is minimized, off screen or on another Space, and returns on restore.
  • Z-order: the panel takes the target's window layer and is ordered .above relative to the target's window number. Whenever something comes between them, it's reordered on the next tick. Windows in front of the target therefore cover the outline too.
  • Never interactive, never captured: the panel is borderless, .nonactivatingPanel, ignoresMouseEvents, can't become key or main, and has sharingType = .none.
  • Switch: desktop_tools_set_highlight(bool), with highlight added to the status JSON. It's off for embedders. The app turns it on, with its own menu item, Highlight agent's window (agentWindow default).
  • Agent name: the MCP server maps clientInfo to Claude, Codex, Cursor, Gemini or Copilot. Otherwise it uses the client's title, and failing that "Agent".
    • The name travels as DESKTOP_TOOLS_AGENT in the native client's environment. createDesktop takes a new optional agent, read at each call.
    • The runtime reads the variable from the requesting peer's process arguments (KERN_PROCARGS2) while its request is pending. The peer is our own signed client.

Peekaboo patch

peekaboo-agent-window.patch goes at the end of the stack. It adds PeekabooEmbeddedBridgeRuntime.Configuration.targetWindowObserver, threaded through the bootstrap to the Bridge server.

  • In route, after access validation and before dispatch, the server reports the peer PID, owner PID and window ID. The window comes from one of:
    • a desktop observation pinned to a window ID
    • the window context of the snapshot named by beginSnapshotMutation or literalInsert
    • the plan's .window operation scope
  • The report is advisory: it can't fail the request and changes no target check, route or outcome. Without an observer, nothing extra runs.
  • Two new tests in PeekabooBridgeTargetWindowObserverTests cover reporting, and the cases that must not report: screen, PID without a window, missing snapshot, a non-window op, no peer.

Verified live (before the screen locked, on the pre-rebase build)

Tested with the dev app on DESKTOP_TOOLS_STATE=/tmp/cursor-state, against a TextEdit fixture window:

  • Inspect shows the outline. Its frame is the window ±16 pt, and it is first in optionOnScreenAboveWindow for the target.
  • Partially covering windows (System Settings, Finder) cover the outline as well. I checked this with a temporary .readOnly build, using screenshots.
  • desktop_move: the outline followed via polling, since the report precedes the move.
  • desktop_minimize hides the outline; desktop_restore brings it back directly above the window.
  • desktop_focus: the outline stays directly above after the window is raised.
  • Inspecting another window moves the outline there. The label shows "Codex is using TextEdit" with agent: "Codex", and "Agent is using TextEdit" with no agent.
  • A snapshot desktop_click shows the outline through the mutation-lease path.
  • With .none, the outline was on screen during desktop_inspect and desktop_screen, and neither image contains it.
  • It fades and orders out after the idle period.

Pending: screen locked

Run these with open -n --env DESKTOP_TOOLS_STATE=/tmp/<dir> "build/Desktop Tools.app", and with the MCP server or createDesktop pointed at the same state.

  1. Re-run the inspect, then click flow on the rebased build (now including feat: answer dialogs, force quit, hide, unhide and maximize #38), and confirm the outline and label appear.
  2. Space switch: inspect a window, then switch to another Space within 4 s. The outline should be gone there and back on return. Hiding relies on kCGWindowIsOnscreen.
  3. Full-screen target: inspect a window in full screen. The outline should sit at the screen edges, above the window.
  4. User drag: drag the target window by hand during the 4 s window. The outline should follow smoothly at 60 Hz.
  5. Dialog targets from feat: answer dialogs, force quit, hide, unhide and maximize #38 (desktop_dialog on a sheet or alert): check which window gets outlined, either the dialog window or nothing.
  6. Menu toggle: turn off Highlight agent's window. Inspect should then show nothing, and turning it back on should restore the behavior.
  7. Clicks pass through: click on the outline's margin and label area. The click should reach whatever is underneath.

Tests: the new observer tests, PeekabooEmbeddedBridgeRuntimeTests and PeekabooBridgeMenuWindowReceiptTests pass. npm run check is clean.

@iamnbutler
iamnbutler merged commit 957d15e into main Oct 17, 2026
2 checks passed
@iamnbutler
iamnbutler deleted the feat/agent-window branch October 17, 2026 02:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant