Repository navigation
feat: outline and label the window the agent is using - #43
Merged
Merged
Conversation
iamnbutler
force-pushed
the
feat/agent-window
branch
from
October 17, 2026 02:05
f391691 to
80c6582
Compare
This was referenced Oct 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows #40 (agent cursor), using the same click-through overlay approach and the same kind of switch.
What
When the host turns it on, the window an inspection or action targets gets an orange outline and a label such as "Claude is using TextEdit".
desktop_inspectof a window, every snapshot action (through its mutation lease or literal insert), and window-scoped mutations such as move, resize and drag. It moves to the new window when the target changes and fades about 4 s after the last request. Requests without one exact window (desktop_screen, apps, menus, clipboard) don't show it..aboverelative to the target's window number. Whenever something comes between them, it's reordered on the next tick. Windows in front of the target therefore cover the outline too..nonactivatingPanel,ignoresMouseEvents, can't become key or main, and hassharingType = .none.desktop_tools_set_highlight(bool), withhighlightadded to the status JSON. It's off for embedders. The app turns it on, with its own menu item, Highlight agent's window (agentWindowdefault).clientInfoto Claude, Codex, Cursor, Gemini or Copilot. Otherwise it uses the client'stitle, and failing that "Agent".DESKTOP_TOOLS_AGENTin the native client's environment.createDesktoptakes a new optionalagent, read at each call.KERN_PROCARGS2) while its request is pending. The peer is our own signed client.Peekaboo patch
peekaboo-agent-window.patchgoes at the end of the stack. It addsPeekabooEmbeddedBridgeRuntime.Configuration.targetWindowObserver, threaded through the bootstrap to the Bridge server.route, after access validation and before dispatch, the server reports the peer PID, owner PID and window ID. The window comes from one of:beginSnapshotMutationorliteralInsert.windowoperation scopePeekabooBridgeTargetWindowObserverTestscover reporting, and the cases that must not report: screen, PID without a window, missing snapshot, a non-window op, no peer.Verified live (before the screen locked, on the pre-rebase build)
Tested with the dev app on
DESKTOP_TOOLS_STATE=/tmp/cursor-state, against a TextEdit fixture window:optionOnScreenAboveWindowfor the target..readOnlybuild, using screenshots.desktop_move: the outline followed via polling, since the report precedes the move.desktop_minimizehides the outline;desktop_restorebrings it back directly above the window.desktop_focus: the outline stays directly above after the window is raised.agent: "Codex", and "Agent is using TextEdit" with no agent.desktop_clickshows the outline through the mutation-lease path..none, the outline was on screen duringdesktop_inspectanddesktop_screen, and neither image contains it.Pending: screen locked
Run these with
open -n --env DESKTOP_TOOLS_STATE=/tmp/<dir> "build/Desktop Tools.app", and with the MCP server orcreateDesktoppointed at the same state.kCGWindowIsOnscreen.desktop_dialogon a sheet or alert): check which window gets outlined, either the dialog window or nothing.Tests: the new observer tests,
PeekabooEmbeddedBridgeRuntimeTestsandPeekabooBridgeMenuWindowReceiptTestspass.npm run checkis clean.