Skip to content

feat: list recent agent activity in the app's menu - #98

Merged
iamnbutler merged 1 commit into
mainfrom
feat/activity-menu
Oct 17, 2026
Merged

iamnbutler merged 1 commit into
mainfrom
feat/activity-menu

Conversation

@iamnbutler

Copy link
Copy Markdown
Contributor

From #27 (its menu was to show the last action and its caller), which also answers #24's visibility concern. The menu-bar app now shows Recent Activity.

Menu

  • The last 5 operations are listed, and All Activity (n) opens a submenu with the last 20.
  • Each line reads Operation ×count · App — Window title, with an outcome icon (completed ✓, unknown ?, refused ✗).
  • Each line's subtitle reads agent (pid) · time · outcome.
  • The agent is DESKTOP_TOOLS_AGENT from the client's environment. Without it, the agent is the name of the client's parent process (for example node for the MCP server), and the pid is that parent's pid, which lasts for the agent's session.
  • Identical operations in a row fold into one line with a count, so a burst of keys or captures takes one line.

What's recorded

  • Every desktop change is recorded, plus captures (desktopObservation) and clipboard reads, because those reveal screen or clipboard contents.
  • Listings, AX reads (including desktop_wait polls) and snapshot bookkeeping are left out.
  • Operations have readable names, such as Click, Key, Type, Insert text, Drag, Capture, Read clipboard, Menu command and Dialog button. A future mutating op without a name falls back to its raw name, so it is never hidden.
  • Only the operation and target are kept. There is no typed text, clipboard content or field value; the record type carries no payload. Everything stays in memory (20 entries), and nothing goes to disk.

How

  • New patch, peekaboo-operation-record.patch (last in the stack):
    • An operationObserver on the embedded runtime configuration. route(plan:) reports a PeekabooBridgeOperationRecord after each authorized request finishes.
    • The target window is read before dispatch, reusing agent-window's lookup, now factored out as requestTargetWindow.
    • Outcome mapping:
      • confirmed or non-mutating success → completed
      • refused mutation → refused
      • other mutation states → unknown
      • errors → unknown when operationMayHaveCompleted, else refused
    • It adds no Bridge op, so the wire count and permission partition are unchanged.
  • Runtime: activity.swift holds the ring buffer and resolves names (agent, app via NSRunningApplication, window title via CGWindowList) while the client still waits for its reply. The new C ABI desktop_tools_activity() returns JSON, and it's documented in docs/embed.md.
  • App: app.swift renders the section on every menu open.
  • Not built: the optional "Pause all input" toggle, pending your OK.

Tests

  • PeekabooBridgeOperationRecordTests has 3 tests, covering:
    • reports with target and outcome
    • no report without an authorized peer
    • the state mapping
  • PeekabooBridgeTargetWindowObserverTests still passes after the refactor.
  • These ran via grouptest with a narrow filter, and no helper was left running.
  • npm test passes 52 of 52, npm run check passes, and the native build and signed dev app build both pass.

Live, with the screen locked: the dev app served desktop_apps, desktop_windows, and inspect (refused while locked). It also handled a stale-snapshot click (refused), a clipboard read and a media key (refused while locked), and the host stayed up with the observer running.

Pending (screen locked): open the menu and check the entries, the folding and the All Activity submenu. Added to #42.

@iamnbutler
iamnbutler merged commit ab64f99 into main Oct 17, 2026
2 checks passed
@iamnbutler
iamnbutler deleted the feat/activity-menu branch October 17, 2026 08:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant