Repository navigation
spike(native2): the single-codebase pure Swift host (#143) - #161
Merged
Merged
Conversation
…-for-field with Peekaboo (#143)
…th Peekaboo (#143) Ports the detached Accessibility walk, exact-window ScreenCaptureKit capture, JPEG sizing and in-memory single-use snapshots that hold live AXUIElement references for later actions. OCR, scope, region, save_path and pixels mode aren't ported yet. Why: native2/Sources/DesktopHost/Inspect.swift | ui_elements are ordered by Peekaboo's type groups with menus first, and bounded keeping scope roles, so element IDs and truncation match what agents saw from the Node server
A single left click is an Accessibility action, as in Peekaboo: focus a text input and read focus back, else AXPress, else the actionable element at the point. Double, triple, middle and unhandled right clicks are mouse events posted to the window's process with the window under the pointer named. Modifiers and expect aren't ported yet. Why: native2/Sources/DesktopMac/Pointer.swift | routed events name the window in fields 51, 91 and 92 and set a window-local point with SkyLight's CGEventSetWindowLocation; Chromium, Electron and Catalyst apps only take them through SLEventPostToPid
Ported from src/wait.ts for app, window and element conditions; dialog and text conditions wait on desktop_dialog and desktop_read. Polls read in the host, so a wait has no per-poll process or socket round trip. Why: native2/Sources/DesktopHost/Wait.swift | app conditions read the whole inventory, never a size-bounded desktop_apps result: a cut list drops apps, so the Node wait's app_running by pid times out while the app runs
…#143) Keys are posted to the window's process and never make the window key: the app's own focused window and element must already be the exact window, checked before every key-down. Native fields take desktop_type as an AXValue write read back; web fields take Unicode key events. Actions now share one runner for snapshot checks, refusal shapes and the follow-up observation. Why: native2/Sources/DesktopMac/Keyboard.swift | AppKit ignores a background Command-A key event, so select-all in a native field goes through AXSelectedTextRange as Peekaboo does Why: native2/Sources/DesktopMac/Keyboard.swift | key codes for characters come from UCKeyTranslate on the current layout, read on the main thread; a character that needs Shift or Option is refused so the model sends the modifier
…e's argument refusals (#143) Scroll sets a scroll bar's value, then page actions, then scroll bar steps, with a wheel only for WebKit views. Drag posts window-routed events along a sampled polyline. Hover moves the real pointer once the target window is uncovered at the point. A bad argument to an action tool now comes back as a refused action, as it does from Node, and expect is checked first. Why: native2/Sources/DesktopMac/Gestures.swift#exposes | Desktop Tools draws a highlight window above the target, so the cover check must skip the Desktop Tools app's windows as well as its own Why: native2/Sources/DesktopHost/Tools.swift | Node checks expect before an action's own arguments, and reports a bad action argument as a refused action rather than a plain error
…oard refusals and wait timing (#143)
…ity (#143) Ports desktop_inspect's remaining options and Node's full target_availability on failure, and refuses capture while the session is locked as Peekaboo does. Why: native2/Sources/DesktopHost/Save.swift#save | Node's inspect region crop reads coordinate_context.logicalBounds, which is never set, so it always refuses; Swift crops correctly, an intended difference Why: native2/Sources/DesktopProtocol/JSON.swift | a ternary's nil next to a JSON value becomes JSON.null through ExpressibleByNilLiteral, not an omitted field; write JSON?.none
…ions (#143) Why: native2/Sources/DesktopHost/Wait.swift#findElements | Peekaboo's element read reports a window Accessibility can't reach (as when the session is locked) as an empty read truncated by ax_incomplete_read, not an error
#143) Why: native2/Sources/DesktopHost/Dialogs.swift | Node reports a bridge listing refusal as internalError and the client's own mismatch as DESKTOP_ERROR; keep the two prefixes apart
The host checks each connection's audit-token-bound signature against the required client, and refuses unapproved callers, input that could reach Desktop Tools' own UI, and input while paused, all before dispatch. Why: native2/Sources/DesktopMac/Peer.swift | Trust only the code-directory hash the kernel reports for the audit token's process generation, before and after reading the on-disk signature; a pid lookup can be redirected by pid reuse. Why: native2/Sources/DesktopHost/Admission.swift | Caller refusals come back as refused actions before dispatch; Node reports a handshake refusal after onDispatch, so it says unknown there.
… with its partial steps (#143)
… and desktop_notification_action (#143) Why: native2/Sources/DesktopHost/Action.swift | the native client refuses an unknown well-formed ps1_ snapshot with DESKTOP_ACTION_FAILED "no longer belongs to the running desktop" and only a malformed one with invalidRequest; desktop_notification_action's gone message depends on it
…ervations without timings (#143)
desktop_launch, desktop_open, desktop_activate, desktop_hide, desktop_unhide, desktop_quit, desktop_close, desktop_focus, desktop_minimize, desktop_restore, desktop_maximize, desktop_fullscreen, desktop_move and desktop_resize, with the Node server's checks and follow-up observation, the native client's result shapes and Peekaboo's exact-target services. parity.mjs gains app and window target placeholders and --ids for steps that launch apps or reopen windows; the manage fixture builds two app bundles under /tmp/dt2-manage. Why: native2/Sources/DesktopHost/Manage.swift | a close failure carries the window's target_receipt only once the pinned-target check passed, as the Bridge attributes it; Node's explainRefusal adds target_availability only when that receipt is there Why: native2/Sources/DesktopHost/Manage.swift | the session-lock check comes before the app-generation and pinned-window checks, so a locked Mac refuses every management call with the lock message, as the native client does
…top_dock in pure Swift (#143) Why: native2/Sources/DesktopMac/MenuBar.swift | Only the AXExtrasMenuBar inventory is ported; where MenuBarAgent doesn't draw the bar the listing errors instead of falling back to WindowServer Why: native2/Sources/DesktopHost/MenuBar.swift | Node's tool layer drops arguments outside the schema, so extra top-level keys are ignored, not refused
…with activity, outline, cursor and fill hooks (#143) Why: native2/Sources/DesktopHost/Server.swift | The accept loop polls so stop() can end it: a blocked accept() doesn't wake when another thread closes the listener, and a republish after the socket was removed can't wake it by connecting either. Why: native2/Sources/DesktopHost/Hooks.swift | The activity outcome reads native_outcome.state, so input sent but not read back is unknown in the menu, as the Bridge recorded it, even when the tool result says completed. Why: native2/Sources/DesktopTools/Activity.swift | Tool names map to the display strings of the Peekaboo operations they replaced; desktop_dock, dialog_answer, system_key and insert pick theirs from their arguments, and the app's menu and log readers depend on these exact strings.
…a resource bundle (#143) Why: scripts/contract.mjs | The embedded dylib and the bundled MCP binary run from app and ace bundles where SwiftPM's Bundle.module lookup finds no resource bundle and crashes on first use.
…, select, insert, fill_password, clipboard (#143) Why: native2/Sources/DesktopHost/TextShim.swift | lane-only copies of swift-spike's ScopeRequest, scopeArgument and the Hooks subset fill uses, with a Keychain stand-in for the runtime's PasswordFill; delete the file when merging into swift-spike Why: native2/Sources/DesktopHost/Action.swift | the native client asks for a fresh observation after its own pre-dispatch checks (unknown or disabled element, precheck), and after a host refusal only when its reason is target_unavailable Why: native2/Sources/DesktopHost/TextRead.swift | Node's table rows omit element (and its missing-ID note never fires) because the native client encodes a nil id by leaving it out; Swift reports element null and the note
Why: native2/Sources/DesktopHost/Wait.swift | dialog waits by pid alone read the whole app inventory; Node's read a size-bounded list and so report a running app as app_not_listed, so parity covers the window form only.
Foreground press, modifier clicks and drags now show the runtime's pointer feedback, after their window check.
Every tool in the contract now runs in the Swift host. The switch itself is checked only up to its refusals: this Mac has one Space, and the calls file leaves out an already-current space_id, since on a Mac with more Spaces it would switch.
Final results take src/compact.ts's shape, checked byte for byte against Node on its recorded fixtures; internal reads keep the whole result. Bounding measures the compact text, desktop_windows takes all, and a window action returns its refreshed row instead of an inspection. Why: native2/Sources/DesktopHost/Notifications.swift | internal reads go through Tools.whole: the compact shape drops is_on_screen, which shown windows filter on
…143) Dialog discovery drops every dialog that encloses another candidate, so an Open panel opened from a sheet resolves to the panel. The runtime's outline is cut away under the target window's own sheets and panels. main's #147 needs no port: the Swift point-click walk already reads children only of elements that hold the point. Why: native2/Sources/DesktopTools/Window.swift | The window server moves a window's sheets and child windows with it as one group; another process's panel can't be ordered between them, so cover them with a mask, not ordering (#149) Why: native2/Sources/DesktopMac/Pointer.swift | The point-click descendant walk must not read children of elements whose frame excludes the point: SwiftUI lists materialize rows when read and grow, moving the target off the point (#147)
…ct rows (#143) JSON no longer takes the nil literal: in a ternary such as ocr ? recognize() : nil, nil became JSON.null, so absent fields went out as null (recognized_text without OCR). The calls file drops the junk-scheme open, which on an unlocked session leaves a CoreServicesUIAgent alert on screen. Why: native2/Sources/DesktopProtocol/JSON.swift | JSON must not be ExpressibleByNilLiteral: c ? JSON(x) : nil then infers JSON.null instead of an absent optional, and fields that Node omits go out as null Why: native2/scripts/calls-manage.json | no entry may open an unregistered URL scheme: unlocked, macOS shows a no-application alert for each call
…o does A focused table marks its cells focused too, so Snapshot.focused found no single receiver: paste refused and type fell back to window focus where the Node server delivered to the observed control.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Spike for #143: all 48 contract tools in one SwiftPM package (
native2/), with no Node server, no Peekaboo bridge and no XPC. It's additive. Nothing shipped changes, and the app, Peekaboo, the C ABI,createDesktop,/protocol,/toolsanddesktop-tools-clientstay, since ace embeds them.Parity with the Node server (0.16.1)
src/compact.tsthat CoreTests check byte for byte againstdist/compact.js.--server.Speed (p50 ms, Node → Swift)
apps 263 → 31 · windows 198 → 20 · inspect 464 → 47 · click 784 → 148 · type 693 → 60. Refusals take about 1 ms instead of about 100 ms.
Not yet as good as the original
See
native2/README.md, andnative2/CUTOVER.mdfor every remaining issue and the steps to swap over.Also
scripts/live-check.mjsgains--server. Check feat: report focus, appearance, session and input state with the app inventory #83 now reads compacted session state; it failed against Node too, since Tool results spend too many tokens: agents using desktop tools compact far more often #144 dropslocked: false.scripts/bench.mjsandscripts/contract.mjsare new.