Skip to content

docs: pick a native select in Chrome with one desktop_key call - #139

Merged
iamnbutler merged 3 commits into
mainfrom
fix/chrome-select
Oct 19, 2026
Merged

iamnbutler merged 3 commits into
mainfrom
fix/chrome-select

Conversation

@iamnbutler

@iamnbutler iamnbutler commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Closes #138.

Summary

The real cause is that a Chromium <select> menu closes between tool calls. Clicking an item or using AXPick from a later snapshot therefore goes stale (SNAPSHOT_STALE). The fix is to pick with one desktop_key call while the select has keyboard focus: keys: [down (opens the menu), down × n, enter], where n is the target option's index minus the current option's index. This works in the background.

  • docs/recipes.md: new recipe "Pick from a native select in a browser". It covers focusing the select (Tab, or a click, which also opens the menu), counting n from the options an open menu lists, the single key call, and reading back the AXPopUpButton value. desktop_activate is mentioned only as a fallback when the value didn't change. Separators and disabled items are marked (unverified). The web form recipe step now points Chromium browsers to the new recipe.
  • docs/tools.md and the desktop_set_value / desktop_element_action descriptions in src/tools.ts keep click + AXPick for native pop-up buttons and point Chromium <select> to the single desktop_key call.
  • CHANGELOG [Unreleased] entry.

An earlier revision downgraded desktop_key to unknown for background Chromium apps. Live checks disproved that premise, so it was removed. There are no code or native changes now, only docs and tool descriptions.

Live evidence

Setup: isolated Chrome, a local page with an <input> and a <select>, Chrome in the background with Preview frontmost, installed host 0.14.1.

  1. Background desktop_key x ×3 into a focused Chrome text input: the keys landed (xxx). The Fill password fields: from the Keychain by reference, or agent-supplied text with approval #117 spike also saw background keys land in a Chrome password field.
  2. Background desktop_key with keys [tab, down]: focus moved to the select and its menu opened (AXMenu plus 5 AXMenuItems in the next snapshot).
  3. desktop_element_action AXPick on an item from that snapshot: refused with SNAPSHOT_STALE, which reproduces the issue. The next inspect showed the menu had closed by itself, and the select kept focus.
  4. One background desktop_key call with keys [down, down, down, down, enter]: the select's AXPopUpButton value read back "English (U.S.)" (the 4th item). Chrome never came forward.

Testing

  • npm test and npm run check pass.

Follow-up

  • A desktop_choose tool, or extending desktop_select, for AXPopUpButton: open the menu, find the item by title, and pick it in one call. This is out of scope here.

…ct in one desktop_key call

Live checks showed background keys do reach Chrome; the select's menu closing between calls is the real cause.
@iamnbutler iamnbutler changed the title fix: report keys to a background Chromium app as unknown, and document picking a native select in Chrome docs: pick a native select in Chrome with one desktop_key call Oct 19, 2026
@iamnbutler
iamnbutler merged commit ed95e8c into main Oct 19, 2026
2 checks passed
@iamnbutler
iamnbutler deleted the fix/chrome-select branch October 19, 2026 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native <select> in Chrome can't be picked by click/AXPick; keyboard only works after desktop_activate

1 participant