Skip to content

Add atomic native drags - #85

Merged
iamnbutler merged 6 commits into
feat/github-cachefrom
codex/desktop-drag
Oct 5, 2026
Merged

iamnbutler merged 6 commits into
feat/github-cachefrom
codex/desktop-drag

Conversation

@iamnbutler

@iamnbutler iamnbutler commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Adds desktop_drag: one bounded, straight-line press/move/release between distinct normalized screenshot points in the observed window, with left or right button. It reuses capture coordinates, single-use snapshots, native gesture ownership/release cleanup, and pi's unsafe-tool outcomes. The request preserves the snapshot's original focus evidence so it matches the complete capture-owned receipt; receipt validation remains strict.

Part of githubnext/desktop-tools#2 and githubnext/ace2#8. No button remains held across calls, and gestures cannot cross windows.

Validation:

  • Types, formatting/lint, Swift parsing, signed native build/signatures, and independent source review passed. Native candidate 2531311999293133809e845a47522ebcb090a4d2 is unchanged by final source bf930584700306ec379036c320dd1c61be4f6db9, which adds only reviewed tool/documentation guidance and passed types/lint again.
  • Real focused AppKit left/right drags moved and dropped the tile with exactly one press and release each. Invalid endpoints, stationary gestures, and invalid duration refused without input. One Retina display (native_scale: 2) and native 960×632 capture coordinates with Ace's resized 1600×1053 image were exercised.
  • The remaining direct check passed at bf930584: a freshly moved window refused its prior observation with a specific stale/bounds diagnostic and no receiver change. Cancelling only after an actual mouse-down returned unknown and delivered exactly one release; all native calls drained and the prior scroll positions remained unchanged. Evidence: drag-current-FkQMdR in the immutable ace-pointer-remaining-fixture.5mF516 directory.
  • A real pi/Anthropic run in the local Workers/Durable Objects runtime passed inspect → Activate click → one scroll → drag → inspect through the actual workspace adapter and signed native bridge. It made exactly five tool calls and three fixture mutations after separate explicit app activation/window focus setup. Receiver callbacks verified one click, scroll from 0 to 229, and one completed tile drop. Native outcomes and images crossed the workspace unchanged. Restarting the same persisted channel with the workspace offline replayed all five exact results/images with unchanged receiver state and provider usage. Evidence: hosted-model-YHsStf under the fresh ace-pointer-hosted-fixture.CPa0AR fixture.

Accepted delivery does not prove a drop: an inactive custom view ignored an earlier delivered gesture. After explicit focus and fresh inspection, separate left/right cases worked. The tool tells the model to verify the effect and explicitly focus then inspect again before deciding on another action; it adds no automatic foreground fallback or blind replay. A Move window setup changed its bounds and truthfully returned unknown after final identity validation, so its successful effect was preserved without replaying it.

Multi-display behavior, another physical machine, and crash-during-gesture recovery remain unverified. Pixels-only observations do not authorize pointer input.

Built in Ace

Git/GitHub, source-diff review, static checks, signed builds, and signature verification ran in dedicated Ace terminals. Native checks use Ace's signed bridge and actual receiver callbacks; the hosted check uses the real channel/workspace route and provider.

Codex external file tools edited source and prepared/read temporary validation artifacts because Ace lacks a file-editing surface (#22) and external-harness integration (#9). Codex computer-use launched the disposable fixture because a native application-launch tool is still pending in githubnext/ace2#8, and read-only observation checked fixture visibility and the desktop lock. No input success is inferred from those observations or compilation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant