feat(screenshot): support selector crops on CoreDevice iPhones

[camchis]

Summary

Accept physical iPhone selector crops only when CoreDevice/devicectl produced the image. XCTest devices, runner fallback, and unknown physical capture provenance are refused with the pending-evidence reason. Physical iPadOS remains rejected. Ordinary screenshots retain fallback behavior.

14 files; scope narrowed to the verified capture route. Refs #2276; addresses review feedback.

agent-device screenshot out.png --crop-on 'label="Continue"' --platform ios --udid "$UDID"

Validation

9cf6410, clean checkout, Node 22.20:

• pnpm check:affected --run --head 9cf64106: passed, exactCommit=true; 4,362 related tests passed. GitHub-owned coverage/native/device jobs and manual replay lanes remain separate obligations.
• Identical three-file regression command: aa9dec28 plus new tests 7 failed/33 passed; 9cf64106 40 passed. Failures detect missing backend facts and wrongly accepted XCTest/unknown crops.
• Independent Astra review: no findings.

Fresh unmodified-head run, 2026-10-20: wired iPhone 12, iOS 27.0.1, Xcode 26.6; rebuilt CLI and isolated daemon/session, subsequently cleaned up.

node bin/agent-device.mjs screenshot crop.png --crop-on 'label="23 objects"' --platform ios --udid "$UDID" --session reviewfix3839 --state-dir /private/tmp/ad-reviewfix-3839-state --json
xcrun devicectl device capture screenshot --device "$UDID" --destination raw.png

Both exit 0. Crop 163×43, raw 1170×2532, origin (767,221): 7,009/7,009 RGBA pixels identical, maximum channel difference 0. Offset controls differ; best ±4px offset (0,0).

Portrait CoreDevice only is live-verified. Landscape is unverified; XCTest/runner fallback crop refusal is unit-tested, without claiming live pixel evidence for those routes.

[thymikee]

The code looks right at 7ffc88b. The change is small: a matrix flag, docs, help text and tests. The one CI check is green, and it covers these files. There are no conflicts.

One question before this goes further: the ios-physical cell in https://github.com/callstack/agent-device/blob/7ffc88b/packages/daemon/src/execution/screenshot-crop-target.ts#L49 covers every physical iPhone and iPad, plus the XCTest-backed physical backend and the devicectl-to-runner fallback, but the live run covers one portrait iPhone 12 on the default devicectl route. On an iPad the largest viewport root can be the app window, so the crop could be wrong. Would you give iPadOS its own rejected cell, or word the help and docs as iPhone-only verified?

Not blocking, and you can take or leave these: the matrix test in https://github.com/callstack/agent-device/blob/7ffc88b/packages/daemon/src/execution/__tests__/screenshot-crop-target.test.ts#L63 hard-codes the accepted cells in an inline if-chain, so one shared list would keep it in sync, and the new fractional fixture adds rounding coverage but repeats the 3x law that the simulator case already pins.

I did not run the tests locally. I took the iPhone 12 numbers from the PR body and could not re-inspect that run. Landscape, iPad, Split View, the XCTest-backed backend, the devicectl-to-runner fallback and Appium-produced physical iOS trees were not exercised.

[camchis]

Thanks. I took the iPad point and gave physical iPadOS its own rejected cell, ipados-physical, in aa9dec28. It refuses with PENDING_PIXEL_IDENTITY_EVIDENCE, and a physical iPad's refusal names its own cell. iPad simulators stay in the simulator cell as before. The help and docs now say "physical iPhones", and the PR body says no iPad, including Split View, was verified.

I also replaced the if-chain in the matrix test with a shared ACCEPTED_CROP_TARGETS list. I left the fractional fixture's 3x check alone, since it also pins the physical iPhone's own scale rather than the simulator's; I'll drop it if you'd rather not carry it.

check:affected --run --head aa9dec28 passes on Node 22.20 with exactCommit=true. The iPhone 12 run in the body is still at 7ffc88be; this commit only moves iPadOS to a rejected cell.

[thymikee]

The iPad point from my 7ffc88b review is fixed in aa9dec2, thank you. The XCTest backend and the runner fallback from that review are still open, so the code is not ready to merge yet.

The ios-physical cell in screenshot-crop-target.ts is accepted for every physical iOS device. The matrix header says a cell is accepted only after a live pixel-identity check. The one live run is a portrait iPhone 12 on the devicectl capture route. The same cell also admits devices with iosPhysicalDeviceBackend: 'xctest' and the devicectl-to-runner fallback. Both capture through the runner, and neither has evidence. On those routes screenshot --crop-on can return a crop box from a frame or density that no run measured. That is a silently wrong crop, not a refusal. An iPad found through xctrace under a name without "iPad" also still lands in ios-physical. Please make the rule that a cell is accepted only for the capture route the pixel-identity run used. The simplest way is to split the cell on device.iosPhysicalDeviceBackend: CoreDevice stays accepted, and xctest becomes a rejected ios-physical-xctest cell with the pending reason, so the one-to-one matrix test covers it. If you prefer to keep xctest accepted, please add live pixel-identity runs on a physical iPhone for the XCTest-backed device and for the runner capture path, and add a productType-based appleOs for xctrace discovery.

For those runs, please show a non-trivially placed --crop-on result where the decoded crop equals an independent raw capture, with the maximum channel difference in the PR body. Please also say how the devicectl-to-runner fallback is covered, or that it is not. A landscape iPhone is not covered yet either, so the help and docs should claim no more than ran.

All 14 checks pass. No device lane runs the physical iOS screenshot route, so CI says nothing about the xctest backend or the runner fallback. There are no known conflicts. The iPhone 12 numbers come from the PR body. Before merge, either narrow the accepted physical iOS cell to the CoreDevice backend with a rejected xctest cell, or add the live evidence above.

[camchis]

Addressed in 9cf64106: XCTest devices are now rejected, and crop checks the actual capture backend so runner fallback cannot bypass the restriction. Help/docs explicitly limit verification to portrait CoreDevice captures.

Retested the unmodified commit on the iPhone 12: all 7,009 crop pixels match the independent raw capture exactly, maximum channel difference 0. Updated evidence is in the PR body. Local validation passed, including 4,362 tests.

[thymikee]

This PR is ready. At 9cf6410, the XCTest and runner-fallback problems from the earlier review are fixed, and the live run covers the accepted portrait CoreDevice case. Help and docs claim only what ran.

All 14 checks pass. No CI lane runs the physical iOS screenshot route, so CI says nothing about the devicectl or runner paths. I did not reproduce the iPhone 12 pixel-identity numbers; they come from the PR body. I did not run the new tests; I read the code and judge that they would fail before this change. Cloud-provider physical iOS routes (Limrun, BrowserStack, AWS) return no capture backend, so they are now refused after capture. That fails closed, but I did not exercise it live.

Not blocking, and you can take or leave these: the 'coredevice' | 'xctest' union at interactor-types.ts#L80 can be typed as DeviceInfo['iosPhysicalDeviceBackend'], and the post-capture refusal at screenshot-crop-target.ts#L169 uses a name that is not a CropTarget cell and fires after the full PNG is already written to outPath, so reusing the ios-physical-xctest cell name and removing outPath on refusal would be tidier.

No conflicts.