# 策展 · X (Twitter) 🔥🔥🔥

> 作者：Michał Pierzchała (@thymikee) · 平台：X (Twitter) · 日期：2026-05-12

> 原始來源：https://x.com/thymikee/status/2053876957125792213

## 中文摘要

agent-device實現Agentic UI開發透過CLI自動化app測試。

Michał Pierzchała（@thymikee）展示Agentic UI開發實例，使用`agent-device`剖析[Expensify](https://github.com/expensify)效能，強調僅需指示Agent呼叫`agent-device`即可運行真實app、檢查UI狀態、互動元素並收集除錯證據。

**專案概述**  
`agent-device`是專為AI Agent設計的裝置自動化CLI，支援行動、TV與桌面app。它以token高效的無障礙快照為核心，而非像素截圖；Agent可讀取精簡UI樹狀結構，透過如`@e3`的refs定位元素，執行觸控與文字動作，並僅在需要時擷取截圖、影片、log、網路、CPU/記憶體/效能資料、崩潰log及React剖析。Agent可直接攝取[llms-full.txt](https://incubator.callstack.com/agent-device/llms-full.txt)文件，CLI的`help`為權威指令語法來源。npm版本最新，CI建置通過，MIT授權。

**Agentic QA與開發應用**  
`agent-device`閉合Agentic開發迴圈：Agent撰寫程式碼、運行真實app、端到端驗證UI、收集截圖/影片/log/效能證據，並將bug、崩潰或效能問題回饋至下次修復迭代，人類僅在PR審核前介入。類似Vercel的[agent-browser](https://github.com/vercel-labs/agent-browser)，但針對app與裝置。  
- **品質保證**：dogfood流程、驗證PR建置、檢查無障礙涵蓋率，將穩定探索轉為`.ad`端到端測試。  
- **開發**：依規格建置、檢查真實運行行為、迭代直至UI符合需求。  
示範：Codex使用`agent-device`檢查iOS Contacts無障礙快照、互動UI，從簡單提示新增聯絡人。

**快速入門**  
先安裝CLI：  
```bash
npm install -g agent-device@latest
agent-device --version
agent-device help workflow
```
CLI help為Agent權威來源。技能為選用，CLI需`>= 0.14.0`以支援workflow help主題。

**AI Agent進入點**  
- **Agent + 終端**：在Cursor、Codex、Claude Code、Windsurf等客戶端整合終端運行`agent-device`，從`agent-device help workflow`規劃。  
- **技能或規則**：`npx skills add callstackincubator/agent-device`，使用內建[agent-device skill](skills/agent-device/SKILL.md)，或鏡像為專案規則，讓Agent檢查版本並讀取help。  
- **MCP路由**：MCP感知客戶端使用`agent-device mcp`，僅暴露`status`、`install`、`help`工具及workflow提示，不包含裝置自動化或shell執行。  
MCP設定範例（貼入Cursor `.cursor/mcp.json`）：  
全球安裝：  
```json
{
  "mcpServers": {
    "agent-device": {
      "command": "agent-device",
      "args": ["mcp"]
    }
  }
}
```  
無全球安裝：  
```json
{
  "mcpServers": {
    "agent-device": {
      "command": "npx",
      "args": ["-y", "agent-device@latest", "mcp"]
    }
  }
}
```  
詳細見[AI Agent Setup](https://incubator.callstack.com/agent-device/docs/agent-setup)及[llms-full.txt](https://incubator.callstack.com/agent-device/llms-full.txt)。

**MCP路由細節**  
`agent-device mcp`為stdio MCP路由，僅用於發現導向客戶端，暴露狀態、安裝及版本對應help；裝置控制仍透過CLI明確指令。註冊元數據使用MCP名稱`io.github.callstackincubator/agent-device`、npm套件`agent-device`、stdio傳輸、`mcpName`驗證、`server.json`及`smithery.yaml`。

**前5分鐘：Expo測試app**  
使用內建Expo fixture進行首次Agent運行，包含設定檢查、截圖、重播、效能證據。需repo checkout，因`examples/test-app`及`pnpm test-app:*`不在npm套件中。  
1. `git clone https://github.com/callstackincubator/agent-device.git`  
2. `cd agent-device`  
第一終端：  
```bash
pnpm test-app:install
cd examples/test-app
npx expo-doctor@latest
cd ../..
pnpm test-app:ios
# 或: pnpm test-app:android
```  
給Agent提示：  
```
Use agent-device to dogfood the bundled Expo app and produce an evidence-backed report.

Setup:
- Read `agent-device help workflow`, `agent-device help dogfood`, `agent-device help debugging`, and `agent-device help react-devtools` before planning commands.
- Confirm the test app setup commands were run: `pnpm test-app:install`, `cd examples/test-app && npx expo-doctor@latest`, then `pnpm test-app:ios` or `pnpm test-app:android`.
- If Metro prints an Expo URL, prefer opening the shell with that URL. On iOS use `agent-device open "Expo Go" <url> --platform ios`; on Android use the visible Expo/dev-client target or URL. Confirm the app UI with `snapshot -i`.

Run:
- Create `./dogfood-output/screenshots`, `./dogfood-output/videos`, `./dogfood-output/traces`, `./dogfood-output/perf`, and `./dogfood-output/replays`.
- Open a named session `expo-qa` and save a replay script to `./dogfood-output/replays/expo-test.ad`.
- Use command shapes like `agent-device --session expo-qa open "Expo Go" <url> --platform ios --save-script ./dogfood-output/replays/expo-test.ad`, `agent-device --session expo-qa screenshot ./dogfood-output/screenshots/home.png`, `agent-device --session expo-qa perf --json > ./dogfood-output/perf/baseline.json`, and `agent-device --session expo-qa record start ./dogfood-output/videos/checkout.mp4`.
- Capture a baseline `snapshot -i`, screenshot, and `perf --json` sample.
- Exercise Home, Catalog, product detail, Checkout, and Settings. Re-snapshot after each mutation and use refs/selectors from fresh snapshots.
- Capture at least one overlay-ref screenshot, one normal screenshot, one short video recording for a meaningful flow, logs marks around any issue, and trace output if a runtime symptom needs diagnostics.
- Run focused performance checks: compare `perf --json` before and after a navigation or form flow; if React DevTools connects, capture profile slow/rerender output. If it cannot connect, include the status and continue.
- Close the session so the `.ad` replay is written.

Report:
- Write `./dogfood-output/report.md`.
- Link every screenshot, video, trace, log path, replay file, and performance artifact you used.
- Include setup results, platform/device, Expo doctor outcome, coverage, severity counts, findings with repro commands, and a short performance section summarizing startup/CPU/memory/frame-health or React profile findings.
- If no issues are found, report covered flows and residual risk instead of claiming the app is bug-free.
```

**執行環境**  

| 路徑 | 最適用途 | 起始點 |
| --- | --- | --- |
| Local | 模擬器、模擬器、實機、macOS app、Linux桌面探索、除錯、開發迴圈 | Quick Start |
| CI/CD | PR與合併驗證，使用重播腳本與擷取文物 | [EAS workflow template](https://github.com/callstackincubator/eas-agent-device/blob/main/.eas/workflows/agent-qa-mobile.yml)（GitHub Actions模板即將推出） |
| Cloud/遠端 | Linux runner、管理裝置、遠端執行 | [Agent Device Cloud](https://agent-device.dev/cloud)，[Commands](https://incubator.callstack.com/agent-device/docs/commands)，或[聯絡Callstack](mailto:hello@callstack.com) |

**核心能力**  
- **平台**：iOS、Android、tvOS、Android TV、macOS、Linux；支援實機與模擬器。  
- **Agent原生UI模型**：token高效無障礙快照、當前畫面refs探索、選擇器耐久重播、技能驗證workflow指引。  
- **擷取與除錯**：截圖、影片、log、網路流量、CPU/記憶體/效能資料、崩潰log、無障礙快照、React渲染剖析。  
- **產出**：可重播`.ad`腳本（本地/CI運行）、端到端測試、快照/截圖差異、除錯文物。  
- **React Native與Expo**：元件樹檢查、props/狀態/hooks、渲染剖析。  
- **MCP邊界**：MCP僅發現與help；app/裝置控制經CLI明確、可稽核指令。  
- **授權**：MIT，免費使用。

**運作原理**  
透過平台後端執行session感知指令：iOS/tvOS用XCTest、Android用ADB加快照helper、macOS用本地helper、Linux桌面用AT-SPI。Node消費者可使用typed client及公開subpaths整合。`agent-device/android-adb`暴露Android ADB提供者契約、logcat/剪貼簿/鍵盤/app helper及port反向管理。前提：Node.js 22+、Xcode（iOS/tvOS/macOS）、Android SDK + ADB、macOS無障礙權限。見[Installation](https://incubator.callstack.com/agent-device/docs/installation)。範例迴圈：  
```bash
# 尋找app
agent-device apps --platform ios

# 啟動session
agent-device open SampleApp --platform ios

# 檢查畫面，-i僅互動元素
agent-device snapshot -i
# @e1 [heading] "Settings"
# @e2 [button] "Sign In"
# @e3 [text-field] "Email"

# 動作、擷取截圖、關閉
agent-device fill @e3 "test"
agent-device screenshot ./artifacts/settings.png
agent-device close
```

**使用團隊與文件**  
Callstack、Expensify、Shopify、Kindred、Total Wine & More、LegendList、HerLyfe、App & Flow等團隊使用中。文件包括[Installation](https://incubator.callstack.com/agent-device/docs/installation)、[AI Agent Setup](https://incubator.callstack.com/agent-device/docs/agent-setup)、[Typed Client](https://incubator.callstack.com/agent-device/docs/client-api)、[Commands](https://incubator.callstack.com/agent-device/docs/commands)、[Replay & E2E](https://incubator.callstack.com/agent-device/docs/replay-e2e)、[Security & Trust](https://incubator.callstack.com/agent-device/docs/security-trust)、[Known limitations](https://incubator.callstack.com/agent-device/docs/known-limitations)、[llms-full.txt](https://incubator.callstack.com/agent-device/llms-full.txt)。Agent整合：[agent-device skill](skills/agent-device/SKILL.md)、[react-devtools skill](skills/react-devtools/SKILL.md)、[dogfood skill](skills/dogfood/SKILL.md)、`agent-device mcp`、[ClawHub skill](https://clawhub.ai/okwasniewski/agent-device)。貢獻見`CONTRIBUTING.md`。官方站[agent-device.dev](https://agent-device.dev/)、[EAS template](https://github.com/callstackincubator/eas-agent-device/blob/main/.eas/workflows/agent-qa-mobile.yml)、[文件](https://incubator.callstack.com/agent-device/)、[聯絡](mailto:hello@callstack.com)。

## 標籤

Agent, CLI, 自動化, 開源專案, agent-device, Expensify