Awesome GUI Agent

一份精选的关于多模态图形用户界面（GUI）Agent 的论文、项目及资源列表。

构建一个屏幕上的数字助理。（图片由 DALL-E-3 生成）

欢迎贡献！

🔥 本项目处于活跃维护状态，欢迎您的贡献。如果您有任何建议，例如发现遗漏的论文或相关信息，请随时提交 Issue 或 Pull Request。

🤖 欢迎试用我们的 Awesome-Paper-Agent。只需提供 arXiv URL 链接，它就会自动返回格式化的信息，如下所示：

User:
https://arxiv.org/abs/2312.13108

GPT:
+ [AssistGUI: Task-Oriented Desktop Graphical User Interface Automation](https://arxiv.org/abs/2312.13108) (Dec. 2023)

  [![Star](https://img.shields.io/github/stars/showlab/assistgui.svg?style=social&label=Star)](https://github.com/showlab/assistgui)
  [![arXiv](https://img.shields.io/badge/arXiv-b31b1b.svg)](https://arxiv.org/abs/2312.13108)
  [![Website](https://img.shields.io/badge/Website-9cf)](https://showlab.github.io/assistgui/)

这样您就可以轻松复制并将其用于您的 Pull Request 中。

⭐ 如果您觉得此仓库对您有帮助，请给个 Star。

---

快速导航： [数据集 / 基准测试] [模型 / Agent] [综述] [项目]

数据集 / 基准测试

• World of Bits: An Open-Domain Platform for Web-Based Agents (Aug. 2017, ICML 2017)

  

• A Unified Solution for Structured Web Data Extraction (Jul. 2011, SIGIR 2011)

  

• Rico: A Mobile App Dataset for Building Data-Driven Design Applications (Oct. 2017)

  

• Reinforcement Learning on Web Interfaces using Workflow-Guided Exploration (Feb. 2018, ICLR 2018)

  

• Mapping Natural Language Instructions to Mobile UI Action Sequences (May. 2020, ACL 2020)

  

• WebSRC: A Dataset for Web-Based Structural Reading Comprehension (Jan. 2021, EMNLP 2021)

  

• AndroidEnv: A Reinforcement Learning Platform for Android (May. 2021)

  

• A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility (Feb. 2022)

  

• META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI (May. 2022)

  

• WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents (Jul. 2022)

  

• Language Models can Solve Computer Tasks