Skip to content

refactor(gui-agent): refactor composable gui agent with improved stability and error handling #1556

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
cjraft merged 6 commits into from
Sep 25, 2025
Merged

refactor(gui-agent): refactor composable gui agent with improved stability and error handling #1556

cjraft merged 6 commits into from
Sep 25, 2025

Conversation

@ZhaoHeh
Copy link
Collaborator

@ZhaoHeh ZhaoHeh commented Sep 17, 2025 •
edited
Loading

🚀Summary

This PR introduces a comprehensive refactoring of the gui Agent architecture to improve composability, stability, and error handling. The changes focus on modernizing the AIO Hybrid Operator, streamlining the GUI Agent T5 Adapter, and enhancing the overall agent plugin system.

🛠Key Improvements

  • AIO Hybrid Operator: Complete refactoring with improved API design and better error recovery mechanisms
  • GUI Agent Adapter: Significant simplification and optimization (377 lines removed, 99 lines added) with refactored action parser
  • GUI Agent Screenshot: Added AfterToolCall hook for screenshot input processing
  • AIO Browser Optimized: Improved connection stability and error recovery in AIO Browser

Details

1. API Modernization

  • Updated AIO hybrid operator with modern API design
  • Removed deprecated utility functions
  • Enhanced package dependencies

2. Action Parser Integration

  • Updated GUI agent with refactored action parser
  • Improved tool call processing pipeline

3. Plugin System Enhancement

  • Refactored agent plugin system with composable architecture
  • Added AfterToolCall hook for screenshot input processing
  • Improved agent lifecycle management

4. Bug Fixes

  • Fixed critical point undefined type error in scroll action handling

5. Enhanced Connection Stability

  • Improved connection stability and error recovery in AIO Browser
  • Added automatic reconnection logic for protocol errors
  • Better handling of connection closed scenarios

Checklist

  • Added or updated necessary tests (Optional).
  • Updated documentation to align with changes (Optional).
  • Verified no breaking changes, or prepared solutions for any occurring breaking changes (Optional).
  • My change does not involve the above items.

@netlify
Copy link

netlify bot commented Sep 17, 2025 •
edited
Loading

✅ Deploy Preview for agent-tars-docs ready!

Name Link
🔨 Latest commit dadd4dd
🔍 Latest deploy log https://app.netlify.com/projects/agent-tars-docs/deploys/68d4112ff064110008f0b385
😎 Deploy Preview https://deploy-preview-1556--agent-tars-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@ZhaoHeh ZhaoHeh force-pushed the dev/refact_composable_gui_agent branch from a8a5c2e to 85fe5e9 Compare September 21, 2025 05:58
@netlify
Copy link

netlify bot commented Sep 21, 2025 •
edited
Loading

✅ Deploy Preview for tarko ready!

Name Link
🔨 Latest commit dadd4dd
🔍 Latest deploy log https://app.netlify.com/projects/tarko/deploys/68d4112f59bbe8000822156a
😎 Deploy Preview https://deploy-preview-1556--tarko.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@ZhaoHeh ZhaoHeh changed the base branch from main to refact/gui_agent_sdk_917_v2 September 21, 2025 05:59
@ZhaoHeh ZhaoHeh marked this pull request as draft September 21, 2025 05:59
@ZhaoHeh ZhaoHeh force-pushed the dev/refact_composable_gui_agent branch 2 times, most recently from 816608a to 5ec438b Compare September 21, 2025 18:41
@ZhaoHeh
Copy link
Collaborator Author

ZhaoHeh commented Sep 21, 2025

Close: #1362

@ZhaoHeh ZhaoHeh mentioned this pull request Sep 21, 2025
Closed
4 tasks
@ZhaoHeh ZhaoHeh force-pushed the refact/gui_agent_sdk_917_v2 branch from 5de7a46 to 035293a Compare September 22, 2025 09:40
@ZhaoHeh ZhaoHeh force-pushed the dev/refact_composable_gui_agent branch from 5ec438b to 2c606f5 Compare September 22, 2025 12:13
Base automatically changed from refact/gui_agent_sdk_917_v2 to main September 23, 2025 06:51
@ZhaoHeh ZhaoHeh force-pushed the dev/refact_composable_gui_agent branch 2 times, most recently from f21d68a to 3198db3 Compare September 24, 2025 13:05
@ZhaoHeh ZhaoHeh changed the title refactor(gui-agent): update aio hybrid operator with refactored api refactor(gui-agent): refactor composable GUI agent architecture with improved stability and error handling Sep 24, 2025
@ZhaoHeh ZhaoHeh changed the title refactor(gui-agent): refactor composable GUI agent architecture with improved stability and error handling refactor(gui-agent): refactor composable gui agent with improved stability and error handling Sep 24, 2025
@ZhaoHeh ZhaoHeh requested review from cjraft and ulivz and removed request for cjraft September 24, 2025 15:33
@ZhaoHeh ZhaoHeh marked this pull request as ready for review September 24, 2025 15:34
@ZhaoHeh ZhaoHeh force-pushed the dev/refact_composable_gui_agent branch from 3198db3 to dadd4dd Compare September 24, 2025 15:41
@cjraft cjraft merged commit 3a5eeb8 into main Sep 25, 2025
10 checks passed
@cjraft cjraft deleted the dev/refact_composable_gui_agent branch September 25, 2025 02:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cjraft cjraft cjraft approved these changes

@ulivz ulivz Awaiting requested review from ulivz

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ZhaoHeh @cjraft