Get the latest tech news
Salesforce’s new CoAct-1 agents don’t just point and click — they write code to accomplish tasks faster and with greater success rates
CoAct-1 is an AI agent that combines GUI control with on-the-fly coding, making computer automation more robust and efficient.
Researchers at Salesforce and the University of Southern California have developed a new technique that gives computer-use agents the ability to execute code while navigating graphical user interfaces (GUIs), that is, writing scripts while also moving a cursor and/or clicking buttons on an application, combining the best of both approaches to speed up workflows and reduce errors. “This dynamic delegation allows CoAct-1 to strategically bypass inefficient GUI sequences in favor of robust, single-shot code execution where appropriate, while still leveraging visual interaction for tasks where it is indispensable,” the paper states. A purely GUI-based agent would need to perform a long, brittle sequence of clicks and drags, opening folders, selecting files, and navigating menus, with a high chance of error at each step.
Or read this on Venture Beat
