What is a computer use AI agent?
A computer use AI agent operates software through its graphical interface: it reads the screen, moves the cursor, fills in fields and clicks, with no API needed on the application side. An API integration calls documented functions instead, and gets a structured response back. Our rule: the API whenever it covers the operation, computer use for systems you cannot integrate.
OpenAI unveiled GPT-6 Astra on 3 September 2026 and put its ability to use a computer the way a person does at the centre of the launch (OpenAI announcement and system card, 3 September). Two days earlier, on 1 September, Anthropic had released Claude Fable 5.1, whose announcement shows progress on OSWorld 2.0, the reference benchmark for computer use, without building its pitch around it. The capability is settled on both sides. What architecture boards are working through this autumn is which mode they take as the default, and what they accept losing by taking it.
A capability in public beta since 2024
The capability itself is not new. Anthropic opened it in public beta on 22 October 2024, describing it at the time as experimental and still error-prone, and vendors have refined it since. What changed in September 2026 is its place in the sales narrative: it moved from side feature to architectural promise.
Why is this decision reaching architecture boards now?
Because the promise speaks first to the budget holders. A computer use agent can reach applications that expose nothing: a thick client from the 2000s, a vendor-locked business package, a supplier extranet with no programmable interface. Most large European enterprises run a few of those, and they stall automation projects at the estimation stage far more often than any limit of the models does.
Since Astra shipped, the same line keeps coming back to us from engineering leaders: integration is optional now. Taken at face value, the line is wrong. It also holds enough truth to deserve a written decision instead of a corridor debate, because the two modes call for different governance.
How does an agent drive an interface, and how does it call an API?
The capture, decide, act loop
In computer use, the agent works in a loop. The program around it, usually called the harness, takes a screenshot and sends it to the model along with the instruction. The model answers with one elementary action: click at these coordinates, type this text, press a key, scroll. The harness executes it in a browser or a virtual machine, takes a new screenshot, and the cycle repeats until the model decides the task is done.
The weak point is how the agent knows the state of the system: it infers it from what it sees. It knows a field is filled because the next screenshot shows it filled, and nothing confirms that the write reached the database unless a confirmation screen says so. A label moved by an update is enough to throw the whole sequence off.
Tool calls and the MCP protocol
An API integration works differently. The agent has tools, meaning functions described by a name, parameters and a response format. When it decides to use one, it produces a structured call, the target system executes it and returns an explicit result: a transaction identifier and a status, or an error code. The agent reads the state of the system in that response.
The Model Context Protocol (MCP), an open standard created by Anthropic and entrusted since 9 December 2025 to the Agentic AI Foundation, hosted by the Linux Foundation, standardises these mechanics. An MCP server exposes an application’s functions, and any Claude agent or other compatible client can discover and call them the same way. The connector is written once, versioned and tested like any other component of the IT estate. It still takes a programmable interface and someone to write that connector.
Many architectures combine both. In a logistics department, for instance, the same agent can read orders from the ERP through an API, then enter a shipping notice on the extranet of a carrier that offers no programmable interface. The mode is chosen action by action, at the same granularity as governance.
Computer use or API integration: how do you decide?
Use the API whenever one exists and covers the operation. Keep computer use for systems you cannot integrate, or cannot integrate within the time the project has. The first mode can be tested and replayed identically, while the second needs continuous monitoring for as long as it runs.
The decision grid, use case by use case
Before any estimate, ask these questions for each use case and record the answers in the architecture file.
- Does an API cover the operation end to end, or only the read side?
- What does an error cost: a line to fix tomorrow, or an irreversible write in a production database?
- How will you know the agent got it wrong, and after how long?
- Who approves before the write, and at which point in the chain?
The approval before the write, often without an owner
The fourth question is the one that stays unanswered in the scoping workshops we run. The agent is scoped technically, its write perimeter is documented, and nobody can say who picks up the alert when it lands at seven on a Friday evening.
Task frequency and volume
When the grid leaves a case open, frequency weighs first. A daily task pays back the cost of writing a connector, and every run becomes safer and uses fewer model calls. A monthly entry into an old screen rarely justifies that work. That is where computer use is easiest to defend, provided the volume stays low enough for a person to review every write.
Stability of the target interface
An agent that clicks depends on the layout. A web application that redesigns its screens with every release breaks sequences without warning, while a thick client frozen for years is a more predictable target. The oldest systems often make the best candidates for screen-driving, precisely because nobody touches them any more.
Reversibility of the write
Reading a screen to extract information carries far less exposure than approving a payment on it. A first computer use deployment is better kept read-only, whether that means opening a case file or checking a status. Writes come later, one operation at a time, once the action log shows that the agent finds its way around the company’s real screens.
Why do benchmarks not decide for you?
Published scores on OSWorld move a lot depending on the evaluation protocol. In its 1 September announcement, Anthropic lists two far-apart OSWorld 2.0 scores for Fable 5.1, one under partial scoring and one under strict scoring, and adds in a footnote that they cannot be compared directly with OSWorld 2.0 results published earlier. The gap stops being a methodological footnote the day it is used to justify an architecture choice in a steering committee.
A measurement to produce on your own tasks
An agent’s success rate on a public bench says little about its behaviour on your ERP, with your permissions and your customised screens, on a month-end closing day. The only measurement that commits you is the one you produce on your own tasks, with your own data. Gartner (2025) expects more than 40% of agentic AI projects to be cancelled by the end of 2027. We often see one cause at the start: an architecture decision settled on figures nobody replayed in house.
What does this change for a European enterprise?
Action traceability under the EU AI Act
Traceability first. An agent that clicks leaves a thin trail: a series of screenshots and coordinates, with no transaction identifier and no return code. An agent that calls an API leaves an auditable record, with the call and its timestamped response. Since 2 August 2026, the Article 50 transparency duties of the EU AI Act apply, and a breach falls under the regulation’s penalty regime. The Digital Omnibus on AI, Regulation (EU) 2026/1744, in force since 27 July 2026, gives systems already on the market before 2 August until 2 December 2026 to apply the Article 50(2) content marking, and pushes the substantive requirements for Annex III high-risk systems back to 2 December 2027. That buys time on heavy documentation and none at all on being able to answer a regulator asking who did what, when and on which system.
National enforcement is still taking shape: several member states, France among them, have yet to formally designate their competent authorities. For a group operating in several EU countries, the audit trail has to satisfy whichever authority asks first.
The identity the agent acts under
Identity second. A computer use agent inherits a session, and with it the permissions of whoever opened it. Many organisations discover at that point that they have no service account for an agent, and no clear answer on accountability for a write performed under the account of a colleague on leave. According to Deloitte (State of AI in the Enterprise, 2026 edition), only 21% of companies have mature governance of their agents. Computer use widens the surface to govern before that figure has moved.
Security teams know the remedy: a dedicated service account for each agent, restricted to the rights its tasks require. In legacy software that never planned for a non-human user, creating that profile takes time, so it is best started before the pilot.
Which governance applies to each mode of action?
The LOOP™ governance protocol assigns every possible action of an agent to one of its 4 zones before go-live. We recommend bringing the technical mode into that assignment: the same operation can land in a different zone depending on whether it runs through an API or through computer use, because the quality of the trail and the confidence level differ.
Green zone: reads and reversible operations
In the green zone, the agent acts without validation, with confidence above 90%, and every action is traced. For a computer use agent, we advise keeping that zone mostly to reads. A write through an API, reversible and logged with its transaction identifier, qualifies more easily than a write through computer use, whose trail comes down to screenshots.
Orange and red zones: writes and unexpected situations
The orange zone covers actions with confidence between 70 and 90%: the agent proposes, and a designated person validates before execution. We recommend placing any computer use write into a production system there by default, until the log has shown the agent to be reliable. Below 70% confidence, the red zone requires escalation: the agent stops, documents what it understood and what it is missing, alerts the designated owner, and a human decision is required within 4 hours, with no automatic action. A screen that differs from the expected one or an unfamiliar error message should push the agent into that zone.
Black zone: what the screen can let in
An agent that reads the screen also reads whatever a third party has put there. A web page or an email can carry hidden instructions meant to divert the agent from its task, which is known as indirect prompt injection. LOOP™ places injection attempts in the black zone, with an immediate block, a CISO alert and a full stop of execution, in the same way as access to data outside the authorised classification. For an agent that browses external sites, this risk is handled at design time, by restricting the domains it can reach.
This assignment is prepared before deployment. Claude Ops, our industrialisation programme, starts with an IT and data audit that maps available APIs and the integrations still to build, and that map is then used to decide, system by system, which mode each agent will use.
Which mistakes come up in early projects?
Using computer use to bypass IT
A business team that has been waiting months for an integration sees screen-driving as a way around it. The agent then runs on an employee’s workstation, under that employee’s account, outside any inventory. The result looks like the spreadsheet macros nobody dares to open any more, with a model on top: IT cannot tell how many agents are running or what they change. Screen-driving belongs in the IT catalogue, under the same registration rules as every other automation.
Measuring success without measuring errors
Many pilots track a task completion rate. The metric that matters in operations is the time between an agent error and its detection, which answers the third question of the grid. An agent that gets most of its entries right but whose errors stay invisible for a month costs more than a slower agent whose every slip surfaces the same day.
Where should you start?
With an inventory, before any pilot. List the tasks you wanted to automate and shelved because no API was available. We keep finding that list in IT departments, in the form of tickets closed as out of scope. It marks the real ground for computer use, narrower than the sales promise, and it can be costed task by task.
A first instrumented task
Then take a single task from that list, one with a low error cost, and instrument it before automating it: an action log and a named human approval point. According to S&P Global (2025), 46% of AI proofs of concept are abandoned before production. From what we observe, the projects that get past that point have usually settled observability before writing the first prompt.
To see where your organisation stands, our AI maturity assessment scores it in three minutes on the 6 dimensions of the Koneetiv framework (2026 edition): Strategy & vision, Data & technical foundations, Adoption & usage, Industrialisation, Skills and Governance. The questions in this article fall mainly under Industrialisation, which covers IT integration, and Governance, which covers decision traceability. The wider picture is set out on our page about AI agents in the enterprise.
See also our analysis of moving from pilot to production, our guide to what an enterprise AI agent is and the AI agents in production section.