Computers for AI Agents
Give an AI agent a desktop machine it can see, click, type into and run commands on.
A desktop machine is a virtual machine that boots straight into a desktop: Ubuntu 24.04 with Xfce, Firefox and LibreOffice, signed in and ready. An AI agent uses it the way a person would, by looking at the screen and using the mouse and keyboard. It can also run shell commands for work a terminal does better.
You can watch the agent work from the console, and take over the mouse and keyboard yourself at any point.
Like every other service, a desktop machine runs inside your project's private network. An agent working on one can reach your databases and internal services by their private hostnames, and none of them need a public address.
Desktop machines are in beta.
Create one
From the console: open the command palette, choose Virtual Machine, and pick Ubuntu 24.04 Desktop as the image. SSH keys are optional for a desktop machine.
From an AI tool: with the MCP server connected, ask for one: "Create a computer called research in production." The agent calls create-computer and can start using the machine about a minute later.
A desktop machine starts with 2 vCPUs, 4 GB of memory and a 20 GB disk. 4 GB of memory and 20 GB of disk are also the minimums. It is billed as a virtual machine while it runs. Stop it from the console when you are not using it.
Watch and take control
Open the machine on the project canvas. The Desktop tab shows its screen live.
The view opens watching only, so a stray click does not land in the middle of an agent's work. Choose Take control to use the mouse and keyboard yourself, and Stop controlling to hand control back.
Only one person can view a machine's screen at a time. Opening it in a second tab disconnects the first.
Let an agent drive it
With MCP
The MCP server includes four tools for this:
| Tool | What it does |
|---|---|
create-computer | Creates and starts a desktop machine |
computer-screenshot | Returns the screen as an image |
computer-action | Clicks, moves, drags, types, presses keys, scrolls or waits, then returns a new screenshot |
computer-bash | Runs a shell command and returns its output and exit code |
The actions use the same vocabulary as the computer-use tools models are trained on (left_click, type, key, scroll, and so on), so most agents use them without further instruction.
With the API
The computer API lives under the machine's service path:
https://api.cloud.simplifyd.com/v1/workspaces/{workspace}/projects/{project}/envs/{env}/svcs/{service}/computerAuthenticate with an API token as Authorization: Bearer <token>. Request and response bodies are JSON, except the screenshot.
| Method and path | Body | Does |
|---|---|---|
GET /screenshot | The screen as a PNG. Add ?format=base64 for {"image", "media_type"}, ready for a model API | |
GET /display | {"width": 1280, "height": 800} | |
POST /click | {"x", "y", "button": "left" | "middle" | "right", "double": false} | Moves the pointer to (x, y) and clicks |
POST /mouse-move | {"x", "y"} | Moves the pointer |
POST /drag | {"start_x", "start_y", "end_x", "end_y"} | Drags with the left button held |
POST /type | {"text"} | Types into whatever has focus |
POST /key | {"key"} | Presses a key or chord: Return, ctrl+l, alt+Tab |
POST /scroll | {"x", "y", "direction": "up" | "down" | "left" | "right", "amount": 3} | Scrolls at (x, y) |
POST /wait | {"seconds"} | Waits up to 30 seconds |
POST /bash | {"command", "timeout": 60} | Runs a command; returns {"output", "exit_code", "timed_out", "truncated"} |
POST /exec | {"code", "timeout": 60} | Runs Python 3 code, with the same result as bash |
Coordinates are pixels from the top-left corner of the screen. The screen is 1280×800, and screenshots are taken at that size. That is the largest size computer-use models accept without scaling, so a point an agent picks out of a screenshot is the point it clicks.
Commands run as the desktop user (ubuntu, which can sudo) in its home directory, with the display set. A graphical program started from bash opens on the screen. End the command with & so the call returns:
curl -X POST "$SVC/computer/bash" \
-H "Authorization: Bearer $TOKEN" \
-d '{"command": "firefox https://news.ycombinator.com &"}'While the machine is stopped or still starting, every call returns 503. Wait a few seconds and try again.
With the Go SDK
computer := client.Workspace(ws).Project(p).Env(env).Services().Computer(svc)
png, err := computer.Screenshot(ctx)
err = computer.Click(ctx, 640, 400, cloud.MouseLeft)
err = computer.Type(ctx, "simplifyd cloud")
err = computer.Key(ctx, "Return")
out, err := computer.Bash(ctx, "ls ~/Downloads", time.Minute)What an agent can and cannot do
- It is a full machine. The agent can install software, keep files between sessions, and run long jobs. Anything it installs stays on the disk until you delete the machine.
- It has the internet. The browser can reach any public site. Do not sign it in to accounts you would not hand to the agent directly.
- It is yours to inspect. Everything the agent does happens on a screen you can watch, and in a shell you can open from the Console tab.
