computer-use-linux: MCP tool for Hermes Agent

Rust MCP server for controlling a Linux desktop through accessibility trees, screenshots and input

What computer-use-linux does

The server reads AT-SPI accessibility trees, lists and focuses windows, takes screenshots, and performs clicks, drags, scrolls and typing across GNOME, KDE/KWin, Hyprland, niri, i3 and COSMIC. It is Wayland-first with best-effort X11 support. Pointer actions can use the RemoteDesktop portal with ydotool as the fallback, and window targeting tries compositor-specific backends in order, then reports which one worked or why each failed.

Tools such as click, perform_action and set_value accept role, name, text and state selectors backed by AT-SPI, with pixel coordinates kept as a fallback for canvas or game surfaces. A doctor command returns one JSON readiness report covering platform, portals, accessibility, windowing and input, with explicit blockers and a recommended next step that an MCP host can show to the user. The project ships as a Rust crate, an npm wrapper and prebuilt binaries, and it was extracted from codex-desktop-linux, which still bundles it as a built-in plugin.

Key features

Accessibility-tree reading with role, name and text selectors
Screenshots through GNOME Shell, portals or X11 fallbacks
Click, drag, scroll and type on Wayland and X11
Compositor-aware window registry for GNOME, KDE, Hyprland, niri, i3 and COSMIC
doctor command with a JSON readiness report

When to use it

Letting a Hermes agent operate a Linux desktop application through MCP
Diagnosing why desktop control fails on a given Wayland setup
Automating GUI-only applications on a Linux machine

Who it is for: Linux users who want an MCP-capable agent to operate desktop applications.

How it fits with Hermes Agent

A general MCP server that names Hermes Agent as one MCP host that can spawn it, along with Codex Desktop and Claude Desktop.

How to install computer-use-linux

These commands are copied from the project's README. Check the repository for the latest steps before you run them.

npm install -g @agent-sh/computer-use-linux
computer-use-linux doctor | jq .readiness

Requirements: A Linux desktop session (Wayland-first, X11 best-effort), with ydotool and ydotoold as the Wayland input fallback

FAQ

What is computer-use-linux?

computer-use-linux is a Rust MCP server and CLI for Linux desktop control. It reads accessibility trees, lists windows, takes screenshots and performs clicks, scrolls and typing for any MCP host.

Does computer-use-linux work with Hermes Agent?

Yes. The README names Hermes Agent as one of the MCP hosts that can spawn the server, next to Codex Desktop, Claude Desktop and custom clients. The repository is also tagged hermes.

How do I install computer-use-linux?

Run npm install -g @agent-sh/computer-use-linux, then computer-use-linux doctor to get a JSON readiness report. A Rust crate and prebuilt binaries on the latest release are also available.