poligon-mcp

Русская версия

An MCP server that lets an AI agent reproduce bugs in disposable VMs on a Proxmox VE cluster: clone a template, boot it, drive it with a shell (qemu-guest-agent) or with mouse/keyboard over the VNC console, take screenshots, then throw the VM away. Every VM it creates and touches lives in one pool, named <owner>-repro-* after the token's user; it refuses to touch anything else.

Tools

Tool names say what a tool acts on, not which hypervisor runs it: vm_ is the VM's lifecycle and its guest (scenarios become vm_<name> too), ui_ the console screen and input, stand_ the multi-VM stands. Proxmox VE is the only backend today; another one (libvirt) would sit behind the same names.

Every tool takes an optional bench argument naming which stand to run on (default: the default stand, i.e. --api). Configure extra stands with POLIGON_MCP_STANDS (a JSON array of {name, type: pve|pikvm|megarac, api, token?}): a pikvm stand drives a PiKVM/BLIKVM (kvmd: console, HID, ATX power, virtual media), a megarac stand an AMI MegaRAC BMC (IPMI KVM, remote media from POLIGON_MCP_STORAGES); bench_status, bench_lease/bench_release, bench_power, bench_media and storage_list are their tools. See Multi-stand.

Tool	Args	What it does
vm_list_templates	filter?	Clonable templates (not-for-clone ones are hidden). Without filter: TSV family\tvariants, one line per family (p11-workstation, sisyphus-kworkstation, ...); with a substring filter: TSV vmid\tname\tnode, the same name on several nodes folded into one line
vm_list_my_vms	—	Your VMs: <owner>-repro-* in the caller's pool(s), TSV vmid\tname\tnode\tstatus
vm_task_wait	upid, timeout=50	Wait for a PVE task to finish, 50 s per call at most; answers done or still running ..., call again
vm_clone	template_vmid, name?, full=true, pool?, wait=30	Clone a template into the caller's pool as <owner>-repro-<name>; answers vmid N <name> on <node>, with task still running, upid=... appended if the clone outlives wait (max 50); a second clone with the same name is refused, see Long operations
iso_list	filter?, storage?	Installer ISOs on the cluster storage: TSV volid, size
vm_create_iso	name, iso, disk_gb=32, mem_mb=4096, cores=2, disks=1, uefi=false, pool?, storage?	Create a VM that boots from an installer ISO (installer bugs: partitioning, RAID, LUKS, profiles), named <owner>-repro-<name>; no guest agent until installed, so drive the installer over the console; answers vmid N name on node
vm_eject	vmid	Remove the installer CD (yours only) so the VM boots from disk after install
vm_disk_resize	vmid, disk=scsi0, size_gb	Grow a VM disk (yours only) when the installer's auto layout needs more
vm_disk_add	vmid, size_gb, count=1	Add scsi1..N disks on the same storage (yours only), for RAID install bugs
vm_start	vmid, wait=30	Start a VM (yours only)
vm_stop	vmid, wait=30	Stop (power off) a VM (yours only)
vm_wait_boot	vmid, timeout=50	Wait until the guest agent answers (VM booted), 50 s per call at most; not booted after N s means call again
vm_snapshot	vmid, name, wait=30	Create a snapshot of a VM (yours only)
vm_rollback	vmid, name, wait=30	Roll a VM back to a snapshot and start it (yours only)
vm_destroy	vmid, confirm=true, wait=30	Destroy a VM and purge its disks (yours only, confirm=true required); stops it first if running
vm_exec	vmid, command, timeout=120	Run a shell command in the guest as root via qemu-guest-agent with umask 022 (qemu-ga's own is 077, which made files written with > invisible to the desktop user), in the guest's own locale (prefix LC_ALL=C yourself for English rpm/apt output); answers exit N, then stdout as is, then stderr after a --- stderr line when there is any
vm_run	vmid, command, wait=45, restart?	Run a long command detached as root (dist-upgrade, a reboot, a build) and wait up to wait s (max 50): done exit N after S s, failed exit N … or running for S s, then the last 20 lines of its output. Call again with the same command to keep waiting; a finished run is reported, not repeated (restart=true runs it again); a guest agent that stops answering across a reboot is waited for, not an error. State lives in /var/tmp/poligon-run in the guest
vm_exec_gui	vmid, command, user=test, screenshot?, settle_ms=5000, scale?, wait_window?	Launch a GUI program inside the user's desktop session, detached, with the session's own DISPLAY/XAUTHORITY/WAYLAND_DISPLAY/DBUS_SESSION_BUS_ADDRESS/LANG/LC_* (read from the session process); answers whether it still runs after 2 s and its first output lines
vm_file_read	vmid, path, offset=0, max_bytes=16384	Read a file from the guest: text as is, a window of max_bytes from offset with a last line … truncated at N of M bytes, continue with offset=N; a PNG/JPEG comes back as an image
vm_file_write	vmid, path, content_base64	Write a file into the guest
ui_wait_for	vmid, window?, gone?, timeout=20, user=test, screenshot?, scale?	Wait for the screen instead of guessing a delay: with window until a window whose title, class or class name contains it is visible (gone=true: until it is not), without it until the screen stops changing (two frames 400 ms apart identical). Answers window … after 1.8 s at x,y WxH, screen settled after S s or timeout after S s; screenshot=true adds the image
ui_screenshot	vmid, path?, scale?, region?, user=test	Screenshot of the VM console as an image, with one size line (1280x800); scale=0.5 for a cheap state check, region=<X window id, or a substring of the title, class or class name> for one window (an X11 session of user, a Wayland session is refused with a hint; xdotool is installed in the guest on first use with apt-get install -y xdotool, and the answer then says installed xdotool in guest; when that fails the error's hint says to fall back to the full frame; when no window matches, the hint lists the visible ones); path also saves the full-size PNG in the server's evidence dir
ui_see	vmid, path?, scale?, region?, user=test, no_image?	Screenshot plus UI element detection by a vision model: the image and a TSV x\ty\ttype\tlabel in full-size console pixels (detection always runs on the full frame, only the image is downscaled; elements: 0 when it found none). no_image=true returns just the TSV
ui_click	vmid, x, y, button=left, count=1, screenshot?, settle_ms=3000, scale?, wait_window?	Mouse click at console coordinates: move, press 300ms, release; count=2 for double-click. screenshot=true returns the post-action state in the same call as soon as the screen stops changing (settle_ms is the longest wait), or once wait_window is visible: the element TSV plus a timing: line, and the image only with image=true (the list is cheaper and an unchanged screen is served from cache); otherwise ok
ui_read	vmid	On-screen text with coordinates by OCR (tesseract rus+eng): TSV x, y, label, ~0.4 s, no image; a highlighted row (list/menu selection) gets a selected marker. Fast screen read; text only, no element types or icons
ui_click_label	vmid, label, kind?, button=left, count=1, screenshot?, image?	Click a UI element by its visible label, no coordinates needed: OCR first (fast, exact box), then the vision model's element list of the same screen (exact coordinates; OCR misses menu items), then the grounding model (UGround) for icons; the answer says source: ocr|elements|uground. A grounding point into the top ~30 px is refused as not found — with a menu open that click would only close it
ui_drag	vmid, x1, y1, x2, y2, screenshot?, settle_ms?, scale?, wait_window?	Drag with the left button from (x1,y1) to (x2,y2)
ui_type	vmid, text, screenshot?, settle_ms?, scale?, wait_window?	Type text into the console character by character (keysym events; Shift is held for capitals and the shifted US symbols, so & and _ arrive as typed)
ui_key	vmid, key, screenshot?, settle_ms?, scale?, wait_window?	Press a key or combination by qemu sendkey name (ret, tab, esc, ctrl-alt-f2, alt-f2, ctrl-c, ...)
stand_plan	goal	Ask the planner model to write a stand plan for a goal in plain language (see Stands)
stand_apply	plan, pool?	Build a stand (1-4 VMs) in the background; returns at once
stand_status	stand?	Progress of a stand, or the list of your stands
stand_destroy	stand, confirm=true, force?	Destroy every <owner>-repro-<stand>-* VM of yours (force=true also stops a running build)
bench_status	—	The stands (benches): name, type, api, whether you may use it, any active lease, and a description of the machine — pick the right one and find a free one before bench_lease
bench_lease	bench, minutes≤480	Lease a stand so nobody else uses it while you test (extends your own); refused while another holds it; needed before tools on a rental stand
bench_release	bench	Release your lease early
bench_power	bench, action	Power a physical machine: pikvm via the ATX button (`on
bench_media	bench, action, image?, filter?, mode?, storage?, path?	Virtual media on a physical stand: `list
storage_list	—	The distribution storages: named NFS/CIFS shares, HTTP mirrors or ISO pools where install images live (name, type, address, user, description) — where a distro comes from before bench_media or vm_create_iso
dataset_status	—	Training-dataset capture: whether POLIGON_MCP_DATASET is on and what it holds — shots, sessions, AT-SPI/OCR coverage, clicks with and without intent, sessionless skips, rescales, and the share of panel/tree/toolbar screens against the 30% target (JSON; needs the inventory key when one is set)

Console coordinates are console pixels — see the size line a ui_screenshot or ui_see call returns before clicking (with scale it names both sizes, e.g. 640x400 (scale 0.5 of 1280x800); coordinates are always full-size); a text-mode console is commonly 1024x768, a graphical one may be larger.

Element lists (ui_see, POLIGON_MCP_SEE_ALWAYS) are cached by frame digest, not by VM: the same pixels from any VM, bench or caller are served from cache (timing: vision 0.0 s (source: cached)) instead of costing another vision call. Entries live 300 s and at most 512 frames are kept, so a long-running server cannot grow the cache without bound. The cache is process-wide by design: identical pixels from any user, VM or bench share one entry, since the answer depends on the pixels and on the pipeline (fixed for the process's lifetime), not on who asks — a hit does reveal that the same screen was recognized before, which is accepted here. A region= crop is never cached (its coordinates are shifted, so its list answers another question), and neither is a capped or unparseable model answer — a partial list is never served as a complete one.

Answers are plain text or TSV, not JSON, everywhere the model only reads them; errors are error: ... plus a hint: ... line. JSON stays where the model passes the structure on (stand_plan → stand_apply, stand_status).

Scenarios (server-side high-level tools)

A scenario is a YAML file describing a step list against one VM; the server turns each file into its own MCP tool named vm_<name>. Point the server at a directory of them with --scenarios DIR or POLIGON_MCP_SCENARIOS=DIR; they are loaded once at startup, validated, and registered as tools — a missing or empty directory is fine (0 scenarios). Editing or adding a file requires a server restart to take effect; there is no hot reload.

YAML fields:

• name — must match ^[a-z][a-z0-9_]*$; the tool becomes vm_<name>.
• description — the tool's description.
• params — map of paramname: {description?, default?}. A param with no default is required; every param is a plain string. vmid (integer) is always added to the tool's schema and does not need to be listed.
• steps — the step list, run in order.

Step kinds:

Kind	Args	Fails when
exec	a command string, or {command, timeout=120}	never fails on exit code (see expect)
exec_gui	{command, user=test}	the user does not exist in the guest
expect	{command, timeout=120, contains?}	exit code ≠ 0, or contains given and missing from stdout
wait_boot	seconds	guest agent doesn't answer before the timeout
sleep	seconds, ≤ 600	never (just waits)
screenshot	a path, or "" for image-only (response has no path written)	the path leaves the evidence dir, or the server has none (--evidence-dir)
click	{x, y, button=left, count≤3}	coordinates outside the console
click_label	{label, type?, button=left, count=1} — screenshots, asks the vision model where the element labelled label is (type narrows it: icon, button, ...) and clicks there	the vision model is not configured (PV_AI_KEY + PV_AI_API), or it answers with no point
drag	{x1, y1, x2, y2}	coordinates outside the console
type	text, ≤ 4096 chars	text too long
key	a qemu sendkey name	PVE rejects the key name
snapshot	snapshot name	the PVE task fails
rollback	snapshot name (restarts the VM)	the PVE task fails
recipe	{name, <param>: <value>, ...} — runs another scenario's steps inline (see Stands)	unknown recipe, missing required param, unknown param, nesting deeper than 3

Any string field in any step (and type/screenshot text) may contain {{param}} or {{vmid}} placeholders. An unknown placeholder makes the server refuse to start, as does a duplicate scenario name or a name that collides with a built-in tool.

The answer is text plus an image block with the last screenshot taken (if any). Success: ok <scenario> vmid N: <k> steps, followed by the stdout of the last exec/expect step (vm_install ends with rpm -q, so that is one name-version-release.arch line per package). The first failing step stops the run, and the answer is an error result naming only that step: error: step 11 (expect: pgrep ...) failed: expect failed: exitcode 1, stdout "...", stderr "..." (the last 1500 chars of each), a hint: line and 10 steps before it passed. Step labels are rendered from the unrendered YAML template, so a param's YAML default, e.g. a password, is never echoed back even if it was actually used.

Vision-model coordinates (click_label, ui_see) may be off by 5-15%, and a grounding model (Holo2) never says "not found" — asked for a label that is not on screen it still returns a point, typically in the top bar, where the power button lives. So follow such a step with a screenshot or an expect that verifies the click actually landed, and keep labels exact.

Ownership rules are the same as everywhere else: the VM must be yours (<owner>-repro-* in your pool). The vmid lock is held for the whole scenario run, not just one step. While it is held, a plain console tool (ui_screenshot/ui_see/ui_click/ui_drag/ui_type/ui_key) on the same VM waits at most 30 s and then fails with vmid N is busy instead of blocking.

Security note: scenario files are trusted server configuration — their exec commands run as root in the guest, exec_gui runs as the target desktop user via runuser (still trusted input), and their screenshot paths are interpolated with caller-supplied params, so keep the scenarios directory writable by the operator only.

Seven scenarios ship in scenarios/ — four tools and three recipes (tool: false, see Stands). login drives any greeter (ALT's KDE Workstation uses lightdm, lightdm-kde; the GNOME Workstation uses GDM; both take Enter, password, Enter) and picks the session through AccountsService when asked — session=mate logs into MATE installed on top of a GNOME template. gnome_login and lightdm_login are one-recipe wrappers around it:

scenarios/login.yaml:

name: login
description: Log into the graphical session through the display manager (GDM or lightdm, ...) as a desktop user (default test/test) ... picks the session (auto = the display manager's default; gnome, plasma, mate or xfce via AccountsService) ...
params:
  user: {description: "desktop user", default: test}
  password: {description: "password to set and type", default: test}
  session: {description: "auto (display manager default), gnome, plasma, mate or xfce", default: auto}
steps:
  - exec: "echo {{user}}:{{password}} | chpasswd"
  ...
  - expect: {command: "case {{session}} in gnome) p=gnome-shell;; ... esac; pgrep -u {{user}} -x \"$p\"", timeout: 30}
  - screenshot: ""

scenarios/gnome_login.yaml:

name: gnome_login
description: Log into the GNOME session through the GDM greeter as a desktop user (default test/test): vm_login with session=gnome. ...
params:
  user: {description: "desktop user", default: test}
  password: {description: "password to set and type", default: test}
steps:
  - recipe: {name: login, user: "{{user}}", password: "{{password}}", session: gnome}

scenarios/lightdm_login.yaml:

name: lightdm_login
description: Log into the Plasma session through the lightdm greeter (lightdm-kde; ALT's KDE Workstation uses lightdm, not SDDM) as a desktop user (default test/test): vm_login with session=plasma. ...
params:
  user: {description: "desktop user", default: test}
  password: {description: "password to set and type", default: test}
steps:
  - recipe: {name: login, user: "{{user}}", password: "{{password}}", session: plasma}

scenarios/install.yaml:

name: install
description: Install packages with apt-get inside the VM and report their versions (rpm -q). apt-get runs in the background and the call waits up to 40 s for it: on "still installing" call vm_install again with the same packages to keep waiting ...
params:
  packages: {description: "space-separated package names"}
steps:
  - wait_boot: 60
  - exec: ...   # start apt-get detached, state in /run/poligon-install.*, unless it is already running or done for these packages
  - expect: ... # wait up to 40 s; "still installing after N s" (exit 0), the apt-get log tail on failure, else rpm -q {{packages}}

An install takes minutes and an MCP client drops a tool call at about a minute, so vm_install never blocks on apt-get: it answers still installing after 40 s and a second call with the same packages picks the run up where it is (a finished run of the same packages is not repeated, different packages start a new one).

Calling them:

vm_login {"vmid": 130, "session": "mate"}
vm_gnome_login {"vmid": 130}
vm_lightdm_login {"vmid": 130}
vm_install {"vmid": 130, "packages": "kcalc"}

To add your own, drop a .yaml/.yml file into the scenarios directory (~/.config/poligon-mcp/scenarios/ for the container setup below) and restart the server/container.

A scenario with tool: false is not registered as an MCP tool but can still be called from any step list as a recipe. Three ship:

Recipe	Params	What it does
update	—	apt-get update && apt-get dist-upgrade -y && update-kernel -y, reboot, wait_boot 240, then asserts the reboot really happened: a marker touched in /run (tmpfs) before the reboot must be gone
update_branch	branch (required)	apt-repo rm all && apt-repo add branch <branch>, a tolerated dist-upgrade pass (the p10→p11 usrmerge migration exits 100), then update
enable_service	service	systemctl enable --now <service> && systemctl is-active <service>

install, gnome_login and lightdm_login stay tools and are callable as recipes.

Stands

Backends may be assigned a stable identity with --backend-id / POLIGON_MCP_BACKEND_ID, or with "backend_id" on an entry in POLIGON_MCP_STANDS. IDs use ASCII letters, digits, ., _ and - and are at most 64 bytes. An explicit ID makes endpoint-renamed aliases share session caches and VM locks; omit it to retain legacy URL-scoped behavior. IDs are namespaced by backend kind and must not be reused across kinds. Only configure aliases for the same physical cluster or device.

A stand is 1-4 VMs cloned from templates, booted in order and set up with step lists — the same steps a scenario uses. stand_plan asks a small model to write one from a goal in plain language; stand_apply builds it in the background.

{"name": "nginxws",
 "machines": [
   {"role": "server",
    "template": "p11-server-11.1-x86-64-legacy-ext4-textmode-server",
    "steps": [{"recipe": {"name": "install", "packages": "nginx"}},
              {"recipe": {"name": "enable_service", "service": "nginx"}}]},
   {"role": "ws",
    "template": "kworkstation-11.4",
    "steps": [{"expect": {"command": "curl -s http://{{ip.server}}/", "contains": "html"}},
              {"recipe": {"name": "lightdm_login"}}]}],
 "notes": "free text from the planner, never executed"}

• name and role match ^[a-z][a-z0-9]*$ — lowercase letters and digits only, no underscores and no dashes, so that the VM name <owner>-repro-<stand>-<role> stays a valid DNS label and one stand's name cannot be a prefix of another's VMs. For the token user@pve!repro the stand above builds user-repro-nginxws-server and user-repro-nginxws-ws; the whole name must fit in 63 characters.
• template is an exact template name or a substring that matches exactly one template (kworkstation-11.4 above); templates whose name contains not-for-clone are hidden from the planner and refused by stand_apply, so a hand-written plan cannot clone one either.
• steps are the scenario steps of the table above, recipe included.
• Placeholders in a stand's steps: {{vmid}}, {{role}}, {{stand}} and {{ip.<role>}} — the first global IPv4 of another machine. A machine may only reference a machine earlier in the list (that is the boot order), including not its own role.

Limits: at most 4 machines per stand, at most 2 stands building per token, no sleep step longer than 600 s (recipes included, after expansion), and an existing <owner>-repro-<stand>-* VM refuses the apply. Everything is validated before a single VM is cloned. pool says which pool to build in — needed only when the token has more than one. Any VM named <owner>-repro-<stand>-* counts as part of stand <stand>, hand-cloned ones included: they block an apply of that name and are purged by its destroy.

Per machine the builder does: full clone into your pool → start → wait for the guest agent (180 s) → read the IP → run the steps (the vmid lock held for all of them) → snapshot clean. The first failing step stops the stand with status: "failed" and leaves the VMs in place so you can look at them; stand_destroy cleans up. While the stand is still building, destroy refuses (stand <name> is still building) — pass force: true to stop the builder first (it bails before the next clone, the stand is marked failed with destroyed while building) and then purge what exists. Finished stands are forgotten after a day. Stand status lives in server memory only: after a restart stand_status no longer knows the stand, while the VMs are still there under their <owner>-repro-<stand>-* names (and stand_destroy still finds them by name).

The four tools:

stand_plan   {"goal": "Стенд из двух машин: сервер p11 с nginx и рабочая станция kworkstation 11.4 в одной сети; на рабочей станции проверить, что сайт сервера открывается curl"}
  -> {"ok": true, "plan": {...}, "valid": true, "errors": [], "attempts": 1,
      "model": "qwen3.8-27b", "notes": "...", "prompt_tokens": 8150, "completion_tokens": 500}

stand_apply  {"plan": {...}}          -> {"ok": true, "stand": "nginxws", "status": "running"}
stand_status {"stand": "nginxws"}    -> {"ok": true, "status": "running|done|failed",
                                              "machines": [{"role", "template", "vmid", "node", "ip", "steps": [...], "done"}], "error": null}
stand_status {}                       -> {"ok": true, "stands": [...]}
stand_destroy {"stand": "nginxws", "confirm": true, "force": false} -> {"ok": true, "destroyed": [209, 216], "failed": []}

A two-VM stand like the one above takes about 4 minutes (most of it the two full clones).

The planner

stand_plan sends one system prompt (the stand format, every step kind, the recipe list with parameters and defaults, the ALT rules below and the live template list — about 8k tokens) plus the goal to the model named by PV_AI_PLAN_MODEL, using PV_AI_PLAN_KEY / PV_AI_PLAN_API (each defaulting to the vision one, PV_AI_KEY / PV_AI_API). The reply is stripped of <think> blocks and code fences, parsed as JSON and validated by exactly the same code stand_apply uses. If the validator complains — or the reply is not JSON at all — that goes back to the model once; a plan that is still invalid is returned anyway as a draft (error result, valid: false, with errors) — fix it yourself and call stand_apply. A goal the model considers impossible comes back as {"ok": false, "error": ...}. Every reply — refusal, prose-only reply and draft alike — carries attempts, model, prompt_tokens and completion_tokens.

Tuning without a rebuild: PV_AI_PLAN_OPTIONS is merged into the request body (model and messages always win), so top_p, temperature, max_tokens, chat_template_kwargs and the like are just env settings. Measured on the nginx goal above with qwen3.8-27b:

| PV_AI_GROUND_API / PV_AI_GROUND_KEY / PV_AI_GROUND_MODEL / PV_AI_GROUND_COORDS | Grounding endpoint for click_label: a model that returns one click point (e.g. UGround, permille coords). Higher click accuracy than the listing model but no element list; the listing (ui_see/see-always) stays on the vision model. Defaults to the vision endpoint when unset | | PV_AI_OPTIONS | Extra JSON fields of the vision request body (default {"temperature":0.1,"repetition_penalty":1.05}); repetition_penalty>1 stops holo2 looping over the same UI elements — measured 59 s/3000 tokens down to ~9 s on a busy screen; must be a JSON object |

PV_AI_PLAN_OPTIONS	Result
{"temperature":0.1,"max_tokens":3000} (default)	valid on the first attempt, ~500 completion tokens, 3-7 s
{"temperature":0,"max_tokens":3000}	same, byte-identical across runs
{"temperature":0.1,"max_tokens":3000,"chat_template_kwargs":{"enable_thinking":true}}	fails: the reasoning eats the 3000-token budget and the reply never reaches the JSON
{"temperature":0.1,"max_tokens":12000,"chat_template_kwargs":{"enable_thinking":true}}	valid, but 3-5k completion tokens and 26-48 s

So: with thinking enabled, raise max_tokens well above the plan size.

Recipe notes (ALT facts the planner is told about)

The system prompt encodes what live runs cost us; keep these in mind when writing steps by hand too:

• ALT's hostname has no -I: read an address with ip -4 -o addr show scope global.
• Guest steps run with umask 0077, so files and directories a service must read need an explicit chmod — every directory on the path, not just the last one (mkdir -p /var/www/html && chmod 0755 /var/www /var/www/html, otherwise nginx answers 403).
• A changed listen address needs systemctl restart, not reload.
• The stock nginx has no enabled site, and its default.conf listens on 127.0.0.1 only — so do not enable that one: drop your own server { listen 80 default_server; root /var/www/html; } into /etc/nginx/sites-enabled.d/<name>.conf and systemctl restart nginx. ALT's nginx has no other config directory.
• p11 templates are already on p11: use update, and update_branch only to change branch.
• A p10 → p11 migration only fits on server/minimal templates — the kworkstation template's 25 GB disk runs out of space.
• Check the branch after a migration with the apt-repo output (it contains p11) or /etc/altlinux-release — never with /etc/os-release (the branding package is not rebuilt for the new branch and keeps reporting the old version) and never with a package version, which says nothing about a branch.
• Packages are installed with apt-get (recipe install); there is no installation from an ISO — a stand is built from existing templates only.

Build and run

cargo build --release
./target/release/poligon-mcp                 # HTTP on 127.0.0.1:8811 (Streamable HTTP at /mcp)
./target/release/poligon-mcp --stdio         # stdio transport
./target/release/poligon-mcp --bind 0.0.0.0:8811 --allowed-host myhost.example.org   # listen on all interfaces; see below
./target/release/poligon-mcp --evidence-dir ~/evidence   # let `path` save PNGs there

Environment

Var	Meaning
PVE_API	PVE API base, e.g. https://pve.example.org:8006/api2/json — required
PVE_TOKEN	user@realm!tokenid=secret — honoured only in --stdio mode; ignored over HTTP
PV_AI_KEY	Bearer key for the vision/planner model; ui_see, click_label and stand_plan need it together with PV_AI_API, everything else works without. The multi-user server takes its vision and grounding models from Settings → Models first; these variables serve a purpose the catalog has no model for
PV_AI_API	Vision/planner model chat/completions endpoint (OpenAI-style), e.g. https://ai.example.org/api/chat/completions
PV_AI_MODEL	Vision model name (default qwen3.8-27b)
PV_AI_COORDS	How the vision model reports coordinates: pixels (default) or permille of the frame — the Qwen3-VL family, Holo2 included, answers in 0–1000 whatever the prompt says
PV_AI_PLAN_API / PV_AI_PLAN_KEY	Planner endpoint and key when they differ from the vision ones (a small UI model such as holo2-4b sees, a bigger one plans); default: PV_AI_API / PV_AI_KEY
PV_AI_PLAN_MODEL	Planner model for stand_plan (default qwen3.8-27b)
PV_AI_PLAN_OPTIONS	Extra JSON fields of the planner request body (default {"temperature":0.1,"max_tokens":3000}); must be a JSON object or the server refuses to start
POLIGON_MCP_SCENARIOS	Directory of scenario YAML files; each becomes a vm_<name> tool (same as --scenarios)
POLIGON_MCP_SEE_ALWAYS	1: every screenshot answer (ui_screenshot, screenshot=true after an action, ui_wait_for) also carries the vision model's x y type label list, as ui_see does — exact click targets for one holo2 call (a second or two, ~200 tokens) per frame; needs PV_AI_API/PV_AI_KEY (same as --see-always)
POLIGON_MCP_EVENTS	Directory for model-failure events: one JSON object per line in model-events.jsonl (same as --events-dir). Unset = nothing is recorded beyond the log
POLIGON_MCP_EVENT_SAMPLES	1: with POLIGON_MCP_EVENTS, also save each failure's frame (PNG) and raw model answer under <dir>/samples/ for annotation and replay (same as --event-samples). Those files hold screen text, so this is opt-in
POLIGON_MCP_EVENTS_MAX_MB	Rotate model-events.jsonl past this many MB, keeping one .1 generation (default 64, same as --events-max-mb)
POLIGON_MCP_DATASET	Directory to capture a training dataset into: every console shot is saved as a full-size PNG with its AT-SPI dump, its OCR words and one index.jsonl line (same as --dataset-dir). Unset = no capture; see Training dataset capture
POLIGON_MCP_DATASET_AGENT	The name captured files start with, <agent>-shot-NNNN.png (same as --dataset-agent); default: the caller's owner
POLIGON_MCP_DATASET_XRANDR	Screen mode set in the guest once per boot while capturing (1280x800 by default, empty = leave the guest alone), so its coordinates match the console frame; same as --dataset-xrandr
POLIGON_MCP_DATASET_MIN_FREE_MB	Capture stops while the dataset volume has less than this many MB free (default 2048, same as --dataset-min-free-mb): a run of thousands of full frames fills a disk, and a full disk takes the whole server down; counted in dataset.counters.no_space, tool answers unaffected
POLIGON_MCP_LOG_PAYLOAD	1: also log the full tool error text — guest stdout/stderr, model answers, server bodies — at debug level (same as --log-payload). Off by default: at info an error line stops where the payload begins
POLIGON_MCP_ALLOWED_HOSTS	Comma-separated Host values (host or host:port) accepted over HTTP in addition to loopback (same as repeated --allowed-host); * accepts any Host; see Reverse proxy note
POLIGON_MCP_EVIDENCE	Directory the screenshot path argument may write into (same as --evidence-dir); unset = path refused. /evidence in the container image
SSL_CERT_FILE / SSL_CERT_DIR	PEM file / directory of PEM files to trust for outbound TLS instead of the system store (the CA that signs PVE_API and PV_AI_API); the image sets SSL_CERT_DIR=/etc/poligon-mcp/certs. No CA at all → the server does not start

.env.example is the template for ~/.config/poligon-mcp/env, the file the quadlet and podman run --env-file read:

mkdir -p ~/.config/poligon-mcp
cp .env.example ~/.config/poligon-mcp/env && chmod 600 ~/.config/poligon-mcp/env

then fill in the real values — never commit them.

Health endpoints

/healthz (liveness) and /readyz (readiness) sit next to /mcp on the same port and answer JSON. They need no PVE token, but they do keep the Host guard /mcp has: loopback, or a name passed with --allowed-host (POLIGON_MCP_ALLOWED_HOSTS); anything else gets 421.

curl -s localhost:8811/healthz
# {"status":"ok","version":"0.1.0","commit":"2f1a899"}
curl -s -w '\n%{http_code}\n' localhost:8811/readyz
# {"status":"ready","version":"0.1.0","commit":"2f1a899","benches":1,"storages":0,
#  "scenarios":7,"vision":true,"planner":true,"grounding":false,"see_always":false,"problems":[]}
# 200
curl -s localhost:8811/metrics
# {"version":"0.1.0","commit":"2f1a899","uptime_s":318,
#  "models":{"vision":{"calls":12,"failed":0,"incomplete":1,"usage_missing":0,"ms_total":9411,
#                      "ms_avg":784,"prompt_tokens":15820,"completion_tokens":4311,"cache_hits":5},
#            "grounding":{...},"planner":{...}},
#  "caches":{"vision_entries":4,"vision_cap":512,"consoles":2}}

/healthz is liveness: 200 whenever the process answers. /readyz adds the local configuration summary and answers 503 with a problems list when this instance cannot serve as configured — POLIGON_MCP_SEE_ALWAYS=1 without PV_AI_API/PV_AI_KEY, or an POLIGON_MCP_EVIDENCE directory that is gone (an unmounted volume). Neither endpoint probes PVE or the model endpoints, so a healthcheck adds no backend load and does not flap when PVE is briefly slow; ready therefore does not mean the PVE or AI endpoints are reachable. The answers carry no URLs, tokens or host paths, so a proxy may log them.

/metrics is the third endpoint on the same guard: counters since startup for each model path the server itself calls — calls, failed (transport error or non-2xx), incomplete (cut off at the token cap or unparseable), usage_missing (the endpoint reported no token counts), ms_total/ms_avg, prompt_tokens/completion_tokens, and cache_hits for the vision path — plus the cache sizes and the uptime. That is the data to decide whether a persistent vision cache, a bigger model or another settle_ms default is worth it. It counts only this server's own calls: a client's LLM traffic is invisible here. Counters reset on restart; nothing per-call and no screen content is published, so the endpoint is safe to scrape.

The binary probes itself, so the container healthcheck needs no curl/wget in the image:

poligon-mcp --healthcheck http://127.0.0.1:8811/readyz   # exit 0 on 2xx, 1 otherwise

It is parsed before the other arguments, so it works without PVE_API. --version prints the crate version, the full git commit and tree the binary was built from (unknown outside a git checkout) and a clean flag; the same values head the startup log. A checkout build always says clean false: only a bundle from ops/staging/build_bundle.py, built from a clean git archive of an exact ref, carries clean true (see docs/staging.md).

Auth model

Over HTTP every request must carry the token as a header, since PVE_TOKEN is not read in HTTP mode:

X-PVE-Token: user@realm!tokenid=secret

On each call the server re-derives the caller's pools from what the token's PVE permissions grant (cached 60s, per PVE endpoint and token — the same token on two clusters keeps two separate caches, as do the VM list, the consoles and the per-VM locks) — a token with VM.Allocate on no pool is refused outright (token has no pool with VM.Allocate), so the server's PV_AI_KEY serves pool holders only, not every PVE user — and reads the VM list from /cluster/resources (cached 10s, invalidated after any mutation). A VM is "yours" — modifiable by vm_start/vm_stop/vm_snapshot/vm_rollback/ vm_destroy/vm_exec/vm_file_write/console tools — only if it sits in one of the token's pools and its name starts with <owner>-repro-. Anything else can be listed but not touched. vm_destroy additionally requires confirm=true; it stops the VM first if running, then purges it and its disks.

VM naming and ownership

<owner> is the user part of the token id, lowercased and stripped to [a-z0-9]: user@pve!repro → user, Ivan.Petrov@ad!tok → ivanpetrov (a token whose user leaves nothing is refused: cannot derive owner from token id). Every VM the server creates carries it — <owner>-repro-<name> for vm_clone (a bare repro- on the requested name is dropped, an already owner-prefixed name is kept) and <owner>-repro-<stand>-<role> for stands — so several users, each with their own token, can share one pool and still tell their machines apart in the PVE UI, and none can start, stop, exec into or destroy another's. The owner is only a guard against an agent touching the wrong VM, not an authorisation layer: two PVE users whose sanitised user part is equal (a.b/ab, user@pam/user@pve) share one owner, and the hard boundary stays the token's pool ACL. vm_list_my_vms shows only <owner>-repro-* VMs and reports the owner. VMs made by earlier versions (bare repro-*) are no longer considered yours: remove them once by hand in the PVE UI.

Long operations

vm_clone, vm_start, vm_stop, vm_snapshot, vm_rollback, vm_destroy and vm_task_wait accept a wait (or timeout) argument in seconds, capped at 50 on the server: MCP clients drop a tool call at about a minute, and a dropped call loses its answer (a vm_clone with wait=120 once timed out on the client side and its upid with it). If the PVE task hasn't finished by then, the answer ends with task still running, upid=..., call vm_task_wait — poll with vm_task_wait {"upid": "..."}. The clone POST goes out before any waiting, so a call the client cut off has most likely created the VM anyway — it shows up in vm_list_my_vms under its name once the clone lands, which can be a minute later. Look there before cloning again; a second vm_clone with the same name is refused with the existing vmid. A full clone of a template on ceph storage takes roughly a minute, so expect at least one vm_task_wait round-trip after vm_clone.

The same rule holds inside the guest: vm_exec blocks on its command, so a dist-upgrade or anything else that runs for minutes goes through vm_run, which returns within wait seconds and is called again to keep waiting. A guest agent busy with rpm's IO answers guest-exec failed - got timeout for a while; vm_exec and vm_run retry that until their own deadline instead of handing the 500 to the model. A command with non-ASCII bytes (a Cyrillic grep pattern) never comes back from qemu-guest-agent 8.2 through PVE, so such a command travels base64-encoded and is decoded by the guest's own shell; the guest never sees the difference.

/cluster/resources (used by vm_list_my_vms and vm_list_templates) can lag a mutation by up to ~1 minute — a freshly cloned VM may show a placeholder name/status briefly. Ownership checks (vm_start and friends) re-read the VM's own config instead, so those work immediately.

Podman

Step by step, for a fresh host with rootless podman:

1. Build the image from the checkout (the Containerfile compiles the server and runs the license check):

   podman build -t poligon-mcp .
   

2. Configuration directory ~/.config/poligon-mcp/ with three things in it: env, certs/, scenarios/.

   mkdir -p ~/.config/poligon-mcp/{certs,scenarios} ~/evidence
   cp .env.example ~/.config/poligon-mcp/env      # then edit: PVE_API, PV_AI_API, PV_AI_KEY, ...
   cp scenarios/*.yaml ~/.config/poligon-mcp/scenarios/
   

   env is the container's environment (see Environment); PVE_API is required. The shipped scenarios and recipes are not baked into the image; stands need the recipes, so copy all seven files.

3. The CA certificate of the PVE API goes into certs/ as one or more PEM files. The image has no CA store of its own: it trusts only these files (SSL_CERT_DIR is baked in) and refuses to start with the directory empty (outbound TLS: ... No CA certificates were loaded). Where to get it depends on who signed the certificate PVE_API serves:

   • PVE with its default self-signed certificate: the cluster CA is on every node, scp root@pve1:/etc/pve/pve-root-ca.pem ~/.config/poligon-mcp/certs/.
   • PVE behind a FreeIPA-issued certificate (the ALT setup): the IPA CA, /etc/ipa/ca.crt on any enrolled host, or curl -o certs/ipa-ca.crt https://<ipa-server>/ipa/config/ca.crt.
   • Any other CA, when you can reach the node: the chain the server sends, openssl s_client -connect pve1:8006 -showcerts </dev/null — the last certificate printed is the CA (or the intermediate that needs its own root).
   • A public CA (Let's Encrypt): the system bundle, cp /etc/pki/tls/certs/ca-bundle.crt ~/.config/poligon-mcp/certs/.

   The same directory must cover PV_AI_API and PV_AI_PLAN_API when they are HTTPS. Check before starting the container:

   openssl verify -CApath ~/.config/poligon-mcp/certs \
     <(openssl s_client -connect pve1:8006 </dev/null 2>/dev/null | openssl x509)
   

   c_rehash ~/.config/poligon-mcp/certs is not needed: the server loads every file in the directory.

4. Run:

   podman run -d --name poligon-mcp --restart unless-stopped \
     --env-file ~/.config/poligon-mcp/env \
     -p 127.0.0.1:8811:8811 \
     -v ~/.config/poligon-mcp/scenarios:/etc/poligon-mcp/scenarios:ro,z \
     -v ~/.config/poligon-mcp/certs:/etc/poligon-mcp/certs:ro,z \
     --userns=keep-id:uid=1000,gid=1000 -v ~/evidence:/evidence:z \
     localhost/poligon-mcp:latest
   podman logs poligon-mcp        # "poligon-mcp listening on /mcp", or the TLS/CA error
   

   To serve other hosts, publish on 0.0.0.0:8811 and set POLIGON_MCP_ALLOWED_HOSTS in env to the name clients will use (see Reverse proxy note). After editing env recreate the container (podman rm -f poligon-mcp and podman run again): a running container keeps its old environment. To stop it for an update use podman stop -t 90: the server finishes the calls in flight first.

5. Register the client with its own PVE token (see Client registration) and check: vm_list_templates must answer with the template families.

Notes:

• PVE_TOKEN in env is ignored in HTTP mode — the client sends X-PVE-Token per request. The scenarios mount is optional; a missing or empty directory just means no scenario tools.
• ui_screenshot/ui_see write their path on the server's filesystem, confined to the evidence dir — /evidence in the image, ~/evidence on the host with the mount above; pass path=x.png or path=/evidence/x.png. A path that leaves the dir (../x.png, /tmp/x.png, a symlink pointing out) is refused.
• The image runs as the unprivileged user pve (uid 1000). With rootless podman that uid is remapped to a subuid unless you pass --userns=keep-id:uid=1000,gid=1000 (the quadlet does), which makes the process run as your own host user: ~/evidence needs no chown and the directories under ~/.config stay readable.

podman-compose

compose.yaml is the same deployment without a unit file, everything inside the checkout: ./scenarios (the shipped ones), ./certs, ./evidence, and .env as both the container's environment and the source of ${...} overrides (POLIGON_PORT, POLIGON_SCENARIOS, POLIGON_CERTS, POLIGON_EVIDENCE — see .env.example). podman only: it relies on userns_mode: keep-id.

cp .env.example .env                              # edit: PVE_API, PV_AI_*
mkdir -p certs && cp /etc/ipa/ca.crt certs/
podman-compose up -d --build
podman-compose up -d --force-recreate             # after editing .env

Quadlet (systemd --user)

podman build -t poligon-mcp .
mkdir -p ~/.config/poligon-mcp/certs && cp /etc/ipa/ca.crt ~/.config/poligon-mcp/certs/
mkdir -p ~/.config/containers/systemd
cp poligon-mcp.container ~/.config/containers/systemd/
systemctl --user daemon-reload
systemctl --user start poligon-mcp

Reverse proxy note

The server's Streamable HTTP transport only accepts loopback Host values (localhost, 127.0.0.1, ::1) to block DNS-rebinding attacks against a server that's only meant to be reached locally. To serve on a network interface directly, list the names clients will use in --allowed-host / POLIGON_MCP_ALLOWED_HOSTS (host or host:port, e.g. myhost.example.org,10.0.0.5), or * to accept any Host; the transport is still plain HTTP and the token travels in a header, so keep that to a trusted network. Otherwise a reverse proxy in front of it must rewrite the Host header and terminate TLS itself, e.g. in nginx:

proxy_set_header Host 127.0.0.1:8811;

Client registration

pi ({"mcpServers": ...} config):

{
  "mcpServers": {
    "pve": {
      "type": "url",
      "url": "http://127.0.0.1:8811/mcp",
      "headers": { "X-PVE-Token": "user@realm!tokenid=secret" }
    }
  }
}

Claude Code:

claude mcp add --transport http pve http://127.0.0.1:8811/mcp \
  --header "X-PVE-Token: user@realm!tokenid=secret"

Training dataset capture

With POLIGON_MCP_DATASET=DIR every console frame the server captures is also saved as training material for the vision model — the frame, what the guest's accessibility tree said was on it, and what OCR read from it. ui_screenshot, ui_see, screenshot=true after an action, the click/drag tools' pre-action frame and every scenario screenshot/click/click_label/drag step; ui_read takes a frame for OCR only and does not enter the dataset. Off by default; nothing about the tools' answers changes when it is on. DATASET-RUNBOOK.md is the operating procedure for a capture run (the app/locale/theme matrix, the calls, what to watch, how to export), and the vm_dataset_app scenario launches an application in the session and takes its first shots.

One shot writes, under DIR/<backend>-<bench>-vm<vmid>-<owner>-<YYYY-MM-DD>/ (one directory per cluster, bench, machine, caller and day — <backend> is the first 8 hex digits of the endpoint digest, so the same owner and vmid on two clusters cannot share a directory):

File	Contents
<agent>-shot-NNNN.png	The full console frame, never scaled and never a JPEG — even when the client asked for scale=0.5 or a region crop
<agent>-shot-NNNN.atspi.json	The guest's AT-SPI tree: role, name, x, y, w, h, states per visible widget, plus screen, locale, desktop, session_type, focused
<agent>-shot-NNNN.ocr.tsv	Tesseract rus+eng words as x y w h text conf in frame pixels — every word, no confidence cut

and appends one line to DIR/index.jsonl:

{"shot":"3f2a91c4-default-vm130-user-2026-09-19/user-shot-0007.png",
 "atspi":"3f2a91c4-default-vm130-user-2026-09-19/user-shot-0007.atspi.json",
 "ocr":"3f2a91c4-default-vm130-user-2026-09-19/user-shot-0007.ocr.tsv",
 "click":[620,412],"click_source":"ocr","label":"Далее",
 "intent":"open the next installer page","drag":null,
 "app":"anaconda","locale":"ru","locale_raw":"ru_RU.UTF-8","theme":"light",
 "session":"3f2a91c4-default-vm130-user-2026-09-19","bench":"default","origin":"ui_click_label",
 "ts":"1789800000","req":"6aae89-3","vmid":130,"node":"pve1","agent":"user",
 "frame_w":1280,"frame_h":800,"guest_w":1920,"guest_h":1080,
 "rescaled":{"from":[1920,1080],"to":[1280,800]},"screen_changed":null,
 "atspi_widgets":214,"atspi_error":null,"ocr_error":null}

Which shots are captured: ui_screenshot, ui_see, ui_wait_for with screenshot=true, screenshot=true after an action (after_ui_click, after_ui_type, …), and — the actual training pairs — the frame a click was aimed at, captured before the screen changes (ui_click, ui_click_label, ui_drag).

• intent is required while capturing. ui_click, ui_click_label, ui_drag and ui_type refuse to act without it (intent="press Далее", intent="open the Server Logs tab"): a click with no stated goal is not a training example, and thousands of them are worse than an early error. ui_screenshot, ui_see and ui_wait_for take it too and record it, but do not require it — a plain look at a screen has no goal to name. user (default test) says whose session the dump is read from; app, locale and theme override the guest probes when the caller knows better.
• What is not saved: the vision model's own x y type label answer. Ground truth comes from AT-SPI and OCR, never from the model being trained.
• X11 and Wayland. An X11 dump is in framebuffer pixels (coord_space: physical). A Wayland session reports logical coordinates: the shot is kept with coord_space: logical, and the exporter sizes the logical screen from the shell's root widget and scales the dump onto the frame (a HiDPI guest's row gets rescaled: {from: [640, 400], to: [1280, 800]}, the note the server writes when it rescales; a dump that cannot be placed is left out). A guest with no desktop session (nobody logged in, no python3) is dropped whole and counted as no_session; the machine is remembered for 120 s, then asked again, so a login brings it back without a server restart.
• The screen mode is reported, not assumed. POLIGON_MCP_DATASET_XRANDR tries xrandr -s <mode> once per boot and, if the mode is not in the list, defines it from cvt and adds it to the connected output. The outcome lands in the index as mode: ok, ok-added, failed or no-xrandr — a frame that stayed 1024x768 says so instead of looking fine. screen_type records what the session was.
• AT-SPI needs python3 with pyatspi in the guest and a running desktop session; the script (guest/atspi-dump.py, embedded in the binary) is run as the session user with the session's own DISPLAY/DBUS_SESSION_BUS_ADDRESS/ XDG_RUNTIME_DIR. A missing binding is installed once per boot (python3-module-pyatspi on ALT, plus at-spi2-core). An installer, a boot menu, or a physical stand leaves atspi: null with the reason in atspi_error — which names the user and how many of their processes the guest saw when no session was found. The frame and its OCR are still captured, and a dump that only reports an error is not written as a sidecar.
• Scenario shots carry labels too. A scenario's own app, locale, theme and intent params are written to the index for every shot and click it captures (vm_dataset_app has all four); without them the guest probes decide.
• Coordinates land on the frame. POLIGON_MCP_DATASET_XRANDR (default 1280x800, empty = leave the guest alone) sets the guest's mode once per boot, so the two usually agree; when they do not, the widget boxes are rescaled onto the frame and both sizes stay in the index (frame_w/frame_h, guest_w/guest_h, rescaled). The frame itself is never scaled.
• The frame and its dump are taken at one moment: the screen is settled first, the caller keeps the VM lock across both, and the dump is read before the (slower) OCR. The screen is then read once more and compared with the captured frame; if it moved in between, the shot is flagged screen_changed: true — the pair is kept but the exporter drops it, because a frame and a widget tree from different moments would label pixels that are not there.
• Only widgets that are actually showing are dumped: AT-SPI VISIBLE without SHOWING is a child of a hidden tab or a closed menu, which is not on the frame. The walk is bounded (6000 widgets, 20000 visited nodes, a 20 s deadline, timeout in the guest so the process really dies) and reports truncated when any limit cut it short.
• The dump script runs from the session user's own runtime directory (mode 0700, owned by them), not from a predictable path in world-writable /tmp.
• locale is normalized to its language (ru_RU.UTF-8, ru_RU and ru all count as ru), with what was actually seen kept in locale_raw — a guest probe reports the full name while a caller passes a short one, and one language must not turn into two rows of every count. The exporter normalizes too, so captures written before this are grouped the same way.
• hard_screen_share is counted over the shots that have a dump (hard_screen_base): without a widget tree there is nothing to tell, so installer frames would otherwise drag the quota down.
• /metrics reports dataset.counters: shots, atspi/atspi_missing, ocr/ocr_missing, clicks, clicks_without_intent, hard_screens, screen_changed, no_session, rescaled, failed. They count this process, so a restart zeroes them while the index keeps the run's totals. Capture is best effort — a tool answer never depends on it.
• dataset_status is the same for an operator or an agent mid-run: the counters plus what the index says — sessions, AT-SPI/OCR coverage, clicks with and without an intent, the distributions by locale/theme/app/origin, why dumps are missing, and the panel/tree/toolbar share against the 30% target. The coverage numbers come from the index, so they survive a restart; counters are per-process and start at zero after one. Rows written before the index carried a hard flag are told from their sidecar (bounded per call), and what cannot be told is reported as hard_unknown with hard_screen_lower_bound: true rather than counted as "not hard". Watch clicks_without_intent, atspi_missing and that share during a run, not after it.
• Capture stops while the dataset volume is below POLIGON_MCP_DATASET_MIN_FREE_MB (2 GB by default), counted as no_space: thousands of full frames fill a disk, and a full disk takes more than the dataset with it. The tool answers are never affected.
• Keep the dataset directory away from holo2-eval/: that corpus is the frozen benchmark, and a frame that was used for training cannot also measure the model.

Exporting a training set

dataset-export.py turns a capture into holo2-train/ (or a tar) and enforces the rules that keep the set usable:

./dataset-export.py --dataset ~/evidence/capture --out holo2-train --require-atspi
./dataset-export.py --dataset ~/evidence/capture --out holo2-train --tar holo2-train.tar.gz --clicks-only
./dataset-export.py --self-test    # the rules, checked on a synthetic capture

• The benchmark stays out. Shots whose agent/session appears in holo2-eval/, and frames that show an eval frame's screen, are dropped and counted under skipped in report.json. With Pillow installed (python3-module-pillow) "the same screen" is perceptual — a 16x16 average hash within 8 bits and a 64x40 thumbnail within 2 grey levels — so an eval frame saved at scale 0.5 still matches its full-size twin; without Pillow only byte-identical files match, and report.json says which check ran (eval_check: pixels, bytes or none). A missing benchmark directory is an error; --no-eval exports without the check and says so.
• --require-atspi checks the sidecar, not the index. A dump that only reports an error (no pyatspi), has no widgets, is unreadable, or is missing on disk is skipped with its own reason — an empty label file is not ground truth. So is a row whose guest_w/guest_h differ from frame_w/frame_h without a rescaled note: its boxes point at pixels that are not on the image. A shot flagged screen_changed is skipped too.
• The index is data, not a path list. shot, atspi, ocr and session are resolved inside the capture directory and the export directory; an absolute path, a .. or a symlink that leaves them is refused, so a captured or hand-edited index can neither overwrite files outside --out nor smuggle host files into the tar.
• An export is all-or-nothing. It is written to a temporary directory and swapped in, so a re-run with another seed or a stricter filter cannot leave the previous run's frames in the result or in the tar. Reusing an existing --out is refused unless --force is passed — and --force replaces a previous export only: an --out that is the capture, the benchmark, inside either, or an ancestor of either is refused outright, since swapping it in would delete the source.
• The split is by session, not by shot (--seed, --val-fraction), so the near-identical frames of one login run cannot end up in both halves.
• A click or a drag without intent is skipped. --require-atspi also drops shots with no accessibility tree (installers, boot menus); leave it off to keep them on their OCR alone. --clicks-only keeps clicks and drags.
• Only the PNG, the AT-SPI dump and the OCR TSV are copied — never a model answer.
• report.json states the coverage: exported and skipped counts by a stable code (skipped) with the prose apart (skip_details), both splits, with_atspi, clicks_with_intent, distributions by locale, theme, app and origin, and the share of frames showing panels, trees, toolbars, lists or tables — with a warning below the 30% target, which is holo2-eval's main recall gap.
• Filters: --locale ru, --theme dark, --app workbench, --clicks-only, --tar, --force, and --dry-run for the report without writing anything.

The exported frames are gigabytes; holo2-train/ and dataset/ are gitignored, so ship the tar or the directory outside the repository.

Debug

• Every model call (vision, planner and grounding) leaves one model line at the default info level: what it was for (vision 1280x800 … with the frame size, plan or ground), the model, ms, HTTP status, prompt/completion tokens from the reply's usage, finish reason and the first 600 characters of the answer on one line. usage=false means the endpoint sent no token counts, so the zeros on that line are placeholders, not a free call. podman logs poligon-mcp | grep model= is the history of how the local models answered and how fast; /metrics is the same data as counters.
• Every tool call gets a short correlation id, and the lines it causes carry it: one tool line per call (req=6aae89-1 rpc=7 tool="stand_plan" ms=15, plus error= when it failed), the same req on the call's model lines and on its request line (RUST_LOG=debug). rpc is the client's JSON-RPC id, which rmcp logs too, so the two trails join. seq numbers the model calls inside one tool call, so a retry is distinguishable from the first attempt. A background stand build keeps the id of the stand_apply that started it, so its later model calls stay attributable. grep req=6aae89-1 is one call's whole trail; ids are unique per run (the boot second is part of them) and are neither a secret nor a capability. Lines outside any tool call (startup, the console reaper) say req="-".
• A model call that never got an answer (connection refused, TLS, timeout) is logged too — one model call failed line at warn with ms and the error chain — so a flapping endpoint is visible instead of looking like silence.
• POLIGON_MCP_EVENTS=DIR turns those failures into data instead of log lines: one JSON object per line in DIR/model-events.jsonl, appended and rotated to one .1 generation past POLIGON_MCP_EVENTS_MAX_MB (64 by default). Each event carries ts, id, req (the correlation id of the tool call), kind (vision/grounding/planner), category (transport, http_error, capped, unparseable, no_point, refused), model, and the numbers that describe the failure — status, ms, tokens, frame size, error_kind (connect/timeout/request/…), and lengths instead of text: body_chars, answer_chars, label_chars, reason_chars. That is all: no prompt, no answer, no error body, no endpoint URL, no pixels, so the file holds no screen content and is safe to collect. POLIGON_MCP_EVENT_SAMPLES=1 is what adds the raw text — error_head, label, reason, the transport error chain — and writes the frame as samples/<id>.png and the raw answer as samples/<id>.json, pointing the event at them. That is the material for annotation, replay against another model, and later fine-tuning, and it does hold screen text (an HTTP error body can also echo the request back, key included), so it is opt-in and wants a directory with the same access rules as the evidence dir. /metrics counts what was recorded (models.events). Recording is best effort: a call never fails because its failure could not be written.
• RUST_LOG=debug — verbose tracing on stderr (never logs the token value, only its user@realm!tokenid prefix); model answers in full.
• A tool line's error= stops where the payload begins: guest stdout/stderr, a model answer or a server's HTML body can carry a password or a key, and truncating to N characters does not help when the secret is at the front. The safe part (which step failed, which VM, which PVE status) stays, error_chars says how much was cut, and POLIGON_MCP_LOG_PAYLOAD=1 adds the full text at debug for one debugging session.
• A vision frame the model could not finish (it looped until the token budget ran out) is saved under <evidence-dir>/vision-capped/ with the partial answer beside it, so the screens that trip holo2 can be collected and replayed against other models. Needs --evidence-dir.
• POLIGON_MCP_DUMP=<file> — capture the raw RFB bytes read from the VNC websocket to <file> (server→client only); see tests/fixtures/README.md for how to re-capture the console test fixture with it.

Security notes

• Outbound HTTPS (PVE API, VNC websocket, AI endpoint) verifies the server certificate against the system trust store — or, when SSL_CERT_FILE / SSL_CERT_DIR is set, against those PEM files only. Endpoints signed by an internal CA (IPA, say) need that CA: an enrolled host already trusts it; the container gets it through the certs/ mount (see Podman). A wrong or missing CA fails every call with invalid peer certificate: UnknownIssuer, never with silent trust.
• The container image runs as uid 1000 (user pve), not root; the quadlet maps it to your host user with UserNS=keep-id, so mounted evidence directories are written as you.
• The path argument of ui_screenshot/ui_see and the screenshot step is confined to the evidence dir (--evidence-dir / POLIGON_MCP_EVIDENCE, /evidence in the image): resolved against it, .. and symlinks — the file or any parent — may not lead outside, and a server started without one refuses every path. It is an evidence trail, not a way to write files on the host.

What is (and is not) isolation today

Read this before pointing several people at one server.

• The real boundary is the PVE token's own ACL. A call runs with the caller's X-PVE-Token; the pools it may touch are the ones that token holds VM.Allocate on (re-read every 60 s), and a VM is modifiable only if it is in one of those pools and named <owner>-repro-*. Give each person their own PVE user, token and pool — that, not anything this server does, is what keeps them apart.
• Per-VM locks belong to the machine, not to the caller. Two tokens that both reach VM 100 on one cluster queue behind one lock, so a scenario run holds off a console click on the same VM (vmid N is busy after 30 s). Consoles and the per-caller caches stay separate: a connection carries the credentials of whoever opened it.
• The <owner>-repro- prefix is a seatbelt, not an identity. The owner is the sanitized user part of the token id, so a.b/ab and user@pam/user@pve collapse into one owner (see VM naming and ownership). It stops an agent touching the wrong VM; it does not authenticate anybody.
• X-Bench-Key authorizes a bench, not a person. Everyone holding the same key (a per-stand key, or POLIGON_MCP_COMMON_KEY) is the same principal as far as the server is concerned: they share bench_status visibility, and one of them can renew or release another's rental lease. Use per-stand keys, and do not treat a shared key as multi-tenant access.
• Physical benches (PiKVM/MegaRAC) carry their own credentials in the server configuration. Whoever passes the bench key drives that machine's console and power; there is no per-user attribution on the device side.
• The model endpoints are shared and unmetered per user. PV_AI_KEY serves every caller that holds a pool, /metrics counts the whole process, and there are no per-user token budgets, rate limits or billing. The only limits today are global: 16 concurrent console slots, and 2 stands building per session (endpoint + token).
• The evidence dir is one shared namespace. path is caller-chosen inside it, so anyone who can call the tools may overwrite an existing file there (escaping the directory is refused); screenshots of every user's screens land in the same tree, readable by whoever can read it on the host.
• Logs identify a call, not a person. A tool/model line carries req, the client's JSON-RPC id and — at RUST_LOG=debug — the token's user@realm!tokenid prefix, never its secret. Real attribution needs the identity work that is not implemented yet: per-user accounts, roles and quotas. Until then, one token per person is the only thing that ties a call to a name.

License

AGPL-3.0-or-later. See LICENSE; the authors are listed in AUTHORS.

Dependencies are permissive (MIT, Apache-2.0, ISC, BSD-3-Clause, Unicode-3.0, ...); about.toml is the allowlist and cargo about generate --locked --fail about.hbs -o THIRD-PARTY.md (cargo-about is packaged in ALT) both checks every crate against it and writes the third-party notices. The container build runs it, so the image carries LICENSE, AUTHORS and THIRD-PARTY.md under /usr/share/licenses/poligon-mcp/.