Map interactive elements for agents.
Adds toAgentMap(): an image with numbered badges on interactive elements and a JSON map of their names, roles, bounds and state. This format is often called Set-of-Mark.
npm install @zumer/snapdom@latest @zumer/snapdom-plugins@latestUsage
import { snapdom } from '@zumer/snapdom';
import { agentMap } from '@zumer/snapdom-plugins/agent-map';
const el = document.querySelector('#target');
const result = await snapdom(el, { plugins: [agentMap()] });
const { image, map, dimensions } = await result.toAgentMap();
// image: data URL of the screenshot with numbered red badges overlaid
// map: [{ i, n, r, b, s? }, …]: index, name, role, bbox, state
// Agent says "click element 2" → map[2].b gives [x, y, w, h]Map entry shape (default fields: 'minimal'):
| Key | Type | Description |
|---|---|---|
i | number | Index matching the badge drawn on the image |
n | string | Accessible name: aria-label, then labelledby, then alt, then title, then labels, then textContent. The textContent fallback is capped at 60 characters |
r | string | Explicit role or an implicit role such as button, link, checkbox, radio, textbox, combobox, slider or heading |
b | [x, y, w, h] | Bounding box in output pixels, scaled with the image and maxImageWidth |
s | object? | Meaningful state: checked, disabled, focus, expanded, pressed, selected, value, hasValue, open, selectedText, covered |
isSemanticOnly | true? | Structural entry added by semantic: true; no badge is drawn for it |
Example map for a checkout form:
[
{ i: 0, n: 'Email', r: 'textbox', b: [28, 80, 280, 34] },
{ i: 1, n: 'Send product updates', r: 'checkbox', b: [28, 134, 13, 13], s: { checked: true } },
{ i: 2, n: 'Apply coupon', r: 'button', b: [28, 176, 114, 38], s: { expanded: false } },
{ i: 3, n: 'Remove coupon', r: 'button', b: [150, 176, 140, 38], s: { disabled: true } },
{ i: 4, n: 'Pay $53.90', r: 'button', b: [28, 220, 97, 38] }
]Options
| Option | Type | Default | Description |
|---|---|---|---|
image | 'annotated' | 'raw' | false | 'annotated' | Annotated image, raw image or map only. false skips this export's image generation; see needs to skip capture rendering too |
fields | 'minimal' | 'full' | 'minimal' | 'full' adds t (raw text content) and a (meaningful attributes) per entry. |
semantic | boolean | false | Include structural elements such as headings, paragraphs and landmarks |
maxImageWidth | number | 1024 | Downscale target for the image; bboxes rescale to match. |
imageFormat | 'png' | 'jpg' | 'webp' | 'png' | Image format (only used when image is rendered). |
imageQuality | number | 0.8 | Quality for lossy formats. |
interactiveSelector | string | see below | CSS selector for interactive elements. |
semanticSelector | string | see below | CSS selector for semantic elements (used when semantic: true). |
labelStyle | object | {} | Override badge styles. |
needs | 'clone' | 'render' | 'render' | Use 'clone' with image: false to skip rendering; only valid on a per-capture plugin |
Defaults:
- Interactive elements: links with
href, buttons, inputs, selects, textareas, summaries,[contenteditable="true"], and elements withtabindexother than-1. Also matches rolesbutton,link,tab,menuitem,checkbox,radio,switch,slider,combobox,textbox. - Structural elements:
h1–h6,nav,main,article,section,header,footer,figcaption,blockquote,legend,p.
Per-call overrides: image, imageFormat, imageQuality, maxImageWidth, plus image sizing (width, height, scale, dpr) and backgroundColor. Selectors, fields, semantics and badge styles are frozen during capture. For a map without rendering:
const result = await snapdom(el, { plugins: [agentMap({ image: false, needs: 'clone' })] });
const { map, dimensions } = await result.toAgentMap();Such a result has no image; result.url and image exports throw. The map includes open shadow roots and slotted content and respects core's exclusion policy. Sensitive input values are omitted; other input/textarea values become up to 12 bullets. This does not redact the image.
Badges are applied only to toAgentMap({ image: 'annotated' }); raw images and other exports keep the captured page. Bounding boxes are not exact with perspective, rotated ancestors, or a root rotation stripped by outerTransforms: false.
When to use
Use it for visual agents that need element coordinates, visual QA, or image/map datasets. It runs inside the web app, including browser extensions and Electron pages, without requiring an external browser automation process.