Jev: small decisions inside a larger task
Before enabling a backend, check computer-use credentials and requirements.
In AgentX, Jev is an optional backend for typed decisions: small questions with a fixed set of possible answers ("which of these controls?", "yes or no?") that a fast model answers in a fraction of a second. The integration gives it structured state and explicit questions, then receives answers used by the calling code. The agent's main model still handles open-ended reasoning and writing.
Start with one example: point at a control
When you run agentx point "the search field":
- The native helper reads the active application's accessibility tree (the list of buttons, fields and labels that macOS exposes for screen readers).
- If the tree is too sparse, local OCR (reading text from the screen image) supplies readable text and its position.
- AgentX creates a bounded list of candidate controls. When a busy screen has more controls than fit, the ones whose names share words with your request are kept first. Pictures, such as icons and small charts, are included.
- The
ui-elementdecision seat asks whether the requested control is present, and which candidate matches it. - The caller checks the answer and maps the chosen ID to coordinates before drawing a highlight.
In a web browser, the list holds the web page and, in a few reserved places, the browser's own controls (its toolbar, tabs and window buttons, called the browser chrome). Each browser control is shown to the model as "outside the page: Reload", so "the search box" still finds the page's search box and not the address bar, while "the reload button" or "close the tab" finds the browser's control. Window buttons with no label are named by what they do, for example "close window (red traffic light)".
With --json, the chosen control shows "inBrowser": true when it sits outside the page.
Controls that close something
Some controls close a tab or a window, or quit the app: the red window button, a tab's "x", menu items such as "Close Window" or "Quit", and an app's own close buttons such as "Close Editor" or "Close (⌘W)". AgentX marks these, so a request that does not ask to close anything never lands on one by mistake. A plain "Close" inside a web page is not marked, because it usually shuts a pop-up on the page, not the page itself.
A request asks to close something when it says "close", "closes", "closing" or "quit". "Exit full screen", "close-up" and "close to" do not count.
What happens when a marked control is chosen and the request does not ask to close anything:
| Command | What it does |
|---|---|
agentx point | Does not point at it. It stops with exit code 4. With --json, the answer has a blocked field, and chosen gives the control's name but no x, y, width or height, so a script that clicks wherever chosen says has nothing to click. |
agentx teach | Stops before a click step on it. A step that only shows a control may still highlight it, because nothing is pressed. |
| Live lessons | Shows the control instead of clicking it or typing into it. It also refuses keyboard shortcuts that close a tab or window (⌘W, ⌘⇧W, ⌘⌥W) and stops the lesson. ⌘Q is always refused, because it leaves the app. |
When the app shows only one control, agentx point uses it without asking the model. With --json, the answer has the same fields as usual, but present is null (nobody judged whether it matches) and confidence is 1. A script that reads blocked or the exit code first, then checks present, handles both cases. If that one control has no place on screen (for example, it is scrolled out of view), agentx point prints a short message and stops with exit code 2, the same code it uses when nothing matches, with or without --json.
A candidate-selection question always has a choice to return. The separate “is it present?” question lets AgentX refuse to highlight an unrelated control.
What is a decision seat?
A seat is a named decision point in the application. Each has its own inputs, questions, and caller behavior.
| Seat | Question it helps answer |
|---|---|
ui-element | Which visible control matches this request? |
screen-state | Does the gathered evidence settle the claim, and does the claim hold? |
wiki-rerank | Which candidate articles are relevant? |
voice-narration | Which predefined spoken phrase fits this event? |
presence-mode | On this voice turn, should the agent talk, act, teach, watch or stay quiet on screen; stay on screen after; and what first? |
guard-risk | Does an action need additional scrutiny? |
request-gate | Would this new request benefit from Jev preprocessing? |
request-context | Which optional pieces of context should this request receive? |
session-continuity | Does this turn need the earlier conversation, or can the session start fresh early? |
intent-path | Which intent-graph category, then which verb within it, does this message belong to? Replaces the model call the classifier makes on a cache miss. |
task-tier | Does this task need the strongest model, or would a cheaper one do? Steers model routing when active and decisions.routing names a cheap model. |
task-edge | Is this task harder than the agent's own model does reliably, and would sub-agents help? When active, a sure "yes" runs that one task on the model in decisions.routing.edgeModels and can ask the agent to use sub-agents. See Edge model and sub-agents. |
task-watch | Does a task that is starting carry a deadline, a promise to the owner, or a hand-off, so something should watch it? Asked at every task start while watchers.enabled is on. Only an active seat's confident "no" skips the watcher; see Watch tasks that must finish. |
wake-gate | Does this event need an agent run at all? Not yet asked on the live path; replayed on recorded runs by the backtest. |
tool-set | Which set of built-in tools does a new Claude Code session need: read-only, change code, or everything? Shadow only: it records, the session still gets every tool. See Tool set. |
More seats exist in src/decisions/seats/; the table lists the ones this page refers to.
The decision API uses typed questions, including categorical choices and yes/no probabilities. Probability estimates are not proof of correctness. Calibration and evaluation require labeled outcomes for the actual seat.
Enable observation before changing behavior
Decisions are disabled by default. Each seat supports off, shadow, and active. Shadow mode records decisions while keeping the existing behavior authoritative.
Turn a seat on in three stages: watch it, check it, then let it act.
- Put the backend's key in the
.envfile next toagentx.json. Thejevadapter readsOPENROUTER_API_KEY(it uses OpenRouter's decisions endpoint); thetypesafeadapter readsTYPESAFE_API_KEYfor the direct endpoint. These are the backend defaults in this code, not a promise that the external service is available. The local andsimple-jevadapters are separate alternatives. - Add the seat in shadow mode. Merge this into your
agentx.json; it isn't a complete file:json{ "decisions": { "enabled": true, "defaultBackend": "jev", "seats": { "ui-element": { "mode": "shadow" } } } } - Terminal: restart the daemon so it reads the change:sh
agentx daemon restart - Use the feature for a while (for
ui-element, runagentx pointa few times). - Terminal: check that the backend answers:sh
agentx decisions backends - Terminal: read the recorded calls, failures and latency for the last day:sh
agentx decisions stats --since 1d - When the answers look right, change
"mode": "shadow"to"mode": "active". - Terminal: restart the daemon again with
agentx daemon restart.
For deeper inspection and calibration, see:
agentx decisions calls --help
agentx decisions calibrate --help
agentx decisions recalibrate --helpHow computer-use verification differs
look sends captured pixels to a vision model for an observation. Jev receives the structured observation and other evidence through screen-state; it does not directly inspect those pixels in this integration.
Verification distinguishes confirmed, refuted, and unknown. A tiny recording indicator may not be legible enough to settle a claim. System probes or checking the resulting media file can provide stronger evidence.
askSeat returns no decision if a backend fails or a seat is unavailable, leaving the caller to choose a fallback. The screen verifier uses the vision model's reading as an explicitly uncalibrated fallback; an unclear reading remains unknown. Do not assume every caller handles failures identically.
See the recording case and the command reference.
Request intake and context selection
flowchart TD
A[New request] --> B[request-gate: is typed preprocessing useful?]
B -->|Yes, active| C[Structured optional context catalogue]
C --> D[request-context: retain relevant blocks]
B -->|No| E[Configured agent and existing context]
B -->|Off, shadow, unavailable| F[Existing dispatch behavior]
D --> G[Mandatory context plus selected optional blocks]
G --> H[Main agent executes request]
E --> H
F --> HThe gate is itself a bounded typed decision. It does not answer arbitrary requests or execute tools. It receives a clipped request, the source channel, agent identity, and the available preprocessing operations. An active negative answer bypasses optional context planning and model routing.
Optional context is a catalogue of stable IDs with source, description, size, bounded preview, and a data-trust label. The context seat can omit clearly irrelevant patterns, references, memory, other-chat summaries, older recall, or wiki blocks. It decides the landscape (the block that tells an agent about the rest of the node) one section at a time: the directory of other agents and channels, how to message another channel, how to look up earlier turns, background monitoring, and agent teams. Uncertain, missing, shadow, and failed selection results retain context. The original request, identity, permissions, runbooks, skills, attachments, same-chat continuity, handover, and the landscape's group-chat rules (for example, stay silent when another agent was mentioned) remain outside its removal allowlist.
The seat waits at most three seconds per turn. The local backend (Claude Code with Haiku) takes about 13 seconds per decision, so with it the seat always times out and keeps everything; use a faster backend such as jev. Measured on 16 labelled messages, the seat removed about half of the landscape with no needed section dropped: see bench/results/jev-and-edit-followups.md in the source code.
This selects AgentX-assembled context before rendering the prompt. It does not erase native CLI session history, filter files/tools the main agent later reads, or avoid retrieval already performed upstream. Context previews are sent to the configured decision backend; this is not a data-isolation boundary.
Enable both seats using the backend you have configured (this example uses the existing direct typesafe adapter):
{
"decisions": {
"enabled": true,
"seats": {
"request-gate": { "mode": "active", "backend": "typesafe", "timeoutMs": 3000 },
"request-context": { "mode": "active", "backend": "typesafe", "timeoutMs": 3000 }
}
}
}An active gate replaces the legacy Haiku context-planner call. When the gate is off/shadow/unavailable, existing non-desktop dispatch remains authoritative. Each of these decisions waits three seconds by default; set timeoutMs on the seat to change it, up to the 5-second limit a message waits for either one. The gate runs while the session is loaded, and the context selection runs at the same time as the model-tier decision, so their waits overlap rather than add up.
Desktop model guarantee
Desktop requests arriving through /ask use the voice channel. Both voice and desktop requests are excluded from automatic cheap-model routing, even for greetings, short confirmations, and cold sessions. They also skip the legacy Haiku context planner. Jev may still perform typed context decisions; the main response and computer-use task stay on the assigned agent's configured model. This preserves that configured model rather than selecting a hard-coded premium model; provider failures do not authorize a downgrade.
Tool set
A Claude Code session carries a description of every built-in tool it may use, about 14,000 tokens of its first message, and re-reads them on every later step. A task that only reads and answers does not need the tools that edit files or run commands. The tool-set seat asks, for each new claude-code session, which of three sets the task needs:
| Set | Built-in tools |
|---|---|
answer | Read, Grep, Glob |
code | those, plus Edit, MultiEdit, Write, Bash |
full | every tool, as today |
Only shadow mode exists: the seat records what it would choose, and the session still starts with every tool. A message never waits for it: the question is asked alongside the run, not before it.
When the run ends, AgentX records the smallest set that covered the tools the run actually used, so every recorded decision is graded without a person. Runs that end in an error, and runs that report no tool calls (one-shot runs that do not stream), are not graded.
To try it:
- Merge this into
agentx.json:json{ "decisions": { "enabled": true, "seats": { "tool-set": { "mode": "shadow", "backend": "typesafe" } } } } - Terminal: restart the daemon with
agentx daemon stop, thenagentx daemon start --detach. - After a few days, Terminal, in a copy of the AgentX source code, from the daemon's folder:shIt prints how many new sessions would have started with fewer tools, and how many of those would have been missing a tool the run used (a "miss"). The daemon log also has one
pnpm exec tsx /path/to/agentx/scripts/tool-set-report.ts --since 7dtool-set shadow:line per graded run, ending inMISSfor a miss.
A wrong choice in a future active mode would stop an agent in the middle of its task, so the miss rate decides whether this ever goes further than shadow.
Edge model and sub-agents
An agent runs every day on one model. A few tasks are harder than that model does reliably: a change to how a system is built, a fault that earlier attempts did not find, a long build with many steps that depend on each other. The task-edge seat finds those tasks, and AgentX runs each of them on a stronger model that you name, called the edge model. Only that one task moves. The next task runs on the agent's own model again, and you do not switch anything back.
The seat answers two questions about each task, in one call:
| Question | What a sure "yes" does when the seat is active |
|---|---|
| Does this task need the most capable model? (0.70 or higher) | The task runs on the edge model. For a Claude Code agent, the sub-agents it starts run on the edge model too. |
| Would separate workers help, because the task has parts that do not depend on each other? (0.70 or higher) | For a Claude Code agent, the task gets one extra line that asks it to start sub-agents (helpers that Claude Code runs at the same time, each on one part). |
The two answers are separate. A task can move to the edge model, get the sub-agent line, both, or neither.
Everything else leaves the task as it is: the seat is off or in shadow, the decision service does not answer, or the answer is under the number above. A wrong "yes" costs money and uses your subscription's limit, so only a sure answer changes a task.
Some tasks are never moved:
- A task that already names its model, such as a scheduled job with its own
model. - A task that the triage rule sent to a cheaper model (A cheaper model for triage events).
- A request on the
voice,desktoporopencodechannel, where a person chose the model. - A task whose agent already runs on the edge model. It stays on that model; there is nothing higher to move to.
The sub-agent line is left out when the session's tool list (session.lean.tools) does not include the Agent tool. If both this seat and task-tier are sure about the same task, the edge model wins. This also holds for an agent that already runs on the edge model: the task is not sent to the cheaper model.
There are two costs to know. When a task in a running conversation moves to the edge model, that model reads the whole conversation once at the full price, because the saved copy of a conversation belongs to one model. An agent with persistentProcess on is also restarted for the task and again for the next one.
To turn it on:
- Merge this into
agentx.json. Start withshadow: the seat records what it would do and changes nothing.json{ "decisions": { "enabled": true, "seats": { "task-edge": { "mode": "shadow", "backend": "typesafe" } }, "routing": { "edgeModels": { "claude-code": "claude-fable-5-2" } } } } - Terminal: restart the daemon with
agentx daemon stop, thenagentx daemon start --detach. - After a few days, Terminal: run
agentx decisions stats --since 7dand read thetask-edgerow. The daemon log has no line for it in shadow mode; the answers are in the decisions database. - When the answers look right, change
"mode": "shadow"to"mode": "active"and restart the daemon again. - In the daemon log, a moved task prints one line, for example
hard task → claude-fable-5-2 for this job, sub-agents advised (edge 0.91, sub-agents 0.84).
To get only the sub-agent line and never a model change, leave edgeModels out.
Messages that describe a problem and ask for nothing
A person often writes about a problem without saying what to do: "we cannot see whether an agent is still working on its job". Read as a sentence it is a remark, so the first question above gives it a low number, even when fixing the problem is a large piece of work.
The setting decisions.routing.impliedWork makes the task-edge seat ask two more questions in the same call:
| Question | What a sure "yes" does when the setting is active |
|---|---|
| Does the message describe a problem and ask for nothing, so work is implied? (0.80 or higher) | The task gets one extra line: the agent starts its reply with one sentence that states the job as it understands it, then does that job. If the message could mean two different jobs, or the work cannot be undone, the agent asks one question first. |
| Does fixing the problem need a change to how the system is designed, not one small fix? (0.70 or higher, and work is implied) | The task also runs on the edge model. |
Your message is never rewritten. The agent states the job in its reply, where you can see it and correct it.
The setting has three values:
| Value | What it does |
|---|---|
off (default) | The two questions are not asked. |
shadow | The answers are recorded, and the daemon log says what would have happened. Nothing changes. |
active | The line is added and the task can move to the edge model. The task-edge seat must be active too. |
Only a message that a person wrote is read this way. A scheduled job, a webhook and a message from another agent are left alone.
To turn it on:
- Turn on the
task-edgeseat with the steps above. - Merge this into
agentx.json:json{ "decisions": { "routing": { "impliedWork": "shadow" } } } - Terminal: restart the daemon with
agentx daemon stop, thenagentx daemon start --detach. - For a few days, read the daemon log for lines that contain
implied work (shadow), for examplethe agent's model is enough (edge 0.31, sub-agents 0.20, implied 0.98, design 0.85) — implied work (shadow): would state the job and move up. - When those lines match messages where you did want work, change
"shadow"to"active"and restart the daemon again. A task read this way then prints a line that containsimplied work: the agent states the job.
The cost to know: more tasks reach the edge model, and a remark you meant only as a comment can start work. Set the value back to off if that happens too often.
Check it worked
- Terminal: run
agentx decisions stats --since 1d. - Your seat appears with its mode (
shadoworactive), a call count above zero, and few or no failed calls.
If something is wrong
- The seat doesn't appear in
decisions stats: check thatdecisions.enabledistrue, that the seat's name is spelled exactly as in the table, that the daemon was restarted, and that the feature that uses the seat actually ran. - Every call fails: run
agentx decisions backends. A missing or wrongOPENROUTER_API_KEYorTYPESAFE_API_KEYis the usual cause. agentx pointpicks a control on the page when you meant the browser's: say so in the request, for exampleagentx point "the browser's reload button". If it still misses, run with--json: the browser's controls inrankedhave ashownname starting with "outside the page:", and the chosen one has"inBrowser": true.- Behavior changed in a way you didn't want: set the seat back to
shadoworoffand restart the daemon. In shadow mode the existing behavior is authoritative again.
