Skip to content

Jev: small decisions inside a larger task ​

Before enabling a backend, check computer-use credentials and requirements.

In AgentX, Jev is an optional backend for typed decisions: small questions with a fixed set of possible answers ("which of these controls?", "yes or no?") that a fast model answers in a fraction of a second. The integration gives it structured state and explicit questions, then receives answers used by the calling code. The agent's main model still handles open-ended reasoning and writing.

Start with one example: point at a control ​

When you run agentx point "the search field":

  1. The native helper reads the active application's accessibility tree (the list of buttons, fields and labels that macOS exposes for screen readers).
  2. If the tree is too sparse, local OCR (reading text from the screen image) supplies readable text and its position.
  3. AgentX creates a bounded list of candidate controls. When a busy screen has more controls than fit, the ones whose names share words with your request are kept first. Pictures, such as icons and small charts, are included.
  4. The ui-element decision seat asks whether the requested control is present, and which candidate matches it.
  5. The caller checks the answer and maps the chosen ID to coordinates before drawing a highlight.

In a web browser, the list holds the web page and, in a few reserved places, the browser's own controls (its toolbar, tabs and window buttons, called the browser chrome). Each browser control is shown to the model as "outside the page: Reload", so "the search box" still finds the page's search box and not the address bar, while "the reload button" or "close the tab" finds the browser's control. Window buttons with no label are named by what they do, for example "close window (red traffic light)".

With --json, the chosen control shows "inBrowser": true when it sits outside the page.

Controls that close something ​

Some controls close a tab or a window, or quit the app: the red window button, a tab's "x", menu items such as "Close Window" or "Quit", and an app's own close buttons such as "Close Editor" or "Close (⌘W)". AgentX marks these, so a request that does not ask to close anything never lands on one by mistake. A plain "Close" inside a web page is not marked, because it usually shuts a pop-up on the page, not the page itself.

A request asks to close something when it says "close", "closes", "closing" or "quit". "Exit full screen", "close-up" and "close to" do not count.

What happens when a marked control is chosen and the request does not ask to close anything:

CommandWhat it does
agentx pointDoes not point at it. It stops with exit code 4. With --json, the answer has a blocked field, and chosen gives the control's name but no x, y, width or height, so a script that clicks wherever chosen says has nothing to click.
agentx teachStops before a click step on it. A step that only shows a control may still highlight it, because nothing is pressed.
Live lessonsShows the control instead of clicking it or typing into it. It also refuses keyboard shortcuts that close a tab or window (⌘W, ⌘⇧W, ⌘⌥W) and stops the lesson. ⌘Q is always refused, because it leaves the app.

When the app shows only one control, agentx point uses it without asking the model. With --json, the answer has the same fields as usual, but present is null (nobody judged whether it matches) and confidence is 1. A script that reads blocked or the exit code first, then checks present, handles both cases. If that one control has no place on screen (for example, it is scrolled out of view), agentx point prints a short message and stops with exit code 2, the same code it uses when nothing matches, with or without --json.

A candidate-selection question always has a choice to return. The separate “is it present?” question lets AgentX refuse to highlight an unrelated control.

What is a decision seat? ​

A seat is a named decision point in the application. Each has its own inputs, questions, and caller behavior.

SeatQuestion it helps answer
ui-elementWhich visible control matches this request?
screen-stateDoes the gathered evidence settle the claim, and does the claim hold?
wiki-rerankWhich candidate articles are relevant?
voice-narrationWhich predefined spoken phrase fits this event?
presence-modeOn this voice turn, should the agent talk, act, teach, watch or stay quiet on screen; stay on screen after; and what first?
guard-riskDoes an action need additional scrutiny?
request-gateWould this new request benefit from Jev preprocessing?
request-contextWhich optional pieces of context should this request receive?
session-continuityDoes this turn need the earlier conversation, or can the session start fresh early?
intent-pathWhich intent-graph category, then which verb within it, does this message belong to? Replaces the model call the classifier makes on a cache miss.
task-tierDoes this task need the strongest model, or would a cheaper one do? Steers model routing when active and decisions.routing names a cheap model.
task-edgeIs this task harder than the agent's own model does reliably, and would sub-agents help? When active, a sure "yes" runs that one task on the model in decisions.routing.edgeModels and can ask the agent to use sub-agents. See Edge model and sub-agents.
task-watchDoes a task that is starting carry a deadline, a promise to the owner, or a hand-off, so something should watch it? Asked at every task start while watchers.enabled is on. Only an active seat's confident "no" skips the watcher; see Watch tasks that must finish.
wake-gateDoes this event need an agent run at all? Not yet asked on the live path; replayed on recorded runs by the backtest.
tool-setWhich set of built-in tools does a new Claude Code session need: read-only, change code, or everything? Shadow only: it records, the session still gets every tool. See Tool set.

More seats exist in src/decisions/seats/; the table lists the ones this page refers to.

The decision API uses typed questions, including categorical choices and yes/no probabilities. Probability estimates are not proof of correctness. Calibration and evaluation require labeled outcomes for the actual seat.

Enable observation before changing behavior ​

Decisions are disabled by default. Each seat supports off, shadow, and active. Shadow mode records decisions while keeping the existing behavior authoritative.

Turn a seat on in three stages: watch it, check it, then let it act.

  1. Put the backend's key in the .env file next to agentx.json. The jev adapter reads OPENROUTER_API_KEY (it uses OpenRouter's decisions endpoint); the typesafe adapter reads TYPESAFE_API_KEY for the direct endpoint. These are the backend defaults in this code, not a promise that the external service is available. The local and simple-jev adapters are separate alternatives.
  2. Add the seat in shadow mode. Merge this into your agentx.json; it isn't a complete file:
    json
    {
      "decisions": {
        "enabled": true,
        "defaultBackend": "jev",
        "seats": {
          "ui-element": { "mode": "shadow" }
        }
      }
    }
  3. Terminal: restart the daemon so it reads the change:
    sh
    agentx daemon restart
  4. Use the feature for a while (for ui-element, run agentx point a few times).
  5. Terminal: check that the backend answers:
    sh
    agentx decisions backends
  6. Terminal: read the recorded calls, failures and latency for the last day:
    sh
    agentx decisions stats --since 1d
  7. When the answers look right, change "mode": "shadow" to "mode": "active".
  8. Terminal: restart the daemon again with agentx daemon restart.

For deeper inspection and calibration, see:

sh
agentx decisions calls --help
agentx decisions calibrate --help
agentx decisions recalibrate --help

How computer-use verification differs ​

look sends captured pixels to a vision model for an observation. Jev receives the structured observation and other evidence through screen-state; it does not directly inspect those pixels in this integration.

Verification distinguishes confirmed, refuted, and unknown. A tiny recording indicator may not be legible enough to settle a claim. System probes or checking the resulting media file can provide stronger evidence.

askSeat returns no decision if a backend fails or a seat is unavailable, leaving the caller to choose a fallback. The screen verifier uses the vision model's reading as an explicitly uncalibrated fallback; an unclear reading remains unknown. Do not assume every caller handles failures identically.

See the recording case and the command reference.

Request intake and context selection ​

mermaid
flowchart TD
  A[New request] --> B[request-gate: is typed preprocessing useful?]
  B -->|Yes, active| C[Structured optional context catalogue]
  C --> D[request-context: retain relevant blocks]
  B -->|No| E[Configured agent and existing context]
  B -->|Off, shadow, unavailable| F[Existing dispatch behavior]
  D --> G[Mandatory context plus selected optional blocks]
  G --> H[Main agent executes request]
  E --> H
  F --> H

The gate is itself a bounded typed decision. It does not answer arbitrary requests or execute tools. It receives a clipped request, the source channel, agent identity, and the available preprocessing operations. An active negative answer bypasses optional context planning and model routing.

Optional context is a catalogue of stable IDs with source, description, size, bounded preview, and a data-trust label. The context seat can omit clearly irrelevant patterns, references, memory, other-chat summaries, older recall, or wiki blocks. It decides the landscape (the block that tells an agent about the rest of the node) one section at a time: the directory of other agents and channels, how to message another channel, how to look up earlier turns, background monitoring, and agent teams. Uncertain, missing, shadow, and failed selection results retain context. The original request, identity, permissions, runbooks, skills, attachments, same-chat continuity, handover, and the landscape's group-chat rules (for example, stay silent when another agent was mentioned) remain outside its removal allowlist.

The seat waits at most three seconds per turn. The local backend (Claude Code with Haiku) takes about 13 seconds per decision, so with it the seat always times out and keeps everything; use a faster backend such as jev. Measured on 16 labelled messages, the seat removed about half of the landscape with no needed section dropped: see bench/results/jev-and-edit-followups.md in the source code.

This selects AgentX-assembled context before rendering the prompt. It does not erase native CLI session history, filter files/tools the main agent later reads, or avoid retrieval already performed upstream. Context previews are sent to the configured decision backend; this is not a data-isolation boundary.

Enable both seats using the backend you have configured (this example uses the existing direct typesafe adapter):

json
{
  "decisions": {
    "enabled": true,
    "seats": {
      "request-gate": { "mode": "active", "backend": "typesafe", "timeoutMs": 3000 },
      "request-context": { "mode": "active", "backend": "typesafe", "timeoutMs": 3000 }
    }
  }
}

An active gate replaces the legacy Haiku context-planner call. When the gate is off/shadow/unavailable, existing non-desktop dispatch remains authoritative. Each of these decisions waits three seconds by default; set timeoutMs on the seat to change it, up to the 5-second limit a message waits for either one. The gate runs while the session is loaded, and the context selection runs at the same time as the model-tier decision, so their waits overlap rather than add up.

Desktop model guarantee ​

Desktop requests arriving through /ask use the voice channel. Both voice and desktop requests are excluded from automatic cheap-model routing, even for greetings, short confirmations, and cold sessions. They also skip the legacy Haiku context planner. Jev may still perform typed context decisions; the main response and computer-use task stay on the assigned agent's configured model. This preserves that configured model rather than selecting a hard-coded premium model; provider failures do not authorize a downgrade.

Tool set ​

A Claude Code session carries a description of every built-in tool it may use, about 14,000 tokens of its first message, and re-reads them on every later step. A task that only reads and answers does not need the tools that edit files or run commands. The tool-set seat asks, for each new claude-code session, which of three sets the task needs:

SetBuilt-in tools
answerRead, Grep, Glob
codethose, plus Edit, MultiEdit, Write, Bash
fullevery tool, as today

Only shadow mode exists: the seat records what it would choose, and the session still starts with every tool. A message never waits for it: the question is asked alongside the run, not before it.

When the run ends, AgentX records the smallest set that covered the tools the run actually used, so every recorded decision is graded without a person. Runs that end in an error, and runs that report no tool calls (one-shot runs that do not stream), are not graded.

To try it:

  1. Merge this into agentx.json:
    json
    { "decisions": { "enabled": true, "seats": { "tool-set": { "mode": "shadow", "backend": "typesafe" } } } }
  2. Terminal: restart the daemon with agentx daemon stop, then agentx daemon start --detach.
  3. After a few days, Terminal, in a copy of the AgentX source code, from the daemon's folder:
    sh
    pnpm exec tsx /path/to/agentx/scripts/tool-set-report.ts --since 7d
    It prints how many new sessions would have started with fewer tools, and how many of those would have been missing a tool the run used (a "miss"). The daemon log also has one tool-set shadow: line per graded run, ending in MISS for a miss.

A wrong choice in a future active mode would stop an agent in the middle of its task, so the miss rate decides whether this ever goes further than shadow.

Edge model and sub-agents ​

An agent runs every day on one model. A few tasks are harder than that model does reliably: a change to how a system is built, a fault that earlier attempts did not find, a long build with many steps that depend on each other. The task-edge seat finds those tasks, and AgentX runs each of them on a stronger model that you name, called the edge model. Only that one task moves. The next task runs on the agent's own model again, and you do not switch anything back.

The seat answers two questions about each task, in one call:

QuestionWhat a sure "yes" does when the seat is active
Does this task need the most capable model? (0.70 or higher)The task runs on the edge model. For a Claude Code agent, the sub-agents it starts run on the edge model too.
Would separate workers help, because the task has parts that do not depend on each other? (0.70 or higher)For a Claude Code agent, the task gets one extra line that asks it to start sub-agents (helpers that Claude Code runs at the same time, each on one part).

The two answers are separate. A task can move to the edge model, get the sub-agent line, both, or neither.

Everything else leaves the task as it is: the seat is off or in shadow, the decision service does not answer, or the answer is under the number above. A wrong "yes" costs money and uses your subscription's limit, so only a sure answer changes a task.

Some tasks are never moved:

  • A task that already names its model, such as a scheduled job with its own model.
  • A task that the triage rule sent to a cheaper model (A cheaper model for triage events).
  • A request on the voice, desktop or opencode channel, where a person chose the model.
  • A task whose agent already runs on the edge model. It stays on that model; there is nothing higher to move to.

The sub-agent line is left out when the session's tool list (session.lean.tools) does not include the Agent tool. If both this seat and task-tier are sure about the same task, the edge model wins. This also holds for an agent that already runs on the edge model: the task is not sent to the cheaper model.

There are two costs to know. When a task in a running conversation moves to the edge model, that model reads the whole conversation once at the full price, because the saved copy of a conversation belongs to one model. An agent with persistentProcess on is also restarted for the task and again for the next one.

To turn it on:

  1. Merge this into agentx.json. Start with shadow: the seat records what it would do and changes nothing.
    json
    {
      "decisions": {
        "enabled": true,
        "seats": { "task-edge": { "mode": "shadow", "backend": "typesafe" } },
        "routing": { "edgeModels": { "claude-code": "claude-fable-5-2" } }
      }
    }
  2. Terminal: restart the daemon with agentx daemon stop, then agentx daemon start --detach.
  3. After a few days, Terminal: run agentx decisions stats --since 7d and read the task-edge row. The daemon log has no line for it in shadow mode; the answers are in the decisions database.
  4. When the answers look right, change "mode": "shadow" to "mode": "active" and restart the daemon again.
  5. In the daemon log, a moved task prints one line, for example hard task → claude-fable-5-2 for this job, sub-agents advised (edge 0.91, sub-agents 0.84).

To get only the sub-agent line and never a model change, leave edgeModels out.

Messages that describe a problem and ask for nothing ​

A person often writes about a problem without saying what to do: "we cannot see whether an agent is still working on its job". Read as a sentence it is a remark, so the first question above gives it a low number, even when fixing the problem is a large piece of work.

The setting decisions.routing.impliedWork makes the task-edge seat ask two more questions in the same call:

QuestionWhat a sure "yes" does when the setting is active
Does the message describe a problem and ask for nothing, so work is implied? (0.80 or higher)The task gets one extra line: the agent starts its reply with one sentence that states the job as it understands it, then does that job. If the message could mean two different jobs, or the work cannot be undone, the agent asks one question first.
Does fixing the problem need a change to how the system is designed, not one small fix? (0.70 or higher, and work is implied)The task also runs on the edge model.

Your message is never rewritten. The agent states the job in its reply, where you can see it and correct it.

The setting has three values:

ValueWhat it does
off (default)The two questions are not asked.
shadowThe answers are recorded, and the daemon log says what would have happened. Nothing changes.
activeThe line is added and the task can move to the edge model. The task-edge seat must be active too.

Only a message that a person wrote is read this way. A scheduled job, a webhook and a message from another agent are left alone.

To turn it on:

  1. Turn on the task-edge seat with the steps above.
  2. Merge this into agentx.json:
    json
    {
      "decisions": {
        "routing": { "impliedWork": "shadow" }
      }
    }
  3. Terminal: restart the daemon with agentx daemon stop, then agentx daemon start --detach.
  4. For a few days, read the daemon log for lines that contain implied work (shadow), for example the agent's model is enough (edge 0.31, sub-agents 0.20, implied 0.98, design 0.85) — implied work (shadow): would state the job and move up.
  5. When those lines match messages where you did want work, change "shadow" to "active" and restart the daemon again. A task read this way then prints a line that contains implied work: the agent states the job.

The cost to know: more tasks reach the edge model, and a remark you meant only as a comment can start work. Set the value back to off if that happens too often.

Check it worked ​

  1. Terminal: run agentx decisions stats --since 1d.
  2. Your seat appears with its mode (shadow or active), a call count above zero, and few or no failed calls.

If something is wrong ​

  • The seat doesn't appear in decisions stats: check that decisions.enabled is true, that the seat's name is spelled exactly as in the table, that the daemon was restarted, and that the feature that uses the seat actually ran.
  • Every call fails: run agentx decisions backends. A missing or wrong OPENROUTER_API_KEY or TYPESAFE_API_KEY is the usual cause.
  • agentx point picks a control on the page when you meant the browser's: say so in the request, for example agentx point "the browser's reload button". If it still misses, run with --json: the browser's controls in ranked have a shown name starting with "outside the page:", and the chosen one has "inBrowser": true.
  • Behavior changed in a way you didn't want: set the seat back to shadow or off and restart the daemon. In shadow mode the existing behavior is authoritative again.

Released under the MIT License.