Autopilot

Autopilot keeps tuning the routes your team has approved, on your own traffic, inside limits you can see. It never starts routing a step to the cheaper model: that takes a person's approval on the dashboard's Decisions page.

On by default

Autopilot is on for every organization unless someone switches it off. Until your team sets other limits it works within the published default, at most 2% added failures at 95% confidence by default. Its changes are recorded as ergoda:default, because no person accepted that limit.

On the dashboard's Autopilot page you can switch it off or set your own limits; the Organization page shows the same standing authorization and its limits with a Turn off button, and ergoda login and ergoda status print whether it is on. Your own limits are recorded under your name. Switching it off makes every move toward the cheaper model wait for a named person. It does not undo changes already made.

What the limits mean

Added failures are answers the cheaper model gets wrong that your frontier model would have got right. Autopilot makes a change only where your evidence shows the rate of added failures stays under the limit.

Confidence is how sure that evidence must be.

At the default, on a step with 10,000 requests a month:

The evidence is split in two. One half chooses the change and the other half checks it, so the figure a change reports is not the one used to choose it.

What it changes

• The bar. How sure the checks must be before a cheaper answer is used. Lowering it lets the cheaper model answer more of the step.
• The check weights. How much each check counts, refitted on your own traffic.
• Kinds of decision. Where a step makes decisions, such as which tool to call, the cheaper model can take on one more kind at a time, and a share of those decisions is audited against your frontier.
• Actions that can't be undone. Only on a step whose route declares its own limit for them, only from none to one, and only when the evidence clears that limit. See below.

A route gets at most one change per run, and each change needs fresh evidence gathered after the last one: two changes never rest on the same requests.

What takes a change back

• An escalation takes back a lowered bar or new weights, and a failed audit takes back a kind of decision. Neither needs anyone's approval.
• A route autopilot changed that sees no traffic for 90 days goes back to what its owner set.
• So does a route whose cheaper or frontier model changed after the change was made: a new snapshot behind the same name, or a gateway moving a slug to another model. Every event records the model your provider said answered, and the hourly check compares it. Autopilot's next change on that route uses only evidence from after the switch. A provider that does not name the model in its response cannot trigger this.
• A person can send any step back to the frontier on the Decisions page, at any time.

Actions that can't be undone

Payments, sent messages and deleted records can't be taken back once they run. Today your frontier model makes these: when the cheaper model proposes one, Ergoda sets it aside, lets your frontier act, and records whether the two agreed.

The cheaper model earns them step by step:

1. Each step sets a bar: how often the cheaper model must agree with your frontier on these actions, such as 99.8%. A re-read and a payment are different risks, so there is no bar for the whole organization; each step sets its own. Set it on the Autopilot page under Your steps, where it is recorded under your name, or in the step's route as irreversible_alpha (0.002 is a bar of 99.8%). A step with no bar leaves all of these to your frontier.
2. Ergoda counts every proposal the cheaper model makes against what your frontier did, while your frontier keeps acting.
3. Autopilot hands them over, a little at a time, once it is 95% sure the agreement clears the step's bar with a margin: a quarter of what the bar allows is kept unspent, so a bar of 99% needs at least 99.25%.

A track record is only as strong as its size. If the cheaper model agreed 998 times out of 1,000, Ergoda can be 95% sure it agrees at least 99.37% of the time: enough for a bar of 99%, not yet for 99.5%. At 9,990 out of 10,000 it is at least 99.83%, which clears 99.5% but not yet 99.8%. The Autopilot page has a calculator for your own numbers.

Changing or removing a step's bar takes back anything autopilot handed over on that step under the old one; it earns it again against the new bar.

The Autopilot page can also set your organization's minimum: the lowest bar any step may set.

Write validators

Once the cheaper model makes a kind of write, nothing in Ergoda's checks asks whether the person in the conversation may make it: a real order id that belongs to someone else passes them. So autopilot hands over a kind of write only where its tool has a validator, or where an owner has acknowledged the risk. Until then the frontier makes them all.

A validator is a function your process runs before the write commits: it asks your own system whether this session's user may make this call, and returns True, or a reason to refuse. The SDK's helpers build one in a line. owned_by checks that the signed-in user owns the record the call acts on, and http asks your validation endpoint. Your endpoint answers {"ok": true} (or {"valid": true}, or an empty 204) to accept, and {"ok": false, "reason": ...}, an error, or a 4xx such as 422 to refuse. Anything else, including a redirect, a 5xx, a 401/403/404/429, a timeout or a body that says neither, is no verdict, and the write goes to your frontier.

from ergoda import validators
check = validators.all_of(validators.owned_by(lambda a: orders.owner_of(a["order_id"])), validators.http("https://api.internal/orders/cancel/validate", dry_run_param="dry_run"))
client = Client(..., validators={"cancel_order": check})

Set the session's user around each conversation with with validators.acting_for(user_id):. With no user set, owned_by gives no verdict and the write goes to your frontier. from_callable(fn) wraps a validate function you already have: returning nothing accepts, and raising ValueError refuses. In our retail tests, checking that the signed-in user owns the record stopped every write the model made on another customer's account. Then name the tool in the route's validated_tools, or let the Writes page or ergoda writes validate cancel_order --by "Your Name" do it:

{"step_cluster_id": "...", "validated_tools": ["cancel_order"]}

The TypeScript SDK has the same helpers (validators.allOf, ownedBy, http, fromCallable, actingFor) and takes the same functions (see its Write validators section). A refusal sends the step to your frontier and does not count against the tool. A tool marked validated before your process has its validator also goes to your frontier, so the order you do the two in can cost a saving but never skips a check.

Where you have no validator, an owner can acknowledge the risk instead. Acknowledging publishes the route with your acknowledgement under your name, after showing what it means: wrong but valid writes, which nothing undoes, can then ship at up to the rate autopilot licenses on the step. It hands nothing over by itself. Autopilot still needs the evidence to clear the step's bar, and the audits that watch every kind it hands over watch this one. The acknowledgement shows in the policy history. To withdraw it, use Withdraw or ergoda writes withdraw. The kinds it covered go back to the frontier in the same change.

Keeping it current

Your agent changes, so declaring its writes is not a one-time job. The dashboard's Writes page lists every write tool on each step with its state: validated, acknowledged by whom and when, needs a decision, or changed since acknowledged. Each row has one action: Connect a validator (the three lines above, filled in with the tool's name), Acknowledge, Re-confirm or Withdraw. Each row also shows what the tool is worth: its share of the step's bill, and about how much of that would be freed if the cheaper model made it. When a new write appears, or a step's tools change after an acknowledgement, your alert webhook (or the account's email) gets one message with a link to that step's row.

What a change does:

• A validated tool needs nothing. The validator checks every call as it is made.
• An acknowledged tool on a step whose tools changed is flagged for you to re-confirm. It is not withdrawn, because Ergoda sees one fingerprint for the whole tool list and cannot tell which tool changed. Whatever autopilot had handed over on that step goes back to the frontier at the change either way, and is earned again on the new tools.
• A tool that gains a required argument (a new record id, an amount) now acts on something nobody acknowledged it acting on. ergoda writes check sees the schema in your own catalogue and withdraws the acknowledgement for you; acknowledge it again if it still holds.
• A tool with no calls in 30 days is marked as not used. Nothing changes.

Ergoda knows each tool by a fingerprint of its name, not the name. The page names a tool only where your route already does. ergoda writes puts the real names back from your own catalogue on your machine and sends none of them:

# what needs you, by name
ergoda writes --tools tools.json
# acknowledge, or re-confirm
ergoda writes ack cancel_order --by "Ada"
# mark validated, print the code
ergoda writes validate cancel_order --by "Ada"
# in CI: fail on a to-do
ergoda writes check --tools tools.json

Each declaration made with the command records the tool's schema in ergoda.writes.json, which you commit, and check compares your catalogue against it on every build. An acknowledgement with no schema on record (one made on the dashboard, or with no catalogue to read) fails check until you re-confirm it with ergoda writes ack TOOL --by "Your Name" --tools FILE, which records it. One signature covers a step's whole acknowledgement, so re-confirming one tool re-signs every tool it names, and the command lists them. A tool that writes on several steps needs --step. The actions need a full key (`ergoda login --scope full`), so an agent you granted one can keep these current too. Its declarations are recorded as the name it gives, followed by "asserted via" the key.

When it runs

Every day at 04:43 UTC. Each change is listed on the Autopilot page under What it changed as soon as it is made, with the evidence it rests on, and in the policy history over the API.

Where its evidence comes from

Everything autopilot decides on is reported by your own process: the gate's verdicts, the outcomes you record, the frontier's comparisons and the audits. Ergoda serves none of that traffic and never sees its content, so it cannot check that an outcome you reported really happened, and every change says so: its evidence is marked self_reported, with server_observed: 0. The attestation states the same thing.

What Ergoda does check, before a row counts toward a change:

• Each piece of evidence is a request you were billed for, in production, on the same step and in the same conversation as the row that describes it. A verdict or outcome that names a request from another step, or another conversation, does not count.
• One request is at most one piece of evidence, and at most one conversation in the counts the limits are tested on. Proposals from one request with no conversation id count as one.
• An audit that agrees counts only beside the commit it audits. An audit that disagrees always counts, wherever it points.
• Nothing reported later can undo a failure already recorded.

Evidence that fails these checks is not used, and the change reports how much of it there was (provenance.unanchored). On a working integration that is zero, so anything else is worth looking into.

So a key that can send events can still make up evidence, but each fake request costs the normal per-request fee, has to match a billed request, and shows in the change's evidence counts. Keep send-only keys as secret as full keys, and switch autopilot off on any site where you can't vouch for what sends to it.