Pelorus · bearings for AI
Is your AI still on the course you set?
Models are updated without notice. The same model can behave differently depending on how you reach it. Agents drift from the rules they were given. Pelorus takes independent, dated bearings on all three, and publishes them.
The water moves under every agent.
A navigator takes bearings because a ship drifts even when the wheel is held steady. AI is the same. Three things move, and most teams find out when a customer does.
It changes
Providers update and retire models on their own schedule. What you tested last month may not be what answers today.
It depends how you reach it
The same model through a different provider, framework or harness can answer differently, and nobody tells you.
They slip
An agent's written instructions get read, then bent: by context, by long chats, by pressure from the person on the other end.
"It's not your agent. It's Tuesday's model update."What a dated record lets you say
Six instruments, one dated record.
Each answers one question. Each carries its own International Code flag, chosen for what the flag means at sea.
-
Course Check
Does it follow its rules?
Flag C · "Yes"A fixed rulebook, many situations built to strain it: plain, edge, pressure, harmless look-alikes. Graded followed, partly, broke, or over-applied. Course Check 001 and 002 are out.
Live -
Soundings
Is it the same model today?
Flag SThe same sealed pack, run again across dates, providers and harnesses, with repeat runs to measure normal noise. A difference beyond the noise is a finding. So is no difference. Sounding 001 is out.
Live -
Survey
Is my agent healthy?
Flag Q · "My vessel is healthy"A check-up that runs on your own agent, locally and read-only: fingerprints of its skills and tools, a map of what it can reach, what it remembers, and a test of whether it did what it says it did.
In the yard -
Migration Rite
What changed when the model did?
Flag E · "Altering course"When a model is retired or a runtime upgraded, your agent is read before and after: its own account beside a set of probes and the change notes. What the notes explain is closed. What they don't stays open, and is never blamed on the agent.
In the yard -
Handshake
How should this agent work with this person today?
Flag K · "I wish to communicate"A short state reading that a person and an agent swap before work starts. Agents get instructions, never reasons. The person sees and controls everything, and an employer never sees an individual line.
Drawing board -
.md
state.md
What is this agent, and what was measured?
Open standardOne small file any agent can carry: its pinned setup, what it says about itself, what was measured, and the gap between them. The agent can never edit its own measurements. Free and open.
Drawing board
The latest bearings.
Four models, one rule broken the same way.
We gave four Claude models a written safety rulebook and 36 sealed situations. All four missed one rule: a plan, with nothing said about what it was for. We rewrote it twice and re-tested. Now all four check the ambiguous plan and leave the harmless one alone.
Two routes, one model. No difference beyond the noise.
Three Claude models through Anthropic and Amazon Bedrock, three runs each, 648 replies. The route gap was no bigger than the gap between two runs.
Read the sounding→Five more models, the same rulebook. One wrong country.
The v0.4 rulebook on models from five other labs. One critical reply in 180: the right words, with the wrong country's crisis line.
Read the check→Sealed, run, published.
Every bearing makes the same passage. Scroll it through.
Fix the pack first.
The situations are written and sealed before any model sees them. Its fingerprint is taken then, so no one can quietly change the test later.
Run every route, more than once.
Each model goes through each provider, three times over, and is graded blind. The repeat runs set the normal noise, so a real shift stands out.
Date it and put it on the record.
The results go out route by route, with the raw replies kept. A shift beyond the noise is a finding. So is no shift at all.
The rules every bearing keeps.
A bearing is only worth something if anyone can check how it was taken. These rules hold for every instrument.
- Sealed before it runs. A test pack is fixed before the first run and never edited. A change makes a new pack.
- Results and fingerprints, not questions. We publish what happened and a fingerprint of the pack, not the full situations, so no one can tune for them.
- Every result names its route. Model, version, provider, harness and date, in full.
- Graded blind. Graders never see which model wrote a reply. Plain code checks run alongside them. Where graders disagree, the stricter grade stands.
- Everything is kept. Raw replies, code and fingerprints for every run, so any result can be re-checked later.
- The measured never pay. No money from anyone we measure, and no token.
Free to read. Paid to watch.
Every published bearing is free to read. Agents can be watched or brought in for a tune-up, and platforms can get dated attestations.
Read
Every published bearing, the state.md format, a local Survey and a self-check you run yourself.
Open to anyone
Keep watch
Continuous checks on the models and routes your agents run on, an alert when something shifts, and fixes as soon as they're proven.
Monthly
Tune-up
We copy your agent's setup into our test rack, run the full Survey and Course Checks, fix what's off, re-test, and hand it back with a new state.md.
One-off
Attest
Dated attestations for marketplaces, insurers and certifiers who need to know an agent was checked, when, and how.
By arrangement
Watch and dry dock aren't open yet; pricing comes when they do. Fixes are never applied on their own: each is proposed, tested, reversible, and approved by you.
Why "Pelorus".
A pelorus is a sighting ring. It sits on a ship's deck with two vanes, and a navigator lines them up to take a bearing against the ship's own heading. It doesn't steer. It tells you where you are against where you meant to be.
That's the job here: measure where an AI model or agent sits against its own course and its own baseline, date it, and keep the record.
Pelorus is independent and based in Sydney, Australia.
Want a bearing on your agent?
Tell us what it runs on and what it's meant to do. Early users help shape the Survey and the Watch tier.
[email protected]