Which tool does an AI model pick?
Measured by Soxoa
We gave three inexpensive AI models a real request and a list of tools, and counted how often each model picked the right one. The tools are the real MCP tools of four products Soxoa runs, listed next to made-up competing tools.
This is the first result on this site that is a real measurement and not an illustration. It is a measurement of Soxoa's own products on one date. It is not a client result, and it does not show that any work Soxoa does for a client improves how often an assistant picks their tools.
What was measured
One thing: selection. For each request the model either called the tool we expected, called another tool of the same product, called one of the competing tools, or called nothing. We also checked that the arguments it sent fit the tool's own input schema. Whether the task then finished correctly was not tested.
Method
- Models, with exact ids (all called through OpenRouter):
openai/gpt-6.1-terra,anthropic/claude-haiku-5.5,google/gemini-3.8-flash. Three inexpensive models, chosen for cost. - Settings: temperature 1, up to 1,024 output tokens, tool choice left on automatic, one turn, and a short neutral system prompt that tells the model to call a tool only when one fits. The tools are shuffled into a fixed order for each request and run, the same order for every model.
- Runs: 5 runs of every request on every model. The baseline was 885 model calls in all: 225 for Postedly, 210 for Portravo, 225 for Intakra and 225 for MentionedOn. No call ended in an API error.
- Requests: 15 for Postedly, 14 for Portravo, 15 for Intakra, 15 for MentionedOn. Most (11 or 12 per product) are requests where one of the product's tools is right. Two or three per product have no right tool, so the correct answer is to call nothing, and three products have one control where a competing tool is the right pick. The wording is in the download. Names, addresses and ids in the requests are made up.
- Competing tools: 9 or 10 synthetic tools per product (a schema generator, a page audit, an address book search and so on), written for the test and not taken from any vendor.
- Scoring: a run counts as correct when the first tool the model called is one of the request's expected tools. Where a request has no business or project id, starting with the product's own listing tool is accepted as well, because that is what its description tells the model to do. We made that correction for MentionedOn after seeing the first scoring, and the raw runs were left unchanged: before it, MentionedOn scored 23%, 18% and 15% on the three models.
- Intervals: every rate carries a 95% Wilson interval, which treats each run as an independent trial. That is too narrow. The 5 runs of one request are not independent, and the requests and competing tools were written by hand, so one request moves a rate by several points. The interval across requests, which resamples whole requests, is the one to trust, and it is wide.
Baseline
Share of runs, on requests where one of the product's own tools is the right answer, in which the model picked an expected tool. Tool definitions were read from each product's code on November 16, 2026.
| Model | Picked an expected tool | 95% Wilson interval | 95% interval across requests |
|---|---|---|---|
| openai/gpt-6.1-terra | 85% (51/60) | 74% to 92% | 63% to 100% |
| anthropic/claude-haiku-5.5 | 82% (49/60) | 70% to 89% | 60% to 100% |
| google/gemini-3.8-flash | 58% (35/60) | 46% to 70% | 33% to 83% |
| Model | Picked an expected tool | 95% Wilson interval | 95% interval across requests |
|---|---|---|---|
| openai/gpt-6.1-terra | 35% (19/55) | 23% to 48% | 11% to 62% |
| anthropic/claude-haiku-5.5 | 75% (41/55) | 62% to 84% | 49% to 95% |
| google/gemini-3.8-flash | 73% (40/55) | 60% to 83% | 45% to 100% |
| Model | Picked an expected tool | 95% Wilson interval | 95% interval across requests |
|---|---|---|---|
| openai/gpt-6.1-terra | 83% (50/60) | 72% to 91% | 58% to 100% |
| anthropic/claude-haiku-5.5 | 88% (53/60) | 78% to 94% | 68% to 100% |
| google/gemini-3.8-flash | 83% (50/60) | 72% to 91% | 58% to 100% |
| Model | Picked an expected tool | 95% Wilson interval | 95% interval across requests |
|---|---|---|---|
| openai/gpt-6.1-terra | 80% (48/60) | 68% to 88% | 57% to 98% |
| anthropic/claude-haiku-5.5 | 70% (42/60) | 57% to 80% | 47% to 92% |
| google/gemini-3.8-flash | 70% (42/60) | 57% to 80% | 45% to 92% |
Requests with no right tool were answered without a tool in every run, on every product and model. The one control request for Portravo, Intakra and MentionedOn each picked the competing tool in every run. Arguments fit the tool's schema in every call to an expected tool.
What it found
A schema shape Anthropic's models reject
Two real tools could not be sent to the Anthropic model as they were served: Postedly's postedly_quote, whose input schema had a top-level allOf with conditions, and Portravo's attach_plan_pdf, which had a top-level oneOf. Anthropic's API refuses a tool whose schema has oneOf, allOf or anyOf at the top level, and one such tool fails the whole request, so the model received no tools at all. For the baseline we removed those keywords from the schema sent to the Anthropic model only, and still checked its arguments against the full schema. As served, every request to that model on those two products would have failed.
One sentence in a description
On Portravo, gpt-6.1-terra picked an expected tool in 35% (19/55) of runs, against 75% and 73% for the other two models. The cause was one tool. The description of get_profile began "Use first after connecting", and that model called it first on requests about projects, takeoffs and credits that had nothing to do with the account. The other two models did not.
A rewording that did not move the number
MentionedOn had the weakest spread. We wrote new names and descriptions for its tools and offered them to the same models. This was a simulation only; the product does not use them. The share of expected picks went from 73% (132/180) to 76% (137/180), with overlapping intervals, and one model went down. The requests that missed in the baseline mostly still missed. Two depend on a business id the user had not given, and in one the model reasonably did the arithmetic itself.
Some misses are in the scoring
A model asked what a certified letter costs picked Postedly's postedly_services, which returns starting prices, in all 15 runs. Our expected answer was the quote tool. That may be too narrow. The per-request counts are in the download, so a reader can judge each one.
The flat-schema fix, before and after
After the baseline, both products rewrote those two tools as flat schemas and released them: Postedly at commit 3c6094d and Portravo at 4ca62a9. On the same day we read the tool definitions again from those releases and reran the same requests on the same three models with the same number of runs, settings and tool order.
Anthropic's model can now be given the tools unmodified. In the rerun nothing was removed from any schema sent to it, across 145 runs, and no call ended in an API error.
Selection rates, on the same requests. The right-hand column keeps the new schemas but puts the baseline descriptions back, because those releases also changed some descriptions, including get_profile.
| Model | Before: baseline | After: tools as they are now | After, with the baseline descriptions put back |
|---|---|---|---|
| openai/gpt-6.1-terra | 85% (51/60) | 90% (54/60) | 87% (52/60) |
| anthropic/claude-haiku-5.5 | 82% (49/60) | 78% (47/60) | 77% (46/60) |
| google/gemini-3.8-flash | 58% (35/60) | 58% (35/60) | 58% (35/60) |
| Model | Before: baseline | After: tools as they are now | After, with the baseline descriptions put back |
|---|---|---|---|
| openai/gpt-6.1-terra | 35% (19/55) | 82% (45/55) | 35% (19/55) |
| anthropic/claude-haiku-5.5 | 75% (41/55) | 82% (45/55) | 71% (39/55) |
| google/gemini-3.8-flash | 73% (40/55) | 75% (41/55) | 75% (41/55) |
The Portravo checkout read for the rerun is the commit that records the 4ca62a9 release. Before, the Anthropic model was given the two tools with the keywords removed; after, as they are served.
How to read it. The schema change made the tools usable by Anthropic's model; it did not move selection. With the new schemas and the old descriptions the rates are within a few points of the baseline on both products. On Portravo, gpt-6.1-terra rose from 35% (19/55) to 82% (45/55), and that came from the new get_profile wording. Put back alone, with everything else current, the old get_profile sentence gives 33% (18/55). The Postedly changes are within the noise of 60 runs.
What the score is not
- Not completion. It says which tool was picked first. It does not say the task then succeeded, and tools were never executed.
- Not the assistants people use. Three inexpensive models, called directly with one short system prompt, one turn, no earlier conversation and no connected account. Real assistants add their own instructions and often hold many more tools.
- Not a statistical proof. Small, hand-written sets of requests and competing tools. Different ones would give different rates.
- Not stable. Models change. It is a measurement on November 16, 2026, at one temperature.
- Not a promise. Soxoa does not guarantee that any change will make an assistant pick a tool.
Download
One JSON file holds the summary for every product and model, the settings, the request wording, the competing tools, the rerun summaries and what the runs cost.
Cost of the runs: $3.98 for the baseline and $3.25 for the rerun, paid to the model host.
Related
The agent-ready build includes a measurement like this one before and after the work. The track record lists the registry listings and protocol milestones Soxoa's products have reached. Questions: hello@soxoa.com.