MCP tool selection: getting called for the right request

One intent, two ways to build the questions

We started with flight search. Five applications were available: Iberia, Pegasus Airlines, Skyscanner, Trip.com and Turkish Airlines. Every trial included all 43 candidate tools. We recorded the functions Luna emitted in its initial response, without executing a flight search.

We generated 20 general flight-search questions with no brand names, then 20 from each application’s relevant tool definitions. That gave us 120 questions across six sets, each repeated five times.

A general request looked like this:

With the original descriptions, Trip.com was selected in 60 of the 100 general trials, Skyscanner in 14, and the three airline applications in none. The remaining 26 responses contained no tool call.

Trip.com received the most selections on this set. That tells us where the model sends these requests, without establishing whether each call was appropriate or what flights it would return.

What changes when the questions come from the tools?

The tool-sourced questions reflected the applications’ stated capabilities. One from the Turkish Airlines set was:

We let the generator use the supplied definitions naturally. We did not require it to insert or remove brand names. These are tool-sourced sets: some questions name a brand and others do not. Nine of the 20 Turkish Airlines questions named the airline.

The matrix shows the original run. Each row identifies the source of the questions; each application column shows whose tools were selected. Every row contains 100 trials.

On their own sets, Iberia was selected 47 times, Pegasus 42, Skyscanner 65, Trip.com 90 and Turkish Airlines 31. On the Turkish Airlines set, Trip.com was selected 32 times, almost as often as the airline itself.

Those numbers need the questions beside them. The sets differ in routes, dates, passenger details and brand wording. Publishers did not approve them as examples of their intended demand. A question derived from an airline’s tool can still suit a broader travel service. Comparing the diagonal values would not give a fair ranking.

A call can be a mistake. So can a missed call.

For a publisher, being selected on an unrelated request can mean an unnecessary API call, an unsupported operation or an unhelpful result. Missing a request the tool could handle is a separate failure. A benchmark needs to account for both.

Our selection counts do not distinguish them. Before measurement, we defined a limited rubric for tool eligibility and checked some argument constraints. That was a reviewer assessment, without publisher validation of the expected action for each request. It does not establish comprehensive rates of unnecessary or missed appropriate calls. Trip.com’s higher count alone tells us neither whether it made more mistakes nor whether it served requests better.

The no-call responses show why this distinction matters. One question explicitly requested Turkish Airlines flights from New York to Istanbul. In all five original trials, the model made no call; the visible answers asked for more information, including which New York airport to use. Turkish Airlines’ search schema requires an airport code. Whether asking was the intended behavior depends on the publisher’s policy for resolving city names. Our first-response metric stops before that conversation can continue.

We should decide what a successful response looks like before counting selections:

Make the calling conditions clear

Our design recommendation is to document the job, the information needed to start it, and the conditions under which the tool should wait or stay unused. Descriptions and parameter schemas need to tell the same story.

The schemas deserve attention too. Turkish Airlines takes airport codes; Trip.com accepts location names and says the service resolves them internally. Turkish Airlines’ cabin filter uses fare-family codes; Trip.com lists economy and business explicitly. These differences could affect how readily the model can construct a call. We kept schemas fixed, so we did not isolate their effect.

The original Turkish Airlines description also refers to an originDestinations array for multi-city searches, absent from its schema. Some visible responses declined to start because the tool could not represent all the legs. A clearer description can explain that boundary; it cannot supply a missing capability.

Before generating another benchmark, we would ask the publisher for real request examples, the information they require before a call, and the requests they want the tool to leave alone. We would compare those expectations with the actual MCP definitions and have the publisher review the expected actions. General-intent demand and the publisher’s intended use cases would remain separate views.

What the description replays added

We also replayed all 120 questions twice, changing only three Turkish Airlines tool descriptions. One version added workflow detail; the other stated the purpose and boundaries more briefly. The original, detailed and concise runs each contained 600 trials.

The revisions showed no clear selection improvement. The concise-versus-original difference on the airline’s set was zero percentage points, with a 95% question-cluster bootstrap interval from −8 to +8. Against the detailed version, it was +4 points, with an interval from −3 to +12. The comparison chart and all three description texts are available with the data.

Questions, schemas, other descriptions, settings and per-trial tool order stayed fixed. Application positions were balanced across repeats. The runs happened at different times, and the questions were not held out from the revision process. Both wording and length changed. We cannot isolate a length effect or rule out a better result from another rewrite.

Inspect the evidence

The evidence bundle contains all 120 questions, 1,800 selection observations, visible answers, tool definitions and description comparisons. Preparation calls and the earlier multi-intent pilot are excluded. Model outputs and source definitions are preserved as research records; their contents are not verified travel advice or examples of correct calls.

An application counts once if any of its tools appears in the initial response, including a usage-guide tool. Multiple applications can count. We did not execute calls or assess fares, bookings or final host answers.

The questions, selection records and three result matrices are also available separately. These are generated scenarios on one model, rather than customer traffic. The README explains request assembly and uncertainty estimates.