One description change, and a published ChatGPT app lost the first call in 9 of 10 chats

What I ran

Two flight-search tools were live in the same ChatGPT conversation. One was Flight Network, the published app from Etraveli Group that sits in OpenAI’s curated directory. The other was Vritax Flight Test, a developer-mode connector I built for this, with Flight Network’s own search_flights signature copied over: same title, same input schema, same annotations, neutral server instructions. The only thing I ever changed on my side was the tool description.

The request was the same every time, typed in Turkish, no app named:

Prompt used$ onumuzdeki hafta istanbuldan erzuruma gidecegim. istanbuldaki herhangi bir havalimanini kullanabilirim. pzt veya sali gidebilirim. en ucuz ucak biletini bul

Roughly: I’m going from Istanbul to Erzurum next week, any Istanbul airport is fine, Monday or Tuesday, find me the cheapest ticket. Ten fresh chats per phase, memory off, custom instructions blank, and the score is simply which tool the host called first.

Four phases

Phase	Vritax description	Vritax called first
A1	exact copy of Flight Network’s own description	0/10
B	overfit: names the route and the days	9/10
A2	exact copy again	0/10
R	realistic: no place names, no weekday names, nothing quoted from the prompt	9/10

In A1 and A2 the two tools were, at the text level, identical. The host called Flight Network first in all twenty chats and then, every single time, called mine as well, splitting the two dates between them: Flight Network got Monday, Vritax got Tuesday. Same description, and the published app gets the first call. That is the baseline everything else is measured against.

B is the maximum dose. The description says, more or less, use this for the cheapest one-way from Istanbul to Erzurum next Monday or Tuesday. It won 9 to 1. A2 is the control: put the copied text back and it drops straight to 0 of 10. The winner followed the description, not the server, which is the thing I needed to see before believing any of it.

R is the phase that matters for anyone who has to ship. I wrote that description under three rules: no place names, no weekday names, no echo of the prompt’s wording. Only things that are generally true of the tool and happen to be what this class of request needs: every result carries a total price sorted lowest first, origin and destination accept metro codes that cover all of a city’s airports, and for flexible dates you call once per date and compare. It also won 9 to 1, and in all ten chats the host queried both dates through it, which the copied description never managed once. In B and R the host called both tools in 4 and 5 chats out of 10; I count the first call.

Same thing through the API

I ran all four phases again against the Responses API with the same two tool definitions, Flight Network always first in the tools array, 40 requests, all 200, all resolved to gpt-5.6-sol. Result: 0–10, 10–0, 0–10, 10–0. Same direction as the host in every phase, just more decisive. The API doesn’t execute the calls, so it can’t see published-app priors, the rendered carousel, web fallback, or which fare ends up in the answer. In this setup it was still a good enough compass, which matters because an API run costs cents and a browser run costs an evening.

One API trial (A2-02) read “next week” as the week after and added a direct-flights filter nobody asked for. Noted, not corrected.

Why the overfit description can’t ship

It mirrors the prompt. Route, days, the lot. It is keyword stuffing with extra steps, and I’d expect hosts to start penalizing it the way search engines did. It’s in the study as the upper bound, so I know what the ceiling looks like. The real question was whether a description you could actually put in front of a review team gets anywhere near that ceiling. It got all the way there.

Picked is not used

In R, Vritax got the first call in 9 of 10 chats. Its fare made it into the final answer in 4. In the rest the host went with Flight Network’s rendered carousel or a web search, and in a couple of chats it explicitly named the 1,525 TRY fare my tool returned (a dummy fare, the connector doesn’t sell tickets) and then set it aside as unverified in favor of a 3,599 TRY fare from the airline’s site. Two R chats never produced a scorable final answer, one stuck generating, one hit a rate-limit overlay; first-call scoring was unaffected. Utilization is a second surface with its own levers, and it’s out of scope here.

What this does and doesn’t say

One intent, one route, one competitor, and the test connector is mine. The name “Vritax Flight Test” may itself be a negative signal; I couldn’t change it mid-series without invalidating the earlier phases. A finding this narrow says: on this request, in this host, on this day, the description moved the first call from 0 to 9 out of 10 and back. It does not say yours will. Directional, not a prediction.

If you ship a tool

Write the description against the class of request, not against a prompt. Say what the tool is genuinely good at in the terms that class uses, and tell the model how to handle the flexible parts (several dates, any airport in a city) so it doesn’t have to guess. Then measure the first call and the final answer separately, because you can win one and lose the other in the same chat.