How to choose the AI model behind your support bot

Updated 2026-09-24 · 5 min read · By the Convo360 team

Why published comparisons will not help you

Model comparisons measure reasoning puzzles, coding tasks and exam questions. None of that resembles a customer asking whether you deliver to Nashik by Friday.

Support work has a very different profile. The model needs to retrieve the right fact from your material, phrase it warmly and briefly, recognise when it does not know, and stay on topic when the customer wanders. Those are not what leaderboards test.

The result is that a model ranked lower on public benchmarks frequently performs better on your actual traffic, particularly if it is more willing to say it does not know. For a business, a model that refuses cleanly is worth more than one that is slightly cleverer and occasionally confident about something false.

There is a second reason to be sceptical of published comparisons: they age within weeks. Providers release new versions constantly, and an article comparing three models is describing a moment that has already passed. Your own test, by contrast, can be re-run in an hour whenever you need it.

The thirty-message test

This takes about an hour and settles the question properly.

1. Export thirty real customer messages from last month. Include the awkward ones: vague questions, Hinglish, complaints, and two or three that your bot got wrong.
2. Configure two or three agents in Convo360, identical except for the model.
3. Run the same thirty messages through each.
4. Read the replies side by side, without knowing which is which if you can manage it.

Score on four things: factual accuracy against your knowledge base, whether it refused when it should have, tone, and length. Tone and length are not cosmetic; a correct answer in six paragraphs will not be read.

Include two or three questions your knowledge base deliberately does not answer. This is the most revealing part of the test. A model that confidently invents a delivery timeline you never published will do the same to a real customer, and you want to discover that in an hour rather than in a complaint.

What to weigh, and how much

Speed deserves a note. On chat, two seconds against four is barely noticeable. On a voice call it is the difference between a natural conversation and an awkward one, so voice agents should weigh it much higher.

Small models are usually enough

The most common and most expensive mistake is using a flagship model for work that does not need one.

Retrieving a delivery timeline from a knowledge base and phrasing it warmly is not a hard reasoning task. Small, fast models do it well, at a fraction of the cost and with lower latency.

Reserve the stronger models for where the reasoning genuinely helps: a sales agent weighing several products against a customer's stated needs, or an agent summarising a long complaint for a human to pick up.

On Convo360 the model is set per agent, so this is a dropdown rather than an architectural decision. Running a cheap model on support and an expensive one on sales is the normal configuration, not an optimisation.

Latency has a second effect people underestimate on chat. A reply that takes six seconds often arrives after the customer has sent a follow-up message, which produces a confusing two-message exchange. Under two seconds the conversation feels like a conversation.

What actually fixes a disappointing bot

When a bot gives poor answers, the model is rarely the cause. Check these first.

Is the answer in the knowledge base at all? A surprising number of complaints about AI quality turn out to be questions no document answers.

Is the knowledge base contradictory? Two documents with different return windows produce an unpredictable bot, and no model resolves that for you.

Is the prompt specific? "You are a helpful assistant" produces a generic bot. Three lines about your business, your tone and your boundaries produce a useful one.

Is the handover rule explicit? Decide which topics always go to a person, and say so in the prompt.

Work through those four before changing models. In most cases the disappointment disappears without touching the provider.

A fifth check is worth adding: does the bot know its own limits about your business specifically? A model told it handles support for a jewellery brand should decline to advise on gold investment, and whether it does is a prompt decision rather than a model one.

Keeping the decision current

Keep your thirty test messages in a file. They are the most valuable asset in this whole process, because they turn every future model decision into a one-hour exercise rather than a debate.

Re-run them quarterly. Model prices have fallen repeatedly, new versions appear, and the model you chose in January may now be both slower and more expensive than an alternative.

Keep a second provider key configured and tested even if you never use it. Outages happen, and switching a dropdown during one is a far better position than discovering mid-incident that your only key is with the provider that is down.

Finally, write down why you chose what you chose. Six months later, somebody will ask, and a one-paragraph note prevents the entire evaluation being repeated from scratch.

One organisational note. Whoever runs this test should be somebody who reads customer messages regularly, not only whoever configures the platform. The judgement being made is about tone and usefulness, and that is a support skill rather than a technical one.

Step by step

1. Collect thirty real messagesFrom last month, including vague questions, Hinglish, complaints and cases your bot previously got wrong.
2. Build identical agentsSame prompt, same knowledge base, different model. The only variable should be the one you are testing.
3. Read the replies blindScore factual accuracy, whether it refused when it should have, tone and length. Hide which model produced which if you can.
4. Put routine work on a small modelOrder status and policy questions need retrieval, not reasoning. Save the stronger models for sales and summarising.
5. Keep a second provider configuredTested and ready, so an outage is a dropdown change rather than an incident.
6. Re-run the test quarterlyKeep the thirty messages in a file. They turn every future model decision into an hour of work.

Frequently asked questions

Related reading

Available across India — see WhatsApp & AI calling automation by state and city.