The challenge
The client, a subscription language-learning app, was growing subscribers faster than its support could follow. The chatbot ran on keyword-matched canned replies: ask where the pronunciation course lives or why you were charged twice, and you got the same apology. The most common user behavior was asking three times, then demanding a human — so the support team spent its days cleaning up after the bot, and first-contact resolution stayed stuck at the bottom.
How we researched it
We did not start by picking a model. We started by classifying three months of real support conversations. Roughly seventy percent of questions had standard answers — they were just scattered across the help center, course pages, and old announcements where the bot could not reach them. The cases that genuinely needed a human, billing disputes and refunds, were under ten percent. That distribution decided the whole solution: the problem was retrieval, not generation.
What we did
We consolidated help docs, course information, and plan rules into a retrievable knowledge base and rebuilt support as retrieval-augmented generation: find the evidence first, then write the answer, with every sentence traceable to a source. For model selection we blind-tested mainstream AI engines against the same set of real support questions and picked the best cost-quality balance. We also drew hard handoff rules — billing issues and escalating emotions go straight to a person.
Results and the verdict
Eight weeks after launch, first-contact resolution climbed and average response time dropped sharply; the support team went from firefighting to handling the genuinely hard cases. An honest note: mid-project we tried dropping retrieval and letting the model answer freely. The replies read smoother and the error rate doubled overnight, so we killed it. This case was won by the quality of the knowledge base, not by the model.
