Text analytics without a silver bullet: where the real value is in reviews and tickets
Why processing customer text starts not with understanding language, but with routing and classifying reasons.
When the conversation turns to analysing customer reviews, support emails, or service requests, the first impulse is almost always the same: "let's build something that understands what the customer means." That sounds like the right goal. In practice it turns out to be too broad to be useful.
Understanding a text is not a task. Making a decision based on a text is. The moment you look at it that way, it becomes much clearer where to start and where to stop. This is the same principle behind The Watson lesson: business buys faster answers, not smarter machines - the value is in reduced decision time, not in the sophistication of the machine.
What text analytics actually does in a company
Customer reviews, support letters, service requests - these are not data for its own sake. Behind every text there is some action: reply, route to the right person, feed back into product work, track as a trend.
That leads to a short list of tasks where text processing delivers measurable results:
- routing a ticket to the right department or person;
- classifying the reason for the contact against a fixed vocabulary;
- flagging urgent or anomalous cases that need fast handling;
- grouping similar contacts to track frequency over time.
All of this is achievable without deep "language understanding." What it needs is solid labelling, clear categories, and consistent classification logic.
Where companies usually lose time
The most common mistake is starting by trying to cover everything. A system is built to extract topics, assess sentiment, detect intent, and produce summaries. Three months later there are attractive charts that nobody trusts, and not a single process that has changed.
The second mistake is expecting automation to replace the manual work of building categories. It will not. If the company has no clear vocabulary of contact reasons, no algorithm will create one. It will reproduce whatever is already in the data, including all the accumulated ambiguity.
The third mistake is measuring the system by "accuracy on the test set" instead of asking what changed in the operational process.
Where real value comes from
Value appears not when the algorithm "understood" a text, but when someone made a faster or better decision because of it.
The most direct path is automatic routing. If a ticket reaches the right person without manual sorting, that already saves time and reduces errors. It is measurable, it is felt, it changes the process.
The next step is reason classification. Here you need to do the work manually first: take a few hundred contacts, label them by hand, and build a vocabulary of 10-20 categories that are actually used in decision-making. Then automate. Not the other way around.
What a labelling scheme that works looks like
A good labelling scheme for customer texts answers three questions:
- What type is this contact? - not a broad topic, but a specific operational category.
- What action is needed? - reply, route, log, escalate.
- Are there signals of urgency or anomaly? - a flag, not a rating.
Categories should be mutually exclusive and exhaustive for the real data. A simple check: if two analysts label the same text independently and disagree half the time, the scheme is broken, and no algorithm will save it.
A simple filter before you start
Before investing in text analytics, I usually check a few things:
- Does the company have a documented vocabulary of contact reason categories? If not - start there.
- Who acts on the text, and exactly how? If the answer is unclear, it is not clear what to automate.
- Are there 500-1000 labelled examples? If not, they need to be created by hand - that is the foundation.
- What will change in the operational process when the system is running? If the answer is vague, the goal is wrong.
Text analytics works well when the task is specific. When the task is "let's understand what customers want" - that is research, not automation. Research is useful too, but that is a different conversation.