It’s live! Access exclusive 2024 live chat benchmark data & see how well your team is performing.
Get the data
The decision to take on chatbot customer service is an exciting one for companies. Like with any new initiative, there are a series of considerations that organizations should take into account to ensure this endeavor goes according to plan. One of those considerations is metrics.
As Forrester notes in their 2016 report, How Analytics Drives Customer Life-Cycle Management, “Every customer interaction leaves a trail of customer data waiting to be analyzed.” Companies that establish thoughtful metrics for their chatbots will find a wealth of resources waiting to help them optimize their live chat offerings. These metrics can be planned – and checked for quality – by comparing them to your existing agent metrics.
Comparing bot and agent metrics can help your organization deliver consistent service that is aligned with company standards. It can help your business identify any weak spots, or places where your bot or agent-led service needs improving. It can help make valuable management decisions regarding workflow, and help you guarantee that you’re getting the best value from your chatbot and human teams.
But how should human versus chatbot metrics be treated? Which metrics are important for both, and which are chatbot specific? And how should companies address any service metric gaps between the two? This is what we have discovered.
Let’s start with a simple question: Can human live chat agents and chatbots be held to the same standard when it comes to key performance indicators (KPIs)?
This depends on the metric. For time-based metrics, it is impractical to hold bots and human agents to the same standards. As automation systems, chatbots can – and should – respond instantly. Unlike human live chat agents, they don’t need to think or spend time typing out an answer.
However, live chat agents and bots should always be held to the same customer service standards. Whereas agents can be forgiven for their human speed, the automated nature of chatbots is no excuse for a slip in service quality.
Recommended for you: Chatbot Do’s and Don’ts – These Are the Best and Worst Chatbot Practices
A Pelorus research survey found that 74% of contact center managers felt that improved customer service agent technology can decrease error rates and improve the customer experience. Chatbots can’t be expected to meet the customer service requirements of every website visitor, but a successful chatbot should still be able to improve the customer experience.
Now that we know under which circumstances chatbots and live chat agents can be held to the same standards, let’s take a look at the customer service KPIs companies can use to measure the performance of both chatbots and human agents.
1. Total Number of Chats
How many chats is your chatbot picking up? How does this compare to the number of chats that your agents are taking care of each day, week, or month? By measuring these two numbers against each other, you will gain insight into how your chatbot is reducing agent workload and saving your company money. You can also compare the total number of chats that your chatbot is handling on a periodic basis to track how your chatbot is growing.
2. Average Handle Time (AHT)
What is the average duration of a single interaction with your chatbot? This should be shorter than the average handle time of your live chat agents. (According to Comm100’s 2019 Live Chat Benchmark Report, the average chat duration for agents in 2018 is 11 minutes and 53 seconds.)
3. Wait Time
How long are your visitors kept waiting before a live chat session starts up? Chatbots should keep user wait time for bots and live chat agents low. In 2018, the average wait time was 48 seconds. Is your chatbot able to help you beat this average?
4. Response Time
How quickly is your chatbot responding? Responses should be near instantaneous to avoid frustrating customers – after all, speed is one of the chatbot’s greatest strengths.
5. User Satisfaction Rate
How did your users rate their interaction with your chatbot against live chat agents? Both chatbots and human agents should be able to achieve high user satisfaction rates.
6. First Contact Resolution (FCR)
How many queries was your bot able to handle without any assistance? For chatbots, you can also measure FCR by your Support Agent Takeover Rate. How many times did agents have to step in and take over for your chatbot? By comparing these numbers to the total number of chats, you can measure the effectiveness of your live chat agents and your chatbot.
7. Customer Effort Score
How easy was it for your customer to solve their problem with your chatbot or live chat agent? Like the user satisfaction rate, this information can be obtained from a customer satisfaction survey.
8. Net Promoter Score
How likely is your customer to recommend your company’s services and products? A quality chatbot can help your company improve this score, which is also easily calculated via customer surveys.
You can’t paint agents and chatbots with a single metrics brush. Here are some chatbot-specific metrics that don’t apply to agents.
1. Disorientation Rate
Imagine if we had to measure live chat agents with a disorientation rate. This KPI measures how many times the chatbot did not understand a question or asked the customer to repeat themselves. It can be used to determine the quality of your chatbot’s decision tree, and identify any potential areas of improvement. Note that the disorientation rate can also account for user confusion, which is why it’s important that your chatbot is straight-forward and easy to use.
2. Activation Rate
To what extent is your chatbot being used? Per Telus International, “This KPI encompasses multiple metrics, including the total number of users, how many users opened a message they received from a chatbot and the number of users who engaged with the bot by sending back a message.” As John Asher, CEO of growth strategy consulting firm Asher Strategies explains, “With activation rate, you want to look at new, active, and engaged users separately and on a monthly basis.”
3. Retention Rate
How many users revisit your chatbot within a given time frame? Depending on the industry and what use cases your chatbot can help with, this may vary. For example, a customer service chatbot for an ecommerce company may see repeat use from the same visitor every six months, whereas a banking chatbot may get weekly use by users. Keep in mind that the retention rate can reveal a few different things. If your company has a high first contact resolution rate, then your customer may not have to revisit your chatbot. However, retention rate can also indicate that your customers prefer using your chatbot to other contact channels. Use this number in conjunction with user satisfaction rate to get the best picture of why your customers are returning to your bot, and whether it is a good thing.
4. Volunteer Users
Volunteer users are the number of users that interact with your chatbot on their own, without being prompted to initiate a conversation (think: pro-active chat invitation). According to Asher, “This is typically a quasi-indication of your marketing and branding efforts. Volunteer users are what you want. These people come to your site because they’ve heard about (your chatbot) and recognize its value. That usually means they’re coming in with a real purpose, and will be more engaged.”
5. Number of Interactions Per User
How many messages are users exchanging with your chatbot? For this metric, a higher number may reflect a helpful chatbot with good engagement – but it may also be indicative of a chatbot that customers are struggling with. You can monitor the quality of your chatbot interactions or compare this KPI to customer service ratings to better understand this number.
6. Goal Completion Rate
Like any other business initiative, every chatbot has a goal (in fact, some have many goals). Just as companies may incentivize their live chat agents to upsell a service or promote a specific product, some customer service chatbots may have a similar objective. For example, aside from delivering excellent service, a car dealership’s chatbot may have another goal: to help users book test drives. By looking at the chatbot’s most used phrases, you can see what your users are interested in, and how well your chatbot is completing its intended functions.
Let intelligent bots handle 80% of your customer needs so your team can focus their time where it really matters.
Learn moreThe ultimate purpose of the customer service chatbot is to answer simple queries 24 hours a day – saving your company money, lowering wait times, and freeing your live chat agents up to handle more complex customer service scenarios.
This goal means that your metric-handling duties won’t stop at setting and tracking chatbot’s metrics: you may have to make adjustments to the metrics that are in place for your live chat agents as well.
Chatbots take care of the simple queries that your human agents can answer quickly. Pre-chatbot, these were the queries that your live chat agents may have handled via quick canned messages and copy-paste responses. Once you adopt chatbot customer service, your bot will be able to take care of a large portion of these simple queries. Comm100’s own chatbot takes care of around 26.65% of incoming queries, a number that continues to grow as our chatbot technology advances. Researchers predict that within the next five years, between 75 and 90 percent of queries will be dealt with by chatbots in some industries.
As a result, be prepared to see your live chat agents’ average handle time go up. This shouldn’t be seen as a bad thing: With your live chat agents handling more complex queries, they will need more time to offer the helpful, nuanced responses that your customers expect of them.
The more complicated queries your agents handle, the better they will become at taking care of them. As your agents become experts in more involved customer service scenarios, you can expect their handle time to begin to decrease once again. Just remember not to prioritize speed over quality care, or you risk hurting the customer experience.
Once you have set your chatbot metrics and readjusted your human agent metrics accordingly, things may sail smoothly. However, in the world of business, smooth sailing is rarely a permanent state.
There will likely be times when you notice a performance gap between these two important parts of your live chat team. Take the following steps to close the gap between your chatbot and human-led customer service, and to prevent performance slips from arising in the future.
1. Identify Where Metrics Are Slipping
The first step to closing the gap between bot-led and human-led service is attentiveness. Performance gaps can usually be caught early by keeping your eye on your most valuable metrics. Pay attention to your customer service and time-based metrics, and make sure that all parties are performing as expected. Some performance slips may be temporary and reflect the conditions of a certain day, such as your wait time increasing on a day with a high volume of chat requests. Others – such as a rise in customer complaints – may point to a bigger underlying problem. For chatbots, review all questions that the bot failed to recognize, and queries that were marked as “Not Helpful” by visitors. Review chat transcripts between the chatbot and visitors, and proactively go over visitor ratings of the chatbot. With careful attention and good judgement, you will be able to intervene before performance gaps take their toll on the customer experience.
2. Apply Better Chatbot or Agent Training
To err is human. As it turns out, it is also chatbot. Depending on where you see room to improve your customer service metrics, turn to your chatbot and/or agent training process to get to the root of the problem, and begin to remedy it. Chatbot “training” is possible in the sense that chatbots can use machine learning to convert user interactions into structured data. Comm100’s chatbot uses Natural Language Processing (NLP), which allows your chatbot to “learn” from examples that you provide it, and use these examples to improve its algorithm. You can train your chatbot using real conversation logs, which helps it match user cues with the appropriate action as often as possible. You can also expand your chatbot’s glossary to help your chatbot better guide users towards resolution – regardless of their jargon.
Agent training is more traditional, but less straightforward than chatbot training. Not every agent performance gap will be remedied in the same way. One way to take care of this gap is for management to address the problem head-on by providing examples of the problem (such as reading customer complaints or showing image captures of handle time charts), and offering a friendly reminder to aspire to quality service.
Recommended for you: Winning Live Chat Training for Your Customer Service Team
Other possible options include the use of training activities to encourage your agents to practice skills that will improve their customer service. Remember that motivation also plays a role in performance slips – sometimes motivational activities are more important than actual training to improving performance and closing metrics gaps. For common and reoccurring live chat metric gaps, consider how you can add awareness of that problem to your training program to help prevent it from happening with future live chat agents.
3. Use Real-Time Quality Assurance
Unlike monitoring phone customer service, checking live chat customer service for quality is easy – and can be done in real-time.
Real-time quality monitoring can be used to keep an eye on the performance of chatbots and agents in times of metric gaps. This type of chatbot monitoring is especially important for keeping tabs on areas where your chatbot may have a higher disorientation rate. It can also be used to make sure that chatbots don’t mess up when handling important queries. To combat this, you can set up your chatbot service to notify agents about certain categories of visitor questions, helping your team keep an eye on the user queries that matter most. By engaging in real-time monitoring, you will be able to make improvements based on real-time information. For gaps in human agent metrics, you can use real-time monitoring to help new agents who are still learning the ropes. For time-based metric gaps, you can also use this feature to make sure that your agents are being productive, and aren’t just “sitting” on an already resolved chat.
4. Test Customer Service with Mystery Shopping
Another way to address and prevent customer service metric gaps is through mystery shopping. Mystery shopping is a tool used by organizations or third-party groups to test the quality of a company’s service. Mystery shoppers may take on specific tasks, such as asking about a product, making a purchase, registering complaints, or exhibiting a specific set of behaviors to make sure that the service is on-par with company standards.
Mystery shopping can be used as a means of testing chatbots and auditing the quality of service provided by live chat agents. The findings from mystery shopping reports can be put towards improving chatbot and agent training, and can be used to close metrics gaps.
Even better than closing metric gaps, this form of customer service quality measurement can be used to expose issues that may not be accurately reflected in live chat KPIs, such as customer satisfaction and customer effort. Remember that only 4 percent of dissatisfied customers complain, while 96 percent will never let you know that there was a problem.
5. Analyze Comparison Reports
Comm100 has a powerful reporting system – and that extends to comparing the performance of live chat agents against that of your chatbot. Our Comparison Report helps companies better understand differences in productivity and satisfaction levels between human agents and the chatbot. Currently, reports include several metrics including total chats, average chat time, and average chat rating.
The side-by-side visibility of this report helps you to make better resourcing and management decisions, and helps you to ensure you’re getting the best return from both your Chatbot and human teams. These decisions can be used to combat KPI gaps before they even happen.
6. Reflect on Your Metrics
Agents will probably never be able to answer as quickly as a chatbot – and that’s okay! In the end, chatbots are here to help your agents and contribute to the team – not compete with them.
Make sure that your KPIs are realistic and attainable. If your agents seem unaware of your customer service metrics, reflect on how clearly and frequently you are communicating your metrics to them. Are there any steps that you could take to achieve greater metric transparency?
Your chatbots are not human agents, and your human agents are not chatbots.
It is important that you assess your customer service metrics for each independently, while also putting custom practices into place to help both your agents and your chatbot succeed.
By using our comprehensive reports and real-time transcripts, coupled with quality management audits and chatbot/agent training, you will be able to tackle metrics gaps between bot-lead and human-led service, creating a customer service experience that is stronger and more cohesive than ever.
For more information on how to create a successful chatbot experience, check out our blog post, Journey Mapping for Chatbots: How to Create a Chatbot Decision Tree from Scratch.
In this guide, we break down some of the considerations and steps you’ll need to follow to ensure that your chatbot adds to customer experiences, without degrading them at any stage.
Download Now
By clicking "Subscribe", you agree to our Privacy Policy.
The way brands promote their solutions has changed over time. We’ve come a long way from traditional marketing tactics and embraced new ways + Read More
Customer Satisfaction (CSAT) scores are pivotal for businesses aiming to thrive in a competitive marketplace. These scores not only reflect customer perceptions of + Read More
The demand for fast, efficient, and personalized customer service is growing – and growing fast. 54% of global consumers have higher customer service + Read More
