Check for False-Positives & Out-of-Scope Queries
Out-of-scope queries refer to questions that the virtual assistant failed to comprehend. In such instances, it’s also possible to identify false positives – situations where the virtual assistant mistakenly believes it has correctly understood the user’s request when in reality, it has misinterpreted the user’s intent.
Below you will find more details about TP, TN, FP, and FN scenarios with examples:
True Positive
True Positives (TP) refer to instances where the virtual assistant correctly identifies the intent of an utterance. For example, if the user says “What’s the weather today?”, and the virtual assistant correctly identifies the intent as “get_weather”, this would be a True Positive.
In this example the intent is correctly mapped to Check Balance, hence it is a true positive
True Negative
True Negatives (TN) refer to instances where the virtual assistant correctly identifies that an utterance did not match any of the defined intents. For example, if the user says “I’m not sure what you mean”, and the virtual assistant correctly identifies that this does not match any of the defined intents, this would be a True Negative.
In the following example, the user utterance “Extremely Likely” did not match with any defined intents and is categorized as Unidentified intent.
False Positive
False Positives (FP) refer to instances where the virtual assistant incorrectly identifies the intent of an utterance. For example, if the user provides his bank account name, and the virtual assistant incorrectly identifies the intent as “Close Account”, this would be a False Positive.
False Negative
False Negatives (FN) refer to instances where the virtual assistant incorrectly identifies that an utterance did not match any of the defined intents. For example, if the user says “What’s the weather today?”, and the virtual assistant incorrectly identifies that this does not match any of the defined intents, this would be a False Negative.
In this example, the “create account” utterance is wrongly mapped as an Unidentified intent, and hence would be False Negative.
Retrain Your Machine Learning Models
Once you’ve identified the false positives and out-of-scope queries, the next step is to add that data, the utterances, or those queries back into the training data. Optimizing your machine learning models through continuous retraining is key to enhancing the intelligence of your virtual assistant. This critical step helps to reduce discrepancies and improve how the virtual assistant understands a user during an engagement.
Method #2 Changing The Scope of Your Use Cases
Another way to improve NLP performance is by changing the scope of your use cases. For instance, you might have two unique use cases that are verbally similar and users might ask their questions in a similar manner for both. For example, ‘ Transfer funds’ and ‘Make a payment’ are two unique use cases that users may request in a similar manner.
This is why the scoping and design phase of your virtual assistant is so critical. You may discover that certain queries are being incorrectly categorized under the wrong intent. This insight allows you to revisit and adjust the scope of each use case, ensuring it’s specific enough to match user queries accurately, while also being comprehensive enough to encompass the variety of ways a topic could be inquired about.
To learn more about building and improving virtual assistants you review our documentation page on improving NLP performance.
Want to Learn More?
We’re here to support your learning journey. Ready to take on bot building but not sure where to start? Learn conversational AI skills and get certified on Kore.ai Experience Optimization (XO) Platform.
As a leader in conversational AI platforms and solutions, Kore.ai helps enterprises automate front and back-office business interactions to deliver extraordinary experiences for their customers, agents, and employees.