1
votes

THe FEB Update of Watson Conversation brought a new way of scoring intents... as described here: https://www.ibm.com/watson/developercloud/doc/conversation/irrelevant_utterance.html#absolute-scoring-and-irrelevant-input

Things which worked before ... are now declared in the TRY IT OUT as IRRELEVANT and it falls into the ANYTHING_ELSE answer.

THe aboce description simply states: "As intent confidence scores change, your dialogs may need restructuring"

  • Is somebody out here who can explain WHAT kind of RESTRUCTURING is required here?
  • How can I control when I get "multiple intents returned" ?
1

1 Answers

2
votes

If all of your inputs are now hitting the anything_else node it is a signal that your intents do not have enough training examples relative to your input utterances.

The updates added for the 2017-02-03 release added the ability to score intents independently (absolute scoring), and also to better detect when an input is irrelevant. Inputs are deemed to be irrelevant if the confidence of their intent is 0.2 or less. If an input previously returned a certain intent, but now shows "Irrelevant" in the Try it out panel then it is a sign that the intent now returns a low confidence. You have a few options in terms of what can be done by way of 'restructuring'.

Probably the best route would be to add more training data (examples) to the intents in question. Adding more training data will allow the Conversation service to more confidently predict whether or not the given utterance matches the provided intents. The other option is to use 'long hand' notation when defining your Dialog node conditions. I suspect today you have syntax like the following in your Dialog node condition:

#my_intent_name

Under the covers the Dialog engine is essentially doing the following check in that case:

if(intents[0].name == "my_intent_name" && intents[0].confidence > 0.2)

If you don't care about potentially irrelevant input you could use the following 'long hand' notation in your Dialog node condition:

<?intents[0].intent == "my_intent_name"?>

This statement will always evaluate to true if the first (top) intent's name is "my_intent_name". The confidence value is ignored. Of course doing this presents its own problems, in that even if the confidence is 0.01 the condition will evaluate to true, even though likely, if that is the confidence, then the utterance is irrelevant/off-topic for the workspace.

We also mention in the documentation that one of the primary advantages of "Absolute Scoring" is the ability to return multiple intents, or more specifically multiple intents with a high confidence. Previously, with relative scoring the sum of all intent confidences amounted to 1. It is worth noting here that even though the 'default' message API call returns one intent in the intents array, under the covers Conversation is evaluating multiple intents but just returning the top most intent (in terms of confidence). In prior releases the sum of the confidence scores of these intents must amount to 1.0. As a result of this normalization confidence scores were skewed a little. Each intent's score was basically relative to the previous intents confidence. If the system was .56 confident in the top intent, the most the remaining intents could amount to was .44. As a result when the user entered something like "Turn on the lights and find the nearest restaurant" the system typically returned a much higher confidence for the first intent. With absolute scoring the system can show higher confidence in multiple intents in such a scenario. As in previously releases, calling the message API with alternate_intents:true as a part of the request payload will result in an array of intents being returned for the given request. In your Dialog node condition if you wish to check an attribute of second or subsequent intent you must use long hand notation similar to what is shown above, replacing the intent index (0) with the index you are interested in. For example if you want to check if the system is confident in the second intent you could use the following notation:

<?intents[1].confidence >= .5?>