Looking for a unique "Conversation ID" in the App Insights for QnA Maker

6
votes

We're using QnA Maker attached to an Azure Bot Service. In the Knowledge Base (KB), we've added a Follow up Prompt to every question that says This did NOT answer my question:

Knowledge Base > Follow Up Prompt

Meaning if a bad response was provided, the end-user could indicate so, and the conversation history would look like this:

Example Conversation

What we're trying to do is replay that conversation history so we can see:

  1. The original user prompt
  2. The original answer
  3. The subsequent followup question / answer

We have Application Insights turned on, so we can see both questions as they come through from the logs with the following query:

traces
| project timestamp, 
          itemId,
          question = customDimensions.Question,
          answer = customDimensions.Answer
| order by timestamp

Which will return these two rows:

App Insights Logs

However, we're trying to find a unique conversation id or session id that can correlate both of those records. Notice that the itemId is very similar, but not identical:

53be8c14-702c-11ea-8c41-11c1c266dc55
53be8c13-702c-11ea-8c41-11c1c266dc55

Is there a unique key that can be used to join these two events?

One workaround is to just use the first 7 digits of the itemID and join based on that partial match like this:

traces
| where customDimensions.Question contains "This did NOT answer my question" 
| project itemId,
          SessionID = extract("^[a-z0-9]{7}", 0, itemId),
          timestamp
| join (
    traces
    | extend question = tostring(customDimensions['Question'])
    | extend answer = tostring(customDimensions['Answer'])
    | where message contains "GenerateAnswer" 
        and question  !contains "This did NOT answer my question" 
    | project itemId,
              SessionID = extract("^[a-z0-9]{7}",0,itemId),
              question,
              answer,
              timestamp
) on SessionID 
| project question, answer, SessionID, timestamp //, itemId, itemId1
| order by timestamp desc, SessionID

But we're not sure that value will reliably only differ by the 8th digit, so would prefer a less fragile ID

1
botframeworkazure-application-insightsqnamakerkql
Are you just trying to examine this via App Insights? It seems Active Learning might work here, though I haven't personally implemented that yet. Then you could handle collecting the feedback and training your KB directly instead of having to review the App Insights reports. – billoverton
itemID seems to be a property of QNA framework:github.com/garypretty/qnamaker-sync/blob/master/README.md - and you may want to check how your QnAItem(s) are defined. From your example it looks that first part of the UUID is not stable (53be8c14, 53be8c13) - while the remaining ones are stable (702c-11ea-8c41-11c1c266dc55). My guess would be that it is better to use trailing parts of the UUID. – Alexander Sloutsky
Have you considered adding conversation history to your bot, then when the "this did not answer my question" button is clicked, send up a custom telemetry event to Application Insights with the data that you want. That way you just query the custom telemetry event details. These could use a combination of the user id, channel id, conversation id, and time as the unique key. – Matt Stannett
You might also find this Power BI template helpful which allows you to drill into details. – Matt Stannett
@MattStannett, can you add that info to an answer? Providing the simplest vehicle toward collecting a Conversation ID if one isn't available in the default AI data is definitely the next step toward a solution – KyleMit

1 Answers

-1
votes

I recently implemented a QnA maker BOT with active learning.

The sample is located here

It is very similar to what you are trying to achieve.

enter image description here