Azure Qna Maker multiple languages

Question

I am working on a bot within Microsoft Azure environment, which is able to answer questions in four languages. The overall scheme is the following: the input of the user is submitted to Azure TextAnalytics, and the deduced language is used to route them - by the help of Cognitive Services (LUIS and QnA Maker) - to a Knowledge Base (in their language) giving them the answers of their questions.

Qna Maker manages the data of the KBs through an Azure search service. But as soon as a KB is assigned to one, an index (kbId) is created within the latter which defines the language of - I assume - all KBs that will be managed by this search service.

Redundancy of resources and most importantly costs are getting quite silly: the need to create a QnA Maker for each language; that is 1 QnA cognitive service, 1 app service, 1 search service (and optionally 1 application insights) in parallel for each language is ridiculous. Surely alternatives in Azure must exist. But I find it very difficult to find them, since the kbId index is unique, it accepts only one language, and it cannot be updated once created.

I would really like to know if other people came across such problems. If solutions exist -other than the pricey one duplicating everything- or if I may have missed out something from Azure QnA Maker...

Thanks a lot for your help!

Answer 1

It's true that QnA maker supports only a single language of your choice to create a knowledge base.

But it is possible to create a chatbot supporting multiple language. We need to use Azure text translation service. Thus the steps would be:

1. User chats with bot in language X.
2. Use translation service to detect language of X.
3. If X is english, then call QnA maker.
4. If X is not english, translate text in english using translate service.
5. Call QnA, get answer, conver answer back to X language and send it to user.

Please use below useful links to achieve this:
https://desflanagan.ie/2016/10/25/build-a-multi-language-bot/
https://docs.microsoft.com/en-us/azure/cognitive-services/translator/reference/v3-0-reference

Answer 2

There is a good article in the documentation provided by Microsoft about this subject.

It says:

QnA Maker supports knowledge base content in many languages. However, each QnA Maker service should be reserved for a single language. The first knowledge base created targeting a particular QnA Maker service sets the language of that service. See here for the list of supported languages.

The language is automatically recognized from the content of the data sources being extracted. Once you create a new QnA Maker Service and a new Knowledge Base in that service, you can verify that the language has been set correctly.

So I'm sorry but yes, you are right when you say the following:

But as soon as a KB is assigned to one, an index (kbId) is created within the latter which defines the language of - I assume - all KBs that will be managed by this search service.

=> You have to create several QnA Maker Service in Azure, one for each language.

Link: https://docs.microsoft.com/en-us/azure/cognitive-services/QnAMaker/how-to/language-knowledge-base

Note: this link also provide a way to check the detected language for the search service

Answer 3

I faced the exact same challenge in implementing a QnA for three different languages.

There is an even more explicit mention in the documentation which says:

Language analyzers, once set, cannot be changed. Also, the language analyzer applies to all the knowledge bases in a QnA Maker service. If you plan to have knowledge bases in different language, you should create them under separate QnA Maker services.

Source: https://docs.microsoft.com/en-us/azure/cognitive-services/QnAMaker/overview/languages-supported

I also worried about the costs of having dedicated instances for each QnA. That's why I looked for a solution.

Most costs for QnA Maker coming from the fact that each instance has its own App Service Plan beneath and the azure portal doesn't allow you to point a new instance to an existing App Service Plan. However, you can do this via ARM template.

Here is my approach:

1. Create a new resource group
2. Create a new App Service Plan in that resource group (SKU: S1 as recommended)
3. Deploy my ARM template against that resource group (see deploy.ps1). Do this once for every parameter file (e.g. 3 different languages) and point every file to the same App Service Plan created in Step 2 (See parameter appServicePlanName).

Step 1 & 2 could also be archived by simply creating your first QnA Maker instance which creates a new resource group and app service plan you can point to in step 3.

In my sample scenario this results in 3 different QnA Maker instances running on the same App Service Plan saving most of the costs enforced by the limitation of the current service release, at least from UI perspective.

Disclaimer: I could not see why Microsoft did block this scenario from UI, so there is the chance that my solution is not supported for some reason. IMHO > If the API let's me do it, it should be supported.

Bonus: There is also a parameter file for using the free tier (only one per tenant) you can use for testing purpose. Here I mix my KBs not taking care of the language issue as it is just for quick and dirty testing.

Let me know if you're having trouble using my sample.

Answer 4

I encountered the same problem and - as the previous answers describe - confirm this restriction. When you assign multiple knowledge-bases to the same Azure service, the Analyzer-setting in the linked Azure Search service is taken from the first knowledge-base for the new index. I don't see a workaround for this.

The main problem is the Azure Search service - only one "free" SKU is allowed per tenant, so this enforces a paid instance per language. In my environment, this is a critical problem and cannot be afforded in many situations (e.g. usually you'd like to have separate instances for test & production).

From a technical perspective it is unclear to me why this restriction exists. There is an independent index per knowledge-base and you always query QnA Maker service for a specifc knowledge-base. So it should be possible to define the Analyzer configuration per index in the Azure service. I created a feature-request for this: https://feedback.azure.com/forums/34192--general-feedback/suggestions/38372725-qna-maker-support-knowledge-bases-for-different-l. Feel free to vote for it.

Answer 5

I don't know if you found a solution or not for it, but what I did is creating a new QnA maker service and merged the web app with the other QnA maker service plan, since the number of documents will split I was able to change the pricing tier of the QnA cognitive app to free, for search, I recommend leaving the search service as is, as per Microsoft the reply as follows:

To use multiple language and multiple knowledge bases, the user has to create an QnA Maker resource for each language. This will create a separate Azure search service per language. Mixing different language knowledge bases in a single Azure search service will result in degraded relevance of results.

to sum up, the pricing is pay as you go, so the extra bit now will be in the number of documents and the 10$ thing.