SOLR indexing SQL data per language /multi language

0
votes

I know that there are similar questions about SOLR, some give insights but not a solution for exaclty what I am trying to do.

I would like to create one core having multi language data. For exemple it is possible to have field like description_fr, description_en. I would like to send description fr when the request whant the data in french and do not send the description_en.

Some of my questions :

  1. How do I define the data to be indexed
  2. How do I tell the application to request the search against the English or French version of the fields ?

Thanks a lot

1
indexingsolrmultilingual
How are you indexing data (through DIH?) - What have you tried? Have you looked at Detecting languages during indexing? - MatsLindh
Yes, I index my data by DIH. Per line I have a data having the both languages. For example I have a row with the following columns - ID, NAME, DESCRIPTION_FR, DESCRIPTION_EN. I want to send all columns to the user depending to its language - ID, NAME, DESCRIPTION_EN if the language is EN and do not send DESCRIPTION_EN in this case. I saw the detection languages, but I do not clearly understand how to use and whether this is what is I should do. I must to try - Vicking
Now looking for the LangDetectLanguageIdentifierUpdateProcessorFactory I am not sure if I can use it because langid.langField require one field for the detection and my row contains the both languages. Do you know if I can use it anyway ? - Vicking
If you have a row which already have both languages separated, you don't need to do anything special except set up separate processing for the fields based on the language. See Language Analysis. - MatsLindh
You are reason, but how I can configure the set up to separate the fields based on the language. In my case as explain here below the API call Solr with a key common for the both languages (example marque of product) and after we send this common column with all other columns that are language specifique. If you have some link as example of configuration, I will be very grateful - Vicking

1 Answers

0
votes

I would recommend a talk from a friend of mine at the latest Berlin Buzzwords [1]

This could be interesting to you for the future.

Sticking to your current question I would proceed identifying the language of the query ( which is an hard task as the query is usually composed by few terms). Then, depending on the language I would sent to Solr a request to return only 1 stored field for the content.

e.g.

In the index I have :

description_it, description_en

q = "prodotto scalare" Language Identified : it

Request : http://localhost:8983/solr/select?q=prodotto scalare&fl=description_it

You just need a library to detect the language[2] and a mapping between the language ISO code and your solr fields. You can build this at API time or directly in Solr as a plugin.

[1] https://berlinbuzzwords.de/sites/berlinbuzzwords.de/files/media/documents/embracing_diversity_searching_over_multiple_languages.pdf

[2] A couple of popular examples : Tika - https://www.programcreek.com/java-api-examples/index.php?api=org.apache.tika.language.LanguageIdentifier Google - https://github.com/shuyo/language-detection