MongoDb text search with language support

Question

I have problems with the language text search of MongoDB. For some records the search works great and for some records it does not work at all.

I have a list of ingredients that I would like to search. The ingredients are in several languages and I like to take care of singular and plural.

Here is my example

Schema

{
  translation: [
    {
      language: {
        type: String,
        required: true
      },
      name: {
        type: String,
        required: true
      }
    }
  ],
  calories: {
    "type": Number
  },
  protein: {
    "type": Number
  },
  carbohydrate: {
    "type": Number
  },
  fat: {
    "type": Number
  }
}

Index

foodSchema.index( { "translation.name": "text" }, { default_language: "german" } )

Read Index from DB

[
    {
        "v" : 2,
        "key" : {
            "_id" : 1
        },
        "name" : "_id_"
    },
    {
        "v" : 2,
        "key" : {
            "_fts" : "text",
            "_ftsx" : 1
        },
        "name" : "translation.name_text",
        "default_language" : "german",
        "background" : true,
        "weights" : {
            "translation.name" : 1
        },
        "language_override" : "language",
        "textIndexVersion" : 3
    }
]

Records

{
  calories: 1,
  protein: 2,
  carbohydrate: 3,
  fat: 4,
  translation: [
    {
      _id: ObjectId('5fba87d13ad6404108191670'),
      language: 'german',
      name: 'gurke'
    },
    {
      _id: ObjectId('5fba87d13ad6404108191671'),
      language: 'english',
      name: 'cucumber'
    },
    {
      _id: ObjectId('5fba87d13ad6404108191672'),
      language: 'spanish',
      name: 'pepino'
    }
  ]
}

// ----

{    
  calories: 4,
  protein: 3,
  carbohydrate: 2,
  fat: 1,
  translation: [
    {
      _id: ObjectId('5fba87d13ad6404108191674'),
      language: 'german',
      name: 'huhn'
    },
    {
      _id: ObjectId('5fba87d13ad6404108191675'),
      language: 'english',
      name: 'chicken'
    },
    {
      _id: ObjectId('5fba87d13ad6404108191676'),
      language: 'spanish',
      name: 'pollo'
    }
  ]
}

Searching data

db.getCollection('foods').find({$text: { $search: "gurke" }}) //works
db.getCollection('foods').find({$text: { $search: "gurken" }}) //works
db.getCollection('foods').find({$text: { $search: "cucumber" }}) //works
db.getCollection('foods').find({$text: { $search: "cucumbers" }}) //works
db.getCollection('foods').find({$text: { $search: "huhn" }}) //works
db.getCollection('foods').find({$text: { $search: "hühner" }}) //works
db.getCollection('foods').find({$text: { $search: "chicken" }}) // no result
db.getCollection('foods').find({$text: { $search: "chickens" }}) //no result
db.getCollection('foods').find({$text: { $search: "pepino" }}) //no result

The documentation from MongoDb says: https://docs.mongodb.com/manual/tutorial/specify-language-for-text-index/

The default language associated with the indexed data determines the rules to parse word roots (i.e. stemming) and ignore stop words.

Does it means that only the default language is supported?
Why is it working for cucumber but not for chicken?

I was also checking the stop words for any chicken. https://github.com/mongodb/mongo/blob/master/src/mongo/db/fts/stop_words_english.txt

Thank you for your help!

Minsky Minsky · Accepted Answer · 2020-11-22T17:06:16

The problem isn't the index, that's correct, but you need to add $language or it uses the default language (at least when using $text). Try

 db.collection.find({$text:{$search:"pollo", $language:"spanish"}})

$language Docs

If not specified, the search uses the default language of the index.

Also, if you run

 db.collection.find({$text:{$search:"pollo"}}).explain()

You'll find out the query is using the default language.

MongoDb text search with language support

1 Answers