MongoDB language_override text search

Question

I have a MongoDB database with documents that may be in different languages. Each document contains a "language" field indicating the language of the document.

Example:

{ "content" : "This document is English", "language" : "en" }
{ "content" : "Ce document est français", "language" : "fr" }

But when I do a search, Mongo does not appear to be using the "language" field even though the index shows "language_override" : "language", it seems to only be using the "default_language" of the index.

I am new to Mongo, so I might just be doing something totally dumb here. I have tried Mongo 2.6.10 and 3.4.1.

I create 2 databases, each with a single French document the first one has the "default_language" : "english", the second database uses "default_language" : "fr". When I specify the "default_language" for the index to be English, the document is not found unless I specify the language explicitly.

> use lang_test_
switched to db lang_test_1
> db.createCollection("docs")
{ "ok" : 1 }
> db.docs.insert({ "content" : "le téléchargement ou le chargement de données.", "language" : "fr" } )
WriteResult({ "nInserted" : 1 })
> db.docs.createIndex({ content : "text" })
{
    "createdCollectionAutomatically" : false,
    "numIndexesBefore" : 1,
    "numIndexesAfter" : 2,
    "ok" : 1
}
> db.docs.getIndexes()
[
    {
        "v" : 1,
        "key" : {
            "_id" : 1
        },
        "name" : "_id_",
        "ns" : "lang_test_1.docs"
    },
    {
        "v" : 1,
        "key" : {
            "_fts" : "text",
            "_ftsx" : 1
        },
        "name" : "content_text",
        "ns" : "lang_test_1.docs",
        "weights" : {
            "content" : 1
        },
        "default_language" : "english",
        "language_override" : "language",
        "textIndexVersion" : 2
    }
]
> db.docs.find({ $text : { $search: "chargement" }}).count()
0
> db.docs.find({ $text : { $search: "chargement", $language: "fr" }}).count()
1
> db.docs.find({ $text : { $search: "chargement", $language: "en" }}).count()
0
> db.docs.find({ $text : { $search: "chargement", $language: "none" }}).count()
0

Now with a default language of French

> use lang_test_2
switched to db lang_test_2
> db.createCollection("docs")
{ "ok" : 1 }
> db.docs.insert({ "content" : "le téléchargement ou le chargement de données.", "language" : "fr" } )
WriteResult({ "nInserted" : 1 })
> db.docs.createIndex({ content : "text"}, {default_language: "fr" })
{
    "createdCollectionAutomatically" : false,
    "numIndexesBefore" : 1,
    "numIndexesAfter" : 2,
    "ok" : 1
}
> db.docs.getIndexes()
[
    {
        "v" : 1,
        "key" : {
            "_id" : 1
        },
        "name" : "_id_",
        "ns" : "lang_test_2.docs"
    },
    {
        "v" : 1,
        "key" : {
            "_fts" : "text",
            "_ftsx" : 1
        },
        "name" : "content_text",
        "ns" : "lang_test_2.docs",
        "default_language" : "fr",
        "weights" : {
            "content" : 1
        },
        "language_override" : "language",
        "textIndexVersion" : 2
    }
]
> db.docs.find({ $text : { $search: "chargement" }}).count()
1
> db.docs.find({ $text : { $search: "chargement", $language : "fr" }}).count()
1
> db.docs.find({ $text : { $search: "chargement", $language : "en" }}).count()
0
> db.docs.find({ $text : { $search: "chargement", $language : "none" }}).count()
0

s7vr s7vr · Accepted Answer · 2017-01-15T16:57:41

The default language in the index is the language used for parsing for both search keyword and document content when they don't have the language field.

In short, you've to specify the language for the search keyword if you want to be parsed in language other than default language, similar to how you are specifying language for the each document.

So, in your first case when you don't specify the language in the search keyword, the text is parsed in default language (english) and the document has a content parsed in french, so the search fails.

Now, when you add the language to the search keyword it gets parsed in french and you find a matching record because the content you're searching is also parsed in french.

For original source and more information, refer this ticket.

https://jira.mongodb.org/browse/SERVER-13238

MongoDB language_override text search

Now with a default language of French

1 Answers