Trouble with Chinese character in Solr DataImport

Question

I am having trouble indexing Chinese/Japanese texts in Solr 3.4. I am importing the data using DIH, the connection block is

<dataSource type="JdbcDataSource"
    driver="com.mysql.jdbc.Driver"
    url="jdbc:mysql://localhost/db_development?useUnicode=true&amp;characterEncoding=UTF-8&amp;characterSetResults=UTF-8"
    user="user"
    useUnicode="true"
    characterEncoding="UTF-8"
    encoding="UTF-8"
    password="password"
    zeroDateTimeBehavior="convertToNull"
    name="app" />

The fieldtype defn of this field goes as

  <fieldType name="text_commongrams" class="solr.TextField">
    <analyzer>
      <charFilter class="solr.HTMLStripCharFilterFactory" />
      <tokenizer class="solr.ICUTokenizerFactory" />
      <filter class="solr.ICUTransformFilterFactory" id="Traditional-Simplified"/>
      <filter class="solr.ICUFoldingFilterFactory"/>
      <filter class="solr.ASCIIFoldingFilterFactory"/>
      <filter class="solr.ICUNormalizer2FilterFactory" name="nfkc_cf" mode="compose"/>
      <filter class="solr.RemoveDuplicatesTokenFilterFactory" />
      <filter class="solr.TrimFilterFactory" />
      <filter class="solr.LowerCaseFilterFactory" />
      <filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/>
    <filter class="solr.SynonymFilterFactory"
      synonyms="synonyms.txt"
      ignoreCase="true"
      expand="true" />
    <filter class="solr.CommonGramsFilterFactory"
      words="stopwords_en.txt"
      ignoreCase="true" />
    <filter class="solr.StopFilterFactory"
      words="stopwords_en.txt"
      ignoreCase="true" />
    <filter class="solr.WordDelimiterFilterFactory"
      generateWordParts="1"
      splitOnNumerics="0"
      generateNumberParts="1"
      catenateWords="1"
      catenateNumbers="1"
      catenateAll="0"
      preserveOriginal="1" />
  </analyzer>
</fieldType>

MySQL character encoding details are as

+--------------------------+-----------------------------------------+
| Variable_name            | Value                                   |
+--------------------------+-----------------------------------------+
| character_set_client     | latin1                                  |
| character_set_connection | latin1                                  |
| character_set_database   | latin1                                  |
| character_set_filesystem | binary                                  |
| character_set_results    | latin1                                  |
| character_set_server     | utf8                                    |
| character_set_system     | utf8                                    |
| character_sets_dir       | /opt/local/share/mysql5/mysql/charsets/ |
+--------------------------+-----------------------------------------+

I am starting Solr with the java param -Dfile.encoding=UTF-8.

The input Text is JavaOne Tokyo 2012での発表スライド When I import it into Solr, and query for that document using ID, I see the text as JavaOne Tokyo 2012ã§ã®ç™ºè¡¨ã‚¹ãƒ©ã‚¤ãƒ‰

Can anyone tell me where I am going wrong?

pr4n pr4n · Accepted Answer · 2012-10-10T06:11:15

So I finally had to alter my MySQL table to store the strings in UTF8. Details of how to convert an existing table from latin1 to utf8 could be found here.

Trouble with Chinese character in Solr DataImport

1 Answers