Solr indexing tsv file with some fields missing

Question

I am trying to index a TSV file in Solr and weirdness with the TSV files I have is, they miss some fields in some lines randomly.

Sample data in my TSV

0 abc 1 The quick brown fox jumps over the lazy dog 
0 abc 1 2 The quick brown fox jumps over the lazy dog 
0 abc 1 2 The quick brown fox jumps over the lazy dog 
0 abc 1 2 3 The quick brown fox jumps over the lazy dog

headers will be something like this

id name num_1 num_2 num_3 description

in line 1 - num_2 and num_3 are not present

in line 2 - num_3 is not present

in line 3 - all are present

How can I handle this file to index in Solr ? Can something like this be handled seamlessly with Solr or do I need to pre process the TSV and handle the missing fields for Solr to index them ?

I get below exception when trying to load this file to solr

<response>
<lst name="responseHeader"><int name="status">400</int><int name="QTime">145</int></lst><lst name="error"><str name="msg">CSVLoader: input=null, line=23,expected 8 values but got 6</str><int name="code">400</int></lst>
</response>

Update: @Fuu Thanks for response.

Another way to put the question: Does Solr provide any capabilities to understand the fields when indexing ? or edit the field before indexing ?

Say, my sample documents looks like

token:n=1 token:name=abc token:num:a=1 token:num:b=2 token:num:c=3 token:desc=...

Is it possible to edit that row to

1 abc 1 2 3 ....

before actually Solr indexing it ?

Fuu Fuu · Accepted Answer · 2015-05-12T21:05:38

I don't think there is any magic within the Solr to figure out which specific fields you are missing. Certainly that magic is not described in the UpdateCSV API. From Solr perspective, all it can deduce is that some fields are not there and throw an error of the length mismatch. So, as in your case, you are only missing fields 4 or 5, that would not work anyway. You are the only one who can know the logic of which fields data is missing.

So, best approach for you would be to do pre-processing as you already mentioned in your question. You can either fix the files before indexing them with Solrs CSV indexer, or use indexing script that fixes the lines on the fly before committing to Solr. Both aprroaches would work equally well.

Addition for the comment and edited question:

Solr does provide a DataImportHandler that can be used for pre-processing of the input data. See https://wiki.apache.org/solr/DataImportHandler for documentation of its features. However, while it is quite flexible and includes features such as number format parsing, I believe it's not possible to achieve exactly what you describe using the DIH.

I see two ways to pre-process the data to fit your use case:

Add placeholder values or pad the rows with extra tabs for the missing values, then import the TSV. The resulting documents in Solr are predictable, where the "missing value" is always the same.
Convert the TSV into another format, such as JSON where you can just drop the keys without values from the pre-processed document.

Solr indexing tsv file with some fields missing

1 Answers