Solr 4.2.1 - Indexing pipe seperated file using DataImportHandler - 2 Issues

Question

I am new to Solr and working on a solr POC. I searched the StackOverflow for a similar issue but couldnot find one I am trying to use Solr 4.2.1 to index a text file containing pipe (|) seperated data. The following is snippet of sample data

cust_id|name1|name2|name3|dob|address|city|pincode|phone|idenfication|salary
1001000003|John|D|Doe|31081962|H-904, Green Mandion, M G Rd,   Santacruz(east)|mumbai|400056|9812030334|AMXPT7702P|50000.56
1001000005|Bob||Taylor|1041982|210, Greek Heights, Khar|mumbai|400057|976130321|AAXZZ2103P|20000.65

I am using the dataimporthandler to import the data into Solr

I have two issues

1. When I do a select query

I get following response

{
'responseHeader'=>{
'status'=>0,
'QTime'=>0},
'response'=>{'numFound'=>3,'start'=>0,'docs'=>[
{
'cust_id'=>'cust_id|name1|name2|name3|dob|address|city|pincode|phone|idenfication|salary'},
{
'cust_id'=>'1001000003|John|D|Doe|31081962|H-904, Green Mandion, M G Rd, Santacruz(east)|mumbai|400056|9812030334|AMXPT7702P|50000.56'},
{
'cust_id'=>'1001000005|Bob||Taylor|1041982|210, Greek Heights, Khar|mumbai|400057|976130321|AAXZZ2103P|20000.65'}]
}}

How do I get this into column:value and not as a string of data, I mean

{
'responseHeader'=>{
'status'=>0,
'QTime'=>0},
'response'=>{'numFound'=>3,'start'=>0,'docs'=>[

{

'cust_id'=>'1001000003',
'name1' => 'John',
'name2' => 'D',
......
......
'salary' => 50000.56
}

,
{
'cust_id'=>'1001000005,
'name1' => 'Bob'
....
'salary' => 20000.65
}]
}}

My config file is as follows

<dataConfig>
<dataSource name="dfs" encoding="UTF-8" type="FileDataSource" />
<document>
<entity name="sourcefile"
processor="FileListEntityProcessor"
newerThan="${dataimporter.last_index_time}"
fileName="sample.txt"
rootEntity="false"
baseDir="C:/mfi_data/"
header=true
>

<entity name="entryline"
processor="LineEntityProcessor"
url="${sourcefile.fileAbsolutePath}"
rootEntity="true"
dataSource="dfs"
separator="|"
transformer="RegexTransformer"

>
<field column="rawLine"
regex="^(.*)|(.*)|(.*)|(.*)|(.*)|(.*)|(.*)|(.*)|(.*)|(.*)|(.*)$"
groupNames="cust_id,name1,name2,name3,dob,address,city,pincode,phone,idenfication,salary"
/>


</entity>
</entity>


</document>
</dataConfig>

My schema.xml

<?xml version="1.0" encoding="UTF-8" ?>
<schema version="1.5">
<fields>
<field name="cust_id" type="string" indexed="true" stored="true" />
<field name="name1" type="string" indexed="true" stored="true" />
<field name="name2" type="string" indexed="true" stored="true" />
<field name="name3" type="string" indexed="true" stored="true" />
<field name="dob" type="string" indexed="true" stored="true" />
<field name="address" type="string" indexed="true" stored="true" />
<field name="city" type="string" indexed="true" stored="true" />
<field name="pincode" type="int" indexed="true" stored="true" />
<field name="phone" type="string" indexed="true" stored="true" />
<field name="identification" type="string" indexed="true" stored="true" />
<field name="salary" type="float" indexed="false" stored="true" />
<field name="rawLine" type="text" indexed="false" stored="false" multiValued="true" />

</fields>
<uniqueKey>cust_id</uniqueKey>
<types>
<fieldType name="string" class="solr.StrField" />
<fieldType name="int" class="solr.TrieIntField" />
<fieldType name="text" class="solr.TextField" />
<fieldType name="float" class="solr.FloatField" />
</types>
</schema>

1. How do I remove header from being considered as data to be indexed? I tried Header="true" in the dataConfig but thats not working

Please guide if you have encountered a way around this, thanks in advance?

Answer 1

Solr accepts index updates in CSV (Comma Separated Values) format. Different separators and escape mechanisms are configurable, and multi-valued fields are supported. http://wiki.apache.org/solr/UpdateCSV

separator

Specifies the character to act as the field separator. Default is separator=,

header true if the first line of the CSV input contains field or column names. The default is header=true. If the fieldnames parameter is absent, these field names will be used when adding documents to the index.