Solr 4.1 not processing Added documents on delta-import

Question

I have the following in my data-config.xml:

<dataConfig>
  <dataSource dataSource info"/>
  <document>
   <entity name="item"
        pk="itemid"
        query=" SELECT itemid,start,end,item_categoryid
                FROM item"
        deltaImportQuery="  SELECT  * FROM item
                            WHERE   itemid = '${dataimporter.delta.item_id}'"
        deltaQuery="    SELECT  itemid
                        FROM    item
                        WHERE   last_mod > '${dataimporter.last_index_time}'
                            OR  create_time> '${dataimporter.last_index_time}'">
        <field column="itemid" name="item_id"/>
        <field column="start" name="start"/>
        <field column="end" name="end"/>
        <entity name="item_category" 
                pk="item_categoryid"
                query=" SELECT  item_categoryid,desc,title 
                        FROM    item_category
                        WHERE   mbpostingid='${item.item_categoryid}'"
                deltaImportQuery="  SELECT  * FROM item_category
                                    WHERE   item_categoryid= '${dataimporter.delta.id}'"
                deltaQuery="SELECT  item_categoryid
                            FROM    item_category
                            WHERE   last_mod > '${dataimporter.last_index_time}'
                                OR  create_time > '${dataimporter.last_index_time}'">
           <field column="item_categoryid" name="id"/>
           <field column="desc" name="desc"/>
           <field column="title" name="title"/>
        </entity>
    </entity>
  </document>
</dataConfig>

Whenever I run a delta-import

http://localhost:8983/solr/dataimport?command=delta-import

Solr is updating rows that have been updated in my database but it is not adding indexes for those that have been added and it is not deleting indexes for those that have been deleted.

Solr recognizes (fetches) my newly added rows to my database but it doesn't "process" them

<str name="Total Documents Processed">0</str>

Is there anything wrong with my dataConfig? Is there anything I can do?

Thanks, Caleb

Answer 1

query="SELECT itemid,start,end,item_categoryid
            FROM item"
deltaImportQuery="SELECT  * FROM item
                  WHERE   itemid = '${dataimporter.delta.item_id}'"
                                                         ^-----^

We have to use the exact column name in the variable. It looks to me like the column name is itemid (with no underscore), so you should give this a try:

query="SELECT itemid,start,end,item_categoryid
            FROM item"
deltaImportQuery="SELECT  * FROM item
                  WHERE   itemid = '${dataimporter.delta.itemid}'"
                                                         ^----^

Answer 2

try

deltaImportQuery="SELECT  * FROM item
                  WHERE   itemid = '${dih.delta.item_id}'"

http://wiki.apache.org/solr/DataImportHandler says:

deltaImportQuery : (Only used in delta-import) . If this is not present , DIH tries to construct the import query by(after identifying the delta) modifying the 'query' (this is error prone). There is a namespace ${dih.delta.column-name} which can be used in this query. e.g: select * from tbl where id=${dih.delta.id}

Answer 3

<entity name="searchitemcontent" pk="itemid" 
        query="select skuid, noofsupp, membership, isnull(negotiable,0) as negotiable from searchskumaster skm "
        deltaImportQuery="select skuid, skuname, combineskuname,Category, Product, Brand, Price, discount, url, image, itemstatus , supplierid,itemid, marketprice,mwsimage, pricestatus, isservice ,suppliername, noofsupp, membership, isnull(negotiable,0) as negotiable from
        searchskumaster skm where itemid='${dih.delta.itemid}'"
        deltaQuery="select itemid  from searchskumaster where modifieddate &gt; '${dataimporter.last_index_time}'"
        deletedPkQuery="SELECT itemid FROM searchskumaster where isdeleted=1 and modifieddate &gt; '${dataimporter.last_index_time}' ">

Note: SELECT itemid . in deltaQuery and deltaImportQuery query, itemid='${dih.delta.itemid} , in both query should itemid be same name