How to do Solr Dataimport (i.e from RDBMS) using Java API?

Question

I am new to Solr, actually I tried Database table indexing in Solr manually, i.e creating a new data-config.xml and running the full import from Solr webUI, successfully did that.

But now I need to do the same thing in Java. So I need know the following things using Java:

How to set Solr datasource in Java api
How to set the entity and query
How to run full import

Actually, for data import we write a configuration file like below

<?xml version="1.0" encoding="UTF-8" ?>
<dataConfig>
<dataSource type="JdbcDataSource" 
            driver="com.mysql.jdbc.Driver"                url="jdbc:mysql://localhost:3306/test_db" 
            user="root" 
            password="cloudera"/>
<document>
  <entity name="emp"  
    query="select id,name from emp">
     <field column="id" name="id"/>
     <field column="name" name="name"/>       
  </entity>
</document>
</dataConfig>

and this configuration file information we will provide in solrconfig.xml like below

<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
        <lst name="defaults">
            <str name="config">db-data-config.xml</str>
        </lst>
</requestHandler>

But my requirement is, I don't want to configure any XML file like above, just I want do it all from Java only, so I need to set all the configurations which I have given in above XML from Java code itself, i.e something like below

 ModifiableSolrParams params = new ModifiableSolrParams();
    params.set("Datasource","JdbcDataSource") 
    params.set("driver","com.mysql.jdbc.Driver")
    params.set("url","jdbc:mysql://localhost:3306/test_db")
    params.set("user","cloudera")
    params.set("password","cloudera")
    params.set("query","select * from emp"),etc.

This is the third time I am posting this question but no one giving me exact solution or just tell me, is it possible or not?

If you have asked the question three times and no-one has given you the exact solution, maybe it's time to either dig into the source yourself (and make a patch if necessary) or realize that it probably isn't possible? — MatsLindh
I'm not sure if this is possible from SolrJ alone. It may be possible from Spring Data Solr (never tried it so can't confirm). Here's a pretty good link to check out: petrikainulainen.net/programming/maven/running-solr-with-maven — TMBT

Marcelo Marcelo · Accepted Answer · 2017-11-28T11:40:47

A little late answer, but it is possible, at least with Solr 7.1 (did not try previous version).

If you check at the http://yoursolrhostsolr/#/yourcore/dataimport//dataimport there is the possibility of executing an import with a custom data-config, by clicking on the debug mode.

By looking at the logs, you can see what commands the admin interfaces sends to solr. It calls the /dataimport handler passing a dataConfig.xml file as an argument.

So it can be done like this with SolrJ:

ModifiableSolrParams params = new ModifiableSolrParams();
params.set("qt", "/dataimport");
params.set("command", "full-import");
params.set("dataConfig", <the data-import xml as string>);
solrClient.query(params);

By passing a dataConfig, the dataImportHandler ignores the configured data-import.xml

for those using solr 8.2.0 the link is http://yoursolrhostsolr/#/yourcore/dataimport//import

How to do Solr Dataimport (i.e from RDBMS) using Java API?

1 Answers