SOLR indexing arbitrary data

Question

Let's say you have a simple forms automation application, and you want to index every submitted form in a Solr collection. Let's also say that form content is open-ended so that the user can create custom fields on the form and so forth.

Since users can define custom forms, you can't really predefine fields to Solr, so we've been using Solr's "schema-less" or managed schema mode. It works well, except for one problem.

Let's say a form comes through with a field called "ID" and a value of "9". If this is the first time Solr has seen a field called "ID", it dutifully updates it's schema, and since the value of this field is numeric, Solr assigns it a data type of one of it's numeric data types (we see "plong" a lot).

Now, let's say that the next day, someone submits another instance of this same form, but in the ID field, they type their name instead of entering a number. Solr spits this out and won't index this record because the schema says ID should be numeric, but on this record, it's not.

The way we've been dealing with this so far is to trap the exception we get when a field's data type disagrees with the schema, and then we use the Solr API to alter the schema, making the field in question a text or string instead of a numeric.

Of course, when we do this, we need to reindex the entire collection since the schema changed, and so we need to persist all the original data just in case we need to re-index everything after one of these schema data-type collisions. We're big Solr fans, but at the same time, we wonder whether the benefits of using the search engine outweigh all this extra work that gets triggered if a user simply enters character data in a previously numeric field.

Is there a way to just have Solr always assign something like "text_general" for every field, or is there some other better way?

Answer 1

I would say that you might need to handle the Id values at your application end. It would be good to add a validation for Id, that Id should be of either string or numberic. This would resolve your issue permanently. If this type is decided you don't have to do anything on the solr side.

The alternative approach would be have a fixed schema.xml. In this add a field Id with a fixed fieldType. I would suggest you to go with string as a fieldType for ID if don't want it to tokenize the data and want the exact match in the search.

If you would like to have flexibility in search for the Id field then you can add a text_general field type for the field.

You can create your own fieldType as well with provided tokenizer and filter according to your requirement for you the field Id.

Also don't use the schemaless mode in production. You can also map your field names to a dynamic field definition. Create a dynamic field such as *_t for the text fields. All your fields with ending with _t will be mapped to this.