Insert geograpic data in Elastic Search from Spark

0
votes

I try to upload an RDD with a latitude and a longitude fields in my ES. I would like to use the geo_point type to plot them on a map. I tried to create a "location" field for each document containing either a string like "12.25, -5.2" or a array of two doubles for lat/long but ES does not detect them as a geo_point. The index does not exist before I insert data.

How can I tell ES that location is a geo_point?

Current code with the elasticsearch-hadoop lib to store:

myRDD.saveToEs(indexName, someConf)

with myRDD an RDD[Map] containing a "location" -> [double, double]

and someConf contains "es.index.auto.create" -> "true"

1
scalaelasticsearchapache-sparkelasticsearch-hadoop
you should add the mapping properly for your index before you go, index auto creation is not a good idea in this case – innovatism
How can i do that? – Benjamin
don't know what your index like but you should put something like : PUT my_index { "mappings": { "my_type": { "properties": { "location": { "type": "geo_point" } } } } } – innovatism
Yes, but that raise two questions: 1) can i send partial information for this index (only the location type) and let ES try for other fields. 2) how to do the PUT in scala – Benjamin
1) yes, for every field that has not been defined, ES will auto create it. Might not be what you want, but it will do it. 2) use any http library such as play-ws or execute a curl command (not recommended) – elmalto

1 Answers

1
votes

A working solution as suggested in comments:

  1. HTTP PUT to create an index with a mapping only this particular field
  2. Insert the RDD normally

If you have inserted data before it is too late