General starter Hadoop/ Spark fiware-cosmos questions

Question

I have some general questions about fiware-cosmos, apologies if they are basic but Im trying to understand the architecture and use of cosmos.

I saw that you are planning to integrate Apache Spark into Cosmos ? Do you have a road map or date for that to happen ? What happens if I want to use Spark now ?

What Hadoop service sources can be used ? I think I read that Cosmos supports Cloudera CDH services and raw Hadoop server services ? What about HortonWorks or MapR ?

I know that non standard file systems can be used with Hadoop, for instance MapR-FS, are options like this possible with Cosmos ?

I also read that Cosmos "sits" on top of fiware and so Hadoop as a service (HaaS) can be used and Hadoop clusters generated using open stack ? However, I saw that people are referring to a shared fiware cloud ? Does fiware run as a remote cloud ? Can a local cloud be used on a customer site ?

Is cosmos the only Apache Hadoop/Spark solution on fiware.org ?

Finally, if Cloudera CDH can be used with Cosmos how does the Cloudera cluster manager fit into the mix ? Can it still be used ?

Sorry for all of the questions :)

Answer 1

Cosmos is the name of the Global Instance of the Big Data GE in FIWARE Lab. It is a shared Hadoop instance already deployed in the cloud, ready to be used by FIWARE users.

In fact, there are two instances: The "old" one, serving a pretty old version of the Hadoop stack and being cosmos.lab.fiware.org its entry point. And the "new" one, which is a pair of Hadoop clusters, one for data storage and another one for data analysis; the entry points are storing.cosmos.lab.fiware.org and computing.cosmos.lab.fiware.org.

Of course, you can deploy any other Hadoop (or even Spark) instance by your own in the FIWARE cloud (or any other cloud, such as Amazon one).

Regarding Spark, since it was initially in our plans to deploy it in FIWARE Lab (that's why it appears in the roadmap), it is not clear nowadays it is gonna be deployed.