How to use Docker volumes as persistent volumes for Mesos/Marathon?

Question

I'm running Mesos and Ceph clusters on CoreOS with a working Ceph RBD Docker volume plugin, but it's very unclear to me how this can be used with Mesos/Marathon... Creating/using rbd volumes for single Docker containers is working flawlessly though.

I can find no article/blog post/whatever that deals with the automated creation (and, in case of "task migration" between Mesos slaves, remapping) of these volumes via Marathon. Especially important to me is how to run multiple instances of a stateful service when each instance needs to have it's own volume (imagine a MongoDB ReplicaSet on Mesos/Marathon).

I know the Mesos persistent volume docs, and I also saw the Marathon issue, but still I'm very confused how or when this will be usable...

There are also other questions here on SO:

which unfortunately don't really have an answer to this specific problem.

The EMC Code example with RexRay also only covers a single instance example, which I could also handle with ease with the volume plugin mentioned above:

{
    "id": "nginx",
    "container": {
        "docker": {
            "image": "million12/nginx",
            "network": "BRIDGE",
            "portMappings": [{
                "containerPort": 80,
                "hostPort": 0,
                "protocol": "tcp"
            }],
            "parameters": [{
                "key": "volume-driver",
                "value": "rbd"
            }, {
                "key": "volume",
                "value": "nginx-data:/data/www"
            }]
        }
    },
    "cpus": 0.2,
    "mem": 32.0,
    "instances": 1
}

The nginx-data volume would be created automatically in this case. But what if I want to use persistent volumes and multiple instances?

Luke Marsden Luke Marsden · Accepted Answer · 2016-01-26T12:12:40

This is a use case that Flocker is meant to solve. (Disclaimer: I'm the CTO at ClusterHQ). See this blog post for a demo of the Flocker <=> Mesos/Marathon interaction, which shows how the Flocker Control Service can act as the "source of truth" for which container volumes exist in a clustered setting. Flocker will then create on-demand, and then co-ordinate mapping and unmapping these volumes between hosts as the containers that reference these volumes move around in the cluster.

Flocker does this by providing a cluster-wide namespace of volume names, these names can then be used via the Flocker plugin for Docker with Marathon to provide portability and high availability for stateful containers in a Mesos cluster.

Flocker also has a Ceph driver:

Google "Flocker Ceph Driver"

And works on CoreOS:

Google "Flocker on CoreOS demo"

You can run multi-instance jobs (like MongoDB with replica sets) by giving each container its own volume name (like mongo_1, mongo_2, etc).

Putting these pieces together would be non-trivial, but I'd be happy to help. I could write up a detailed guide specifically for your stack (Ceph + CoreOS + Docker + Mesos + Marathon) if you like.

How to use Docker volumes as persistent volumes for Mesos/Marathon?

1 Answers