GCE persistent disk, kubernetes, and data persistence

Question

I'm having quite a bit of fun with the gcePersistentDisk in the context of pods inside of kubernetes:

Currently I'm attempting to get my rethinkdb cluster to work well with a gcePersistentDisk mounted volume in order to facilitate backups, data recovery, data integrity, etc. This is proving a bit more difficult than I originally anticipated. So, I have a few questions:

1: Should I even be attempting to use the gcePersistentDisk for this use case? Or should I be using persistentVolumes, and using the file system/persistentVolumes on my host kubelets in order to persist the data, and only using gcePersistentDisk when I'm doing a backup?

2: [EDIT: FIGURED OUT]

3: Pretty sure this is just a bug, but if you attempt to scale up a pod with a gcePersistentDisk mounted as a volume, it does not throw the usual:

'The ReplicationController "rethinkdb" is invalid:spec.template.spec.volumes.GCEPersistentDisk.ReadOnly: invalid value 'false': ReadOnly must be true for replicated pods > 1, as GCE PD can only be mounted on multiple machines if it is read-only.'

, but rather just hangs on the command line and loops forever when I view the kublet's logs.

4: Am I going completely in the wrong direction for solving this issue? And if so, how do I persist the DB data from my pods?

Answer 1

Unfortunately I don't know anything about rethinkdb, but it's very reasonable to use a gcePersistentDisk to store the data. That way if the Kubernetes node running your pod dies, the pod can be restarted on another node (assuming you have more than one node in your Kubernetes cluster) and continue to access the data when it comes back up. I don't think there's any reason you need to use persistent volumes here; straight-up GCEPersistentDisk as the VolumeSource should be fine.

I'm not sure why you're losing your data when you scale the RC down to 0 and back up to 1. My understanding is that the PD should be re-mounted.