Rancher Server (Docker Container) keeps restarting because of mysql

Question

I had a Rancher Server running for few months now and after a server reboot it keeps restarting because of a mysql error:

Rancher Container logs:

180719 16:51:17 [Note] Plugin 'FEDERATED' is disabled. /usr/sbin/mysqld: Can't find file: './mysql/plugin.frm' (errno: 13) 180719 16:51:17 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it. 180719 16:51:17 InnoDB: The InnoDB memory heap is disabled 180719 16:51:17 InnoDB: Mutexes and rw_locks use GCC atomic builtins 180719 16:51:17 InnoDB: Compressed tables use zlib 1.2.8 180719 16:51:17 InnoDB: Using Linux native AIO 180719 16:51:17 InnoDB: Initializing buffer pool, size = 128.0M 180719 16:51:17 InnoDB: Completed initialization of buffer pool 180719 16:51:17 InnoDB: Operating system error number 13 in a file operation. InnoDB: The error means mysqld does not have the access rights to InnoDB: the directory. InnoDB: File name ./ibdata1 InnoDB: File operation call: 'create'. InnoDB: Cannot continue operation.

I couldn't find the ibdata1 file anywhere in the container or the host, how should we go with it without losing the existing rancher config ?

Answer 1

I am not sure about the root cause of the problem but here are some pointers to recover the setup:

1a) Copy the data from rancher/server container. Here is the link where you will find instructions: https://rancher.com/docs/rancher/v1.6/en/upgrading/#single-container

1b) Start the same version of rancher with the copied out data volume instead of upgrading as pointed out in the instructions.

2a) If the existing container is running, you can copy out the mysql data from inside the container.

host# mkdir -p mysql_data_recovery && cd mysql_data_recovery
host# docker cp rancher_server_container_id:/var/lib/mysql .

2b) Start a new rancher/server with mounting this directory.