AWS-EC2 Redis-server RDB snapshot write error

Question

I have a web application running on Laravel5.2 framework, with session driver set to redis with following AWS setup.

• Instance-1: Running web application, with Redis configurations in .env file as follow

  1. Redis-host: aws-private-ip-of-instance-2
  2. Redis-password: NULL
  3. Redis-port: 6379

• Instance-2: Redis-server running with following configuration

  1. Bind aws-private-ip-of-instance-2 and 127.0.0.1
  2. Working directory /var/lib/redis with 775 permission, and ower-group is redis.
  3. RDB snapshot name dump.rdb with 660 permission, and ower-group is redis.

NOTE: In AWS inbound rule for port 6379 is configured for Instance-2.

Everything works fine, until redis tries to write the data on the RDB file. Following error shows on front-end.

MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.

While in the logs of Redis server i got following data.

4873:M 23 Sep 10:08:15.028 * 1 changes in 900 seconds. Saving...
4873:M 23 Sep 10:08:15.028 * Background saving started by pid 7392
7392:C 23 Sep 10:08:15.028 # Failed opening .rdb for saving: Read-only file system
4873:M 23 Sep 10:08:15.128 # Background saving error

Things I have tried

• Add vm.overcommit_memory = 1 to /etc/sysctl.conf, as suggested in Redis-administraition-blog
• Change path to dump.rdb file to tmp folder and change permissions to 777.

Answer 1

This other Stack Exchange thread might help, since you are using a custom /tmp dir for data:

The simple way to do this is to run systemctl edit redis. This will create an override drop-in file /etc/systemd/system/redis.service.d/override.conf, in which you can place your changes (and the proper section):

[Service] ReadWriteDirectories=-/my/custom/data/dir

You may also create that directory and place files ending in .conf in it manually. But do not leave the directory empty, as this will disable the service.

In either case, run systemctl daemon-reload and you are ready to restart your service.

Many threads also point to filesystem inconsistency as root cause. Since you are using EC2, check this AWS forums post:

To fix this, you will have to:

1. Stop the instance

2. Detach the root volume of your instance

3. Attach the volume as a data volume to any running Linux instance in the same availability zone

4. Perform a filesystem check (fsck) on the volume and fix the issues

5. Detach the volume and attach it back to your instance as it's root volume

6. Boot back instance and verify if the volume was able to mount successfully

As a last resort, terminate the instance if possible.

Hope it helps!

Answer 2

Well this is very embarrassing to post answer of own question, which was a really stupid mistake. But hope new folks here learns from my mistake too.

• So first thing I have done is enable detail logs for redis-server in /etc/redis/redis.conf file by changing log_level option to debug.

• Observe the logs and understand that my redis port 6379 was open for everyone on internet.

So from logs I observe that someone else's server is spoofing into my redis server and making it slave of it. And as my redis server is configure in a way that slave is read-only, when i try to access my redis-server it throw error of read-only.

• After applying the fire-wall for redis server port, I have not encounter this issue anymore.