Apache Ignite Cache Read Issue (All affinity node left the grid)

Question

Ignite Version: 2.5
Ignite Cluster Size: 10 nodes

One of our spark job is writing data to ignite cache in every one hour. The total records per hour is 530 million. Another spark job read the cache but when it try to read it, we are getting the error as "Failed to Execute the Query (all affinity nodes left the grid)

Any pointers will be helpful.

Answer 1

If you are using "embedded mode" deployment it will mean that nodes are started when jobs are ran and stopped when jobs finish. If you do not have enough backups you will lose data when this happens. Any chance this may be your problem? Be sure to connect to Ignite cluster with client=true.