Arangodb stops async replication after some time

Question

I use ArangoDB 3.2.2 on Kubernetes. I have one master db and one slave db. on the slave db inside the arangosh I enter the following commands:

db._useDatabase("mydb");

equire("@arangodb/replication").setupReplication({
  endpoint: "tcp://master:8529",
  username: "root",
  password: "123456",
  database: "mydb",
  verbose: true,
  includeSystem: false,
  incremental: true,
  autoResync: true,
  maxConnectRetries: 20,
  adaptivePolling: false,
  idleMinWaitTime: 0.001,
  idleMaxWaitTime: 1.5,
  chunkSize: 5000
});

The command blocks for a few seconds and then the replication is started and I can see that the collections from the master db are replicated to the slave. However, after some time the replication stops.

When I run: require("@arangodb/replication").applier.state This is what I see:

{ 
  "state" : { 
    "running" : true, 
    "lastAppliedContinuousTick" : "6384020", 
    "lastProcessedContinuousTick" : "6384026", 
    "lastAvailableContinuousTick" : "6389353", 
    "safeResumeTick" : "6382367", 
    "progress" : { 
      "time" : "2017-11-15T12:14:09Z", 
      "message" : "fetching master log from tick 6382367, first regular tick 6367894, barrier: 0, open transactions: 0", 
      "failedConnects" : 0 
    }, 
    "totalRequests" : 1155, 
    "totalFailedConnects" : 0, 
    "totalEvents" : 83, 
    "totalOperationsExcluded" : 0, 
    "lastError" : { 
      "errorNum" : 0 
    }, 
    "time" : "2017-11-15T12:15:47Z" 
  }, 
  "server" : { 
    "version" : "3.2.2", 
    "serverId" : "77969163868004" 
  }, 
  "endpoint" : "tcp://master:8529", 
  "database" : "mydb" 
}

When I run the replication command again I see that the collections are being replicated again...

How can I initialize a stable replication process? I understand that this is an asynchronous replication but in my use case I need the slave to be updated as fast as possible... Does my replication config is fit to my needs?

If I understand correctly, the replication applier reports its state as "running" while other analysis shows that is effectively not applying any changes from the master? I would like to understand how you can to the conclusion that the applier does not work. Are you looking at the document counts in collections? Or are you looking at multiple progress results of the applier's state() command? At least here the tick values and the totalRequests value should increase over time. If they do not, then the applier seems to be hanging indeed. — stj
By the way, the configured chunkSize value seems rather low. It's just 5KB which may mean batches returned by the master will be at most that big. 5KB is not much if many documents get inserted/modified/removed or if document size is typically bigger than around 100 bytes. I suggest increasing the chunkSize value to around 1 MB and see if it helps, or even use the default value. I would also suggest to turn on adaptivePolling, because it will mean fewer HTTP requests are being made from slave to master when there is not much to do. — stj
Are there any other processes (apart from arangod itself) that attempt to write data on the slave, specifically into collection my_col? — stj
There are definitely differences in the two storage engines (RocksDB and MMFiles). However, replication is supposed to work with both storage engines and not just one. So on the slave you are only reading data and with RocksDB it gets stuck? That sounds like a bug then! — stj
It may be a side effect of a bug that is fixed in release 3.2.8, but that is pure speculation, as I haven't reproduced the problem yet. — stj

Mizh Mizh · Accepted Answer · 2017-11-21T15:11:45

It seems like it was a bug in ArangoDB 3.2.2 In ArangoDB 3.2.8 The replication seems to work (even with rocksdb).

Arangodb stops async replication after some time

1 Answers