OrientDB: Trouble creating distributed clusters with hotalignment=true

1
votes

I are using OrientDB 2.1.7 in embedded and distributed mode, and the application is running on two nodes (two Java processes).

When using the default JSON database configuration, everything works fine:

{
  "autoDeploy": true,
  "hotAlignment": false,
  "executionMode": "undefined",
  "readQuorum": 1,
  "writeQuorum": 2,
  "failureAvailableNodesLessQuorum": false,
  "readYourWrites": true,
  "servers": {
    "*": "master"
  },
  "clusters": {
    "internal": {
      "replication": false
    },
    "index": {
      "replication": false
    },
    "*": {
      "servers": ["<NEW_NODE>"]
    }
  }
}

However, I want to change hotAlignment to "true", but when I do this, I get the following error when starting node 2 (note: the database is not deployed to node 2 at this point, but it has been created on node 1):

2015-12-17 16:28:32:010 INFO [order2] Saving distributed configuration file for database 'test' to: ./databases/test/distributed-config.json [OHazelcastPlugin]
2015-12-17 16:28:32:012 INFO [order2] received new status order2.test=ONLINE [OHazelcastPlugin]
2015-12-17 16:28:32:019 INFO [order2] found no previous messages in queue orientdb.node.order2.test.request [OHazelcastDistributedMessageService]
2015-12-17 16:28:32:043 INFO [order2] Publishing online status for database order2.test... [OHazelcastDistributedDatabase]
2015-12-17 16:28:32:046 INFO [order2] received updated status order2.test=ONLINE [OHazelcastPlugin]
2015-12-17 16:28:32:052 INFO [order2] class 'ORole', creation of new local cluster 'orole_order2' (id=-1) [OHazelcastPlugin]
2015-12-17 16:28:32:134 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=15). Retrying 0/10... [ODistributedStorage]
2015-12-17 16:28:32:522 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=15). Retrying 1/10... [ODistributedStorage]
2015-12-17 16:28:32:903 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=15). Retrying 2/10... [ODistributedStorage]
2015-12-17 16:28:33:283 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=15). Retrying 3/10... [ODistributedStorage]
2015-12-17 16:28:33:663 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=15). Retrying 4/10... [ODistributedStorage]
2015-12-17 16:28:34:043 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=15). Retrying 5/10... [ODistributedStorage]
2015-12-17 16:28:34:419 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=15). Retrying 6/10... [ODistributedStorage]
2015-12-17 16:28:34:799 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=15). Retrying 7/10... [ODistributedStorage]
2015-12-17 16:28:35:176 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=15). Retrying 8/10... [ODistributedStorage]
2015-12-17 16:28:35:556 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=15). Retrying 9/10... [ODistributedStorage]
2015-12-17 16:28:35:910 SEVER [order2] error on creating cluster 'orole_order2' in class 'ORole': [OHazelcastPlugin][order2] Error on starting distributed plugin
com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster 'orole_order2' in class 'ORole'
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installLocalClusterPerClass(OHazelcastPlugin.java:1624)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDbClustersForLocalNode(OHazelcastPlugin.java:1293)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.loadDistributedDatabases(OHazelcastPlugin.java:1434)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.startup(OHazelcastPlugin.java:187)
at com.orientechnologies.orient.server.OServer.registerPlugins(OServer.java:979)
at com.orientechnologies.orient.server.OServer.activate(OServer.java:346)
at no.kommune.bergen.graftest.GraphDBServerWrapper.startDatabaseServer(GraphDBServerWrapper.java:74)
at no.kommune.bergen.graftest.GraftestApplication.main(GraftestApplication.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:53)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster on distributed nodes: local and remote ids assigned are different
at com.orientechnologies.orient.server.distributed.ODistributedStorage.addCluster(ODistributedStorage.java:1364)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.addCluster(ODatabaseDocumentTx.java:1291)
at com.orientechnologies.orient.core.metadata.schema.OClassImpl.createClusterIfNeeded(OClassImpl.java:2102)
at com.orientechnologies.orient.core.metadata.schema.OClassImpl.addCluster(OClassImpl.java:1068)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installLocalClusterPerClass(OHazelcastPlugin.java:1617)
... 13 more

It seems to have trouble creating local clusters for node2 when turning on hotAlignment. I have also seen the same problem when turning off autodeploy. Is there something missing from my configuration?

EDIT: This is how we start the server:

    OServerConfiguration cfg = new OServerConfiguration();
    //DB server config
    cfg.handlers = new ArrayList<OServerHandlerConfiguration>();
    //Cluster plugin config
    OServerHandlerConfiguration hazelcastHandlerConfig = new OServerHandlerConfiguration();
    hazelcastHandlerConfig.clazz = "com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin";
    hazelcastHandlerConfig.parameters = new OServerParameterConfiguration[6];
    hazelcastHandlerConfig.parameters[0] = new OServerParameterConfiguration("nodeName", nodeName);
    hazelcastHandlerConfig.parameters[1] = new OServerParameterConfiguration("enabled", "true");
    hazelcastHandlerConfig.parameters[2] = new OServerParameterConfiguration("configuration.db.default", "/config/distributed-db-config.json");
    hazelcastHandlerConfig.parameters[3] = new OServerParameterConfiguration("configuration.hazelcast", "/config/hazelcast.xml");
    hazelcastHandlerConfig.parameters[4] = new OServerParameterConfiguration("conflict.resolver.impl", "com.orientechnologies.orient.server.distributed.conflict.ODefaultReplicationConflictResolver");
    hazelcastHandlerConfig.parameters[5] = new OServerParameterConfiguration("sharding.strategy.round-robin", "com.orientechnologies.orient.server.hazelcast.sharding.strategy.ORoundRobinPartitioninStrategy");
    cfg.handlers.add(hazelcastHandlerConfig);
    //Network config
    cfg.network = new OServerNetworkConfiguration();
    cfg.network.protocols = new ArrayList<OServerNetworkProtocolConfiguration>();
    cfg.network.protocols.add(new OServerNetworkProtocolConfiguration("binary", "com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary"));
    OServerNetworkListenerConfiguration binaryListenerCfg = new OServerNetworkListenerConfiguration();
    binaryListenerCfg.ipAddress = "0.0.0.0";
    binaryListenerCfg.portRange = "2424-2430";
    binaryListenerCfg.protocol = "binary";
    cfg.network.listeners = new ArrayList<>();
    cfg.network.listeners.add(binaryListenerCfg);

    //Storage config
    OServerStorageConfiguration storageCfg = new OServerStorageConfiguration();
    storageCfg.path = "plocal:./databases/test";
    storageCfg.name = "test";
    storageCfg.loadOnStartup = true;
    cfg.storages = new OServerStorageConfiguration[]{storageCfg};
    //User config
    OServerUserConfiguration userCfg = new OServerUserConfiguration(username, password, "*");
    cfg.users = new OServerUserConfiguration[]{userCfg};

    databaseServer = OServerMain.create(true);
    databaseServer.startup(cfg);

    databaseServer.activate();
2
orientdb
Hi Knut, it might be useful to you look at the following links groups.google.com/forum/#!topic/orient-database/vRlJq28fh_8 groups.google.com/forum/#!topic/orient-database/5NCen4Zxfjs - Alessandro Rota
Yes, I see. But I also tried with 2.2 beta, and I get the same error then. Any idea about configuring the server to use both autodeploy and hotalignment? - Knut
what is the type of your database? plocal? - Wouter
Yes, the database servers are started using "plocal:./databases/test" as storage. We are running the servers embedded, using OServerMain. - Knut

2 Answers

1
votes

OK, so I have found a solution to this problem (found here)

I added these two lines before creating the server the server:

    String orientdbHome = new File("").getAbsolutePath();
    System.setProperty("ORIENTDB_HOME", orientdbHome);

So the server is started like this:

    String orientdbHome = new File("").getAbsolutePath();
    System.setProperty("ORIENTDB_HOME", orientdbHome);

    databaseServer = OServerMain.create(true);
    databaseServer.startup(cfg);

    databaseServer.activate();

And now there is no problem when starting node 2.

I think this information is missing from this page from the documentation.

0
votes

You should not use hotAligment, it is seriously broken. It will be fixed in 2.2

See for example https://github.com/orientechnologies/orientdb/issues/2270 https://github.com/orientechnologies/orientdb/issues/5455

For now, adding and re-adding a node to a distributed setup needs to do a full sync of the database.