Unable to write data in HDFS "datanode" - Node added in excluded list

Question

I'm running "namenode" and "datanode" in the same jvm, when I try to write data I'm getting the following exception

org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy$NotEnoughReplicasException: at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:836) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:724) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseLocalRack(BlockPlacementPolicyDefault.java:631) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseLocalStorage(BlockPlacementPolicyDefault.java:591) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTargetInOrder(BlockPlacementPolicyDefault.java:490) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:421) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:297) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:148) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:164) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2127) at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:294) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2771) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:876) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:567) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.security.auth.Subject.doAs(Subject.java:423) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)

        final File file = new File("C:\\ManageEngine\\test\\data\\namenode");
        final File file1 = new File("C:\\ManageEngine\\test\\data\\datanode1");
        BasicConfigurator.configure();
        final HdfsConfiguration nameNodeConfiguration = new HdfsConfiguration();
        FileSystem.setDefaultUri(nameNodeConfiguration, "hdfs://localhost:5555");
        nameNodeConfiguration.set(DFSConfigKeys.DFS_NAMENODE_NAME_DIR_KEY, file.toURI().toString());
        nameNodeConfiguration.set(DFSConfigKeys.DFS_REPLICATION_KEY, "1" );
        final NameNode nameNode = new NameNode(nameNodeConfiguration);



        final HdfsConfiguration dataNodeConfiguration1 = new HdfsConfiguration();
        dataNodeConfiguration1.set(DFSConfigKeys.DFS_DATANODE_DATA_DIR_KEY, file1.toURI().toString());
        dataNodeConfiguration1.set(DFSConfigKeys.DFS_DATANODE_ADDRESS_KEY, "localhost:5556" );
        dataNodeConfiguration1.set(DFSConfigKeys.DFS_REPLICATION_KEY, "1" );
        FileSystem.setDefaultUri(dataNodeConfiguration1, "hdfs://localhost:5555");
        final DataNode dataNode1 = DataNode.instantiateDataNode(new String[]{}, dataNodeConfiguration1);



        final FileSystem fs = FileSystem.get(dataNodeConfiguration1);

        Path hdfswritepath = new Path(fileName);
        if(!fs.exists(hdfswritepath)) {
            fs.create(hdfswritepath);
            System.out.println("Path "+hdfswritepath+" created.");
        }
        System.out.println("Begin Write file into hdfs");

        FSDataOutputStream outputStream=fs.create(hdfswritepath);
        //Cassical output stream usage
        outputStream.writeBytes(fileContent);
        outputStream.close();
        System.out.println("End Write file into hdfs");

Request data - Image

You cannot have replication number > number of data nodes. — tk421
Hey, Thanks for your comment, but I want to run single datanode, is there any possible way to archive this. — hirik
Found this line in logs "Failed to find datanode (scope="" excludedScope="/default-rack"). numOfDatanodes=0" — hirik

tk421 tk421 · Accepted Answer · 2019-06-03T19:26:18

You cannot have the number of replicas higher than the number of datanodes.

If you want run on a single node, set dfs.replication to 1 in your hdfs-site.xml.

Unable to write data in HDFS "datanode" - Node added in excluded list

1 Answers