HDFS Namenode format issue with AWS EBS in EKS cluster

Question

I have EKS cluster with EBS storage class/volume. I have elasticsearch cluster running fine with this EBS storage (as persistent volume/pvc). I am trying to deploy hdfs namenode image (bde2020/hadoop-namenode) using statefulset, but it gives me below error always:

2020-05-09 08:59:02,400 INFO util.GSet: capacity      = 2^15 = 32768 entries
2020-05-09 08:59:02,415 INFO common.Storage: Lock on /hadoop/dfs/name/in_use.lock acquired by nodename [email protected]
2020-05-09 08:59:02,417 WARN namenode.FSNamesystem: Encountered exception loading fsimage
java.io.IOException: NameNode is not formatted.
    at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:252)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1105)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:720)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:648)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:710)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:953)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:926)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1692)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1759)

I checked run.sh of this iameg and it does seem to be formatting namenode if dir is empty. But this does not work in may case (with EBS as PVC). Any help would be very appreciated.

My deployment yml is:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: hdfs-name
  labels:
    component: hdfs-name
spec:
  serviceName: hdfs-name
  replicas: 1
  selector:
    matchLabels:
      component: hdfs-name
  template:
    metadata:
      labels:
        component: hdfs-name
    spec:
      containers:
      - name: hdfs-name
        image: bde2020/hadoop-namenode
        env:
        - name: CLUSTER_NAME
          value: hdfs-k8s
        ports:
        - containerPort: 8020
          name: nn-rpc
        - containerPort: 50070
          name: nn-web
        volumeMounts:
        - name: hdfs-name-pv-claim
          mountPath: /hadoop/dfs/name 
  volumeClaimTemplates:
  - metadata:
      name: hdfs-name-pv-claim
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: ebs
      resources:
        requests:
          storage: 1Gi

NumeroUno NumeroUno · Accepted Answer · 2020-05-09T16:19:09

With ebs storage class, lost+found folder is created automatically. Due to this, namenode format does not occur.
Having initcontainer to delete the lost+found folder seems to work.

initContainers:
  - name: delete-lost-found
    image: busybox
    command: ["sh", "-c", "rm -rf /hadoop/dfs/name/lost+found"]
    volumeMounts:
    - name: hdfs-name-pv-claim
      mountPath: /hadoop/dfs/name

HDFS Namenode format issue with AWS EBS in EKS cluster

1 Answers