Spark: Trying to run spark-shell, but get 'cmd' is not recognized as an internal or

Question

I'm trying to install Spark on my Windows desktop. Everything should work fine, but I get an error "'cmd' is not recognized as an internal or external command... "

I installed Scala, Java JDK and unzipped Spark tgz in C:\, but for some reason can't get Spark to start in cmd. Any ideas?

Answer 1

My colleague solved the problem. Although Java seemed to work ok (ref. picture), the Java path Spark was trying to read was incorrect with an extra \bin at the end. When that was removed, Spark started working! @gonbe, thank you so much for your efforts to help!

Answer 2

I had the similar error. I fixed it after following changes:

1. There were multiple Java/bin path in the System Path. So I corrected them to reflect single Java/Bin, which is in sync with JAVA_HOME
2. Added C:Windows\system32 to System Path Variable.
3. My Java_Home and java.exe was pointing different places. I fixed them.

Now it works.

Thanks guys.

Answer 3

(I'm not Windows Spark user) The spark-shell.cmd for Windows source code expects "cmd" command is available in PATH.

https://github.com/apache/spark/blob/master/bin/spark-shell.cmd

Would you try adding the directory that contains "cmd.exe" in PATH environment variable? The directory location is shown title bar in your screenshot, and environment variable setting can be done via control panel.

Answer 4

Check values in JAVA_HOME and make sure it is pointing to correct value. Add %JAVA_HOME%/bin in path value. After modification close command prompt and restart it. Write spark-shell and it will run.

Answer 5

I was the getting the same error while executing Spark-shell in the command prompt.

I tried everything mentioned above but not able to resolve the issue.

So, at last I added "C:\Windows\System32" in 'PATH' variable of System Variable and it worked.

Answer 6

All my variables were OK, so I decided to debug the scripts and I found in "spark-class2.cmd" and put another pair of quotation marks in "%RUNNERS%". BEFORE "%RUNNER%" -Xmx128m -cp "%LAUNCH_CLASSPATH%" .... AFTER ""%RUNNER%"" -Xmx128m -cp "%LAUNCH_CLASSPATH%" ....

Answer 7

For my case I had similar issue. I had to fix a couple of things.

1- Check JAVA_HOME is correct on both places;

2- Then I had to change following lines in spark-2.1.1-bin-hadoop2.7\bin folder.

• Add extra quotation marks around "%RUNNER%". So it will be like ""%RUNNER%""
• Then execute .\spark-shell.cmd again.

Answer 8

Check the java jdk version and scala version according to the below version compatibility table:

Answer 9

I had the same issue (launch spark-shell and get the the system cannot find the path) After following the above process (changing the spark_home to exclude the /bin it worked fine thanks for sharing guys

Answer 10

These are detailed steps to resolve all these issues in Windows

Spark binaries

1. It needs to be in a folder which doesn’t have space, if it must be in a folder which has space then you can use something like Unix’s symlink. I have kept the Spark binaries inside c:\ directly. Set SPARK_HOME to e.g. C:\SparkMedia\spark-2.4.2-bin-hadoop2.7. Please note “bin” is not part of the path.
2. Set “path” for the user and system to %SPARK_HOME%\bin If you decide to keep it outside “program files” (as per point a), then please give permission to the user because Spark keeps temporary files here. If you want to keep Spark inside a directory where user doesn’t have access then you need to put a file %SPARK_HOME%\conf\spark-env.sh with an entry inside it as “SPARK_LOG_DIR=a directory where user has access”.

Winutils

1. It can be inside a folder with space like C:\Program Files (x86)\Winutils Your winutils kept at C:\Program Files (x86)\Winutils works fine.
2. Set HADOOP_HOME to C:\Program Files (x86)\Winutils. Please note “bin” is not part of the path.
3. Set “path” for the user and system to %HADOOP_HOME%\bin

Java

4. It can be inside a folder with space like C:\Program Files\AdoptOpenJDK\jdk-8.0.262.10-hotspot Set JAVA_HOME to C:\Program Files\AdoptOpenJDK\jdk-8.0.262.10-hotspot. Please note “bin” is not part of the path.

5. Set “path” for the user and system to %JAVA_HOME%\bin

With the above setup a user should be able to just type “spark-shell” and spark shell should start.

Answer 11

The spark-shell command runs the spark-shell.cmd file in the spark 'bin' folder. This file contains the following command cmd /V /E /C ""%~dp0spark-shell2.cmd" %*".

The cmd command requires the path system variable to be set to %SystemRoot%\system32