While reading DataFrames, .csv file in PySpark. I am getting this error “ name 'spark' is not defined”

Question

rom pyspark import SparkConf
from pyspark.context import SparkContext
from pyspark.sql import SparkSession, SQLContext

from pyspark.sql.types import *

df = spark.read.csv("/home/sourabh/Downloads/1535617931_LMS_DATA_1.csv",inferSchema=True, header=True) df.printSchema()

Answer 1

That's because you haven't created any instance of spark session before doing spark.read, you will have to create a SparkSession object and that can be done like spark = SparkSession.builder().getOrCreate() This is the very basic way of defining it, you can add configurations to it using .config("<spark-config-key>","<spark-config-value>").