PySpark reading multiple files while creating new column containing existing column name

Question

I would like to read n csv files using pyspark. The csv have the same schema but with different columns names.

While reading those files I would like to create an additional column 'pipeline' that contains a substring of first column name.

How can I implement this?

 df = spark.read.format("csv") \
                .option("header", True) \
                .load(path + "*.csv")
                .withColumn("pipeline", 

Answer 1

df = spark.read.format("csv") \
                .option("header", "false") \
                .load(path + "*.csv")
                .toDF('header_1')
                .withColumn("pipeline", lit(path))