How to read the decimal precision value from spark dataframe using python

0
votes

I have a spark dataframe. From which I have to read the column names , datatype and precision values if any.

I am able to read the name and type as below

    for f in df.schema.fields:
       name = f.name
       type = f.dataType.typeName()

The dataframe schema looks like this-

   [StructField(orgid,StringType,true), StructField(customerid,DecimalType(15,5),true), StructField(oppid,IntegerType,true)]

In the above schema I have to read the decimal precision value i.e., (15,5). Is there any way ? Thank you for any help

1
pythondataframeapache-sparkpysparkapache-spark-sql

1 Answers

2
votes

Check whether the data type is Decimal with isinstance, and then the precision value can be extracted from .precision and .scale:

from pyspark.sql.types import DecimalType

for f in df.schema.fields:
    if isinstance(f.dataType, DecimalType):
        print(f.dataType.precision, f.dataType.scale)

# 15 5