Invalid format: "19690321" is too short

0
votes

I am trying to convert yyyyMMdd format to yyyy/MM/dd format using pig for that i have written below code.

Code:

STOCK_A = LOAD '/user/root/xxxx/*' USING PigStorage('|'); 
data = FILTER STOCK_A BY ($1 matches '.*ID.*'); 
MSH_DATA = FOREACH data GENERATE ToDate($8,'yyyy/MM/dd','UTC') AS dob;

When i am trying to dump the result i am getting below error.

ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR 0: Exception while executing [POUserFunc (Name: POUserFunc(org.apache.pig.builtin.ToDate3ARGS)[datetime] - scope-209 Operator Key: scope-209) children: null at []]: java.lang.IllegalArgumentException: Invalid format: "19690321" is too short

Sample:

EXVORV@#PDULD21F|ID|1|483|1020783||EXVORV@#PDULD||19690321|F|

$8 seems valid to me i am not able to locate the reason the issue is coming. Any help would be really appreciated.

2
hadoopapache-pig

2 Answers

1
votes

You use :

ToDate($8,'yyyy/MM/dd','UTC')

but the format is 

19690321

so you should have

ToDate($8,'yyyyMMdd','UTC')
1
votes

The issue is most likely because of the load statement.Since you are not specifying the schema the datatype by default will be bytearray. You will have to convert it to chararray before passing the field to ToDate

STOCK_A = LOAD '/user/root/xxxx/*' USING PigStorage('|'); 
data = FILTER STOCK_A BY ($1 matches '.*ID.*'); 
MSH_DATA = FOREACH data GENERATE ToDate((chararray)$8,'yyyy/MM/dd','UTC') AS dob;