runtime exception when load and store multiple files using avro in pig

Question

I run into this strange problem when try to load multiple text formatted files and convert them into avro format using pig. However, if I read and convert one file at a time in separated runs, everything is fine. The error message is following

2012-08-21 19:15:32,964 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Unable to recreate exception from backed error: org.apache.avro.file.DataFileWriter$AppendWriteException: java.lang.RuntimeException: Datum 1980-01-01 00:00:00.000 is not in union ["null","long"] at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:263) at org.apache.pig.piggybank.storage.avro.PigAvroRecordWriter.write(PigAvroRecordWriter.java:49) at org.apache.pig.piggybank.storage.avro.AvroStorage.putNext(AvroStorage.java:612) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:531) at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapB

my code is

set1 = load '$input_dir/set1.txt' using PigStorage('|') as (
   id:long,
   f1:long,
   f2:chararray,
   f3:float,
   f4:float,
   f5:float,
   f6:float,
   f7:float,
   f8:float,
   f9:float,
   f10:float,
   f11:float,
   f12:float);
store set1 into '$output_dir/set1.avro'
using org.apache.pig.piggybank.storage.avro.AvroStorage();

set2 = load '$input_dir/set2.txt' using PigStorage('|') as (
   id : int,
   date : chararray);
store set2 into '$output_dir/set2.avro'
using org.apache.pig.piggybank.storage.avro.AvroStorage();

The first file is converted fine, but the 2nd one is failed. The error is coming from the 2nd field in the 2nd file, but the strange thing is that I don’t even have “long” in my schema while the error message is showing ["null","long"].

I use pig 0.10.0 and avro-1.7.1.jar.

I wonder if this is a bug or I missed something.

Thanks. Dan

Here’s set1.txt

827352|740214|Long|26|0.08731795012183759|1661335.541733333|0|0|0.001057865808239878|0.001059541098077884|0.001059541098077821|0.0514156486228232|0.001043980181757539
827353|740214|Short|12|-0.05967910581502997|-1135471.22271|0|0|-0.001185620143839061|-0.001187497751909232|-0.001187497751909183|-0.0747641932858414|-0.0001307449002148424
827354|740214|Total|38|0.02763884430680765|19026277.40819863|0|0|-0.0001277543355991829|-0.0001279566538313473|-0.0001279566538313626|-0.02334854466301821|0.0009132352815426966
827193|739576|Long|26|0.08731795012183759|1661335.541733333|0|0|0.001057865808239878|0.001059541098077884|0.001059541098077821|0.0514156486228232|0.001043980181757539
827194|739576|Short|12|-0.05967910581502997|-1135471.22271|0|0|-0.001185620143839061|-0.001187497751909232|-0.001187497751909183|-0.0747641932858414|-0.0001307449002148424
827195|739576|Total|38|0.02763884430680765|19026277.40819863|0|0|-0.0001277543355991829|-0.0001279566538313473|-0.0001279566538313626|-0.02334854466301821|0.0009132352815426966
827355|740215|Long|51|1.776868012839072|113652088.7063555|0|0|0.01952547658695701|0.0195703176808393|0.01957031768083928|1.164818333642054|0
827356|740215|Short|34|-2.360589090333165|-150988074.9471841|0|0|-0.00868330219442376|-0.008616238065508337|-0.008616238065508375|-0.5943698959308671|-0.02690679230502523
827357|740215|Total|85|-0.5837210774940929|63962032.00527128|0|0|0.01084217439253325|0.01095407961533095|0.0109540796153309|0.5704484377111866|-0.02690679230502523
827202|739590|Long|53|1.777568428360522|113696888.7063555|0|0|0.01952547658695701|0.0195703176808393|0.01957031768083928|1.156653489849146|0

Here’s the set2.txt

1|1980-01-01 00:00:00.000
2|1980-01-02 00:00:00.000
3|1980-01-03 00:00:00.000
4|1980-01-04 00:00:00.000
5|1980-01-07 00:00:00.000
6|1980-01-08 00:00:00.000
7|1980-01-09 00:00:00.000
8|1980-01-10 00:00:00.000
9|1980-01-11 00:00:00.000
10|1980-01-14 00:00:00.000

Plz add your comment as one of the answers it will help others who are facing this issue. — Murali Rao

mark mark · Accepted Answer · 2012-08-23T08:45:10

it seems that pig need to load first, please try

set1 = load '$input_dir/set1.txt' using PigStorage('|');
set1 = load '$input_dir/set1.txt' using PigStorage('|');

--other logic

runtime exception when load and store multiple files using avro in pig

1 Answers