Apache Pig: Facing issue while handling datatype in Pig

Question

I have been facing issue while handling the datatype for field qty and doing a SUM on the same field. Below is the code. I converted the qty to double but still getting the error mentioned below. Can someone please help me to understand this issue and if possible a solution?

A_test1 = load'EXT_OO_IMP' USING PigStorage('\u0001') AS (it: chararray,loc: chararray,qty: chararray,scheddate: chararray,udc_cta_no: chararray,udc_imp_pack_qty: chararray,udc_imp_ready_dt: chararray,udc_imp_ref_no: chararray,udc_ord_sys_cd: chararray,udc_source: chararray,udc_sply_typ: chararray,udc_vend_pack_id: chararray,udc_purch_stg: chararray,srs_pack_flow_indicator_cd: chararray,it_type_cd: chararray,source_owner_cd: chararray,nks_id: chararray,alloc_replen_cd: chararray);

----- ext_oo_import: {it: chararray,loc: chararray,qty: chararray,scheddate: chararray,udc_cta_no: chararray,udc_imp_pack_qty: chararray,udc_imp_ready_dt: chararray,udc_imp_ref_no: chararray,udc_ord_sys_cd: chararray,udc_source: chararray,udc_sply_typ: chararray,udc_vend_pack_id: chararray,udc_purch_stg: chararray,srs_pack_flow_indicator_cd: chararray,it_type_cd: chararray,source_owner_cd: chararray,nks_id: chararray,alloc_replen_cd: chararray}

----- ##############  ##############  ##############

import_on_order = 
        FOREACH A_test1
        GENERATE
            loc,
            it,
            nks_id,
            (double)(qty is NULL ? 0 : qty) as qty:double,
            scheddate,
            ' ' AS order_source,
            ' ' AS chs_it_type_cd;

describe import_on_order;

----- import_on_order: {loc: chararray,it: chararray,nks_id: chararray,qty: int,scheddate: chararray,order_source: chararray,chs_it_type_cd: chararray}


grp_import_on_order = GROUP import_on_order BY (loc,it,nks_id,scheddate,order_source,chs_it_type_cd);


describe grp_import_on_order;

----- grp_import_on_order: {group: (loc: chararray,it: chararray,nks_id: chararray,scheddate: chararray,order_source: chararray,chs_it_type_cd: chararray),import_on_order: {(loc: chararray,it: chararray,nks_id: chararray,qty: int,scheddate: chararray,order_source: chararray,chs_it_type_cd: chararray)}}



------------------------------- STORE TO FILE ---------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------

work__idrp_import_on_order =
                            FOREACH grp_import_on_order 
                            GENERATE    group.loc AS loc,
                                        group.it AS it,
                                        group.nks_id AS nks_id,
                                        SUM(import_on_order.qty) AS qty,
                                        group.scheddate AS scheddate,
                                        group.order_source AS order_source,
                                        group.chs_it_type_cd AS chs_it_type_cd;

describe work__idrp_import_on_order;

----- work__idrp_import_on_order: {loc: chararray,it: chararray,nks_id: chararray,qty: int,scheddate: chararray,order_source: chararray,chs_it_type_cd: chararray}

import_on_order_rp = 
        FOREACH ext_oo_import
        GENERATE
            it AS chs_it,
            loc AS chs_loc,
            (qty is NULL ? 0 : qty) as qty:double,
            scheddate AS current_due_dt, 
            ' ' AS order_source,
            'V' AS source_type_cd,
            udc_sply_typ AS sply_typ,
            udc_ord_sys_cd AS ord_sys_cd;

2019-01-31 09:03:30,819 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 0: Exception while executing (Name: grp_import_on_order: Local Rearrange[tuple]{tuple}(false) - scope-1095 Operator Key: scope-1095): org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while executing (Name: work__idrp_import_on_order: New For Each(false,false)[bag] - scope-1078 Operator Key: scope-1078): org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while executing (Name: Pre Combiner Local Rearrange[tuple]{Unknown} - scope-1097 Operator Key: scope-1097): org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while executing (Name: import_on_order: New For Each(false,false,false,false,false,false,false)[bag] - scope-977 Operator Key: scope-977): org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while executing (Name: ext_oo_import: New For Each(false,false,false,false,false)[bag] - scope-957 Operator Key: scope-957): org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while executing (Name: New For Each(false,false,false,false,false)[bag] - scope-945 Operator Key: scope-945): org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while executing [POCast (Name: Cast[double] - scope-926 Operator Key: scope-926) children: [[POProject (Name: Project[chararray][2] - scope-925 Operator Key: scope-925) children: null at []]] at []]: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.String Details at logfile: /logs/hdidrp/pig/pig_1548942743751.log 2019-01-31 09:03:30,849 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS 2019-01-31 09:03:31,012 [main] WARN org.apache.pig.PigServer - Encountered Warning IMPLICIT_CAST_TO_DOUBLE 1 time(s). import_on_order_rp: {shc_item: chararray,shc_loc: chararray,qty: double,current_due_dt: chararray,order_source: chararray,source_type_cd: chararray,sply_typ: chararray,ord_sys_cd: chararray} 2019-01-31 09:03:31,179 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 0: Exception while executing (Name: grp_import_on_order: Local Rearrange[tuple]{tuple}(false) - scope-1095 Operator Key: scope-1095): org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while executing (Name: work__idrp_import_on_order: New For Each(false,false)[bag] - scope-1078 Operator Key: scope-1078): org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while executing (Name: Pre Combiner Local Rearrange[tuple]{Unknown} - scope-1097 Operator Key: scope-1097): org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while executing (Name: import_on_order: New For Each(false,false,false,false,false,false,false)[bag] - scope-977 Operator Key: scope-977): org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while executing (Name: ext_oo_import: New For Each(false,false,false,false,false)[bag] - scope-957 Operator Key: scope-957): org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while executing (Name: New For Each(false,false,false,false,false)[bag] - scope-945 Operator Key: scope-945): org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while executing [POCast (Name: Cast[double] - scope-926 Operator Key: scope-926) children: [[POProject (Name: Project[chararray][2] - scope-925 Operator Key: scope-925) children: null at []]] at []]: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.String

Ajay Kharade Ajay Kharade · Accepted Answer · 2019-01-31T16:48:22

After looking at code, unable to get in first statement you are loading data and doing steps of transformation but again in last statament why you are transforming first data set again which is of string type and while working on this it giving exception.

import_on_order_rp = FOREACH ext_oo_import GENERATE it AS chs_it, loc AS chs_loc, (qty is NULL ? 0 : qty) as qty:double, scheddate AS current_due_dt, ' ' AS order_source, 'V' AS source_type_cd, udc_sply_typ AS sply_typ, udc_ord_sys_cd AS ord_sys_cd;

See if that is the correct case.

Apache Pig: Facing issue while handling datatype in Pig

1 Answers