Beeline/Hive2 variables passed from script getting truncated

0
votes

I have a script from which i pass the parameters to hive variable. The entire flow is as below.One of the hive variable is getting truncated.

cat << script.sh

beeline -u "$HIVESERVER" \
-f $path_to.hql \
--hivevar TRGT_DB=$HIVE_TRGT_DB \
--hivevar SRC_CLMN=$SOURCE_CLMNS \
--hivevar INBND_DB=$HIVE_SRC_DB \
--hivevar SRC_TBL=$SOURCE_TABLE \
--hivevar TRGT_TBL=$TRGT_TABLE \

All the variables are passed correctly except the column variable (SRC_CLMN) which is getting truncated

SRC_CLMN is grepped from a file as below.

SRC_CLMN=`cat source_column.tbl | grep table | (some sed functions)`

Output of SRC_CLMN variable:

trim(regexp_replace(col1, '[^a-zA-Z]', ' ')) as col1,trim(col2) as col2,trim(regexp_extract(col1,"\(([^)]+)\)", 1)) as col3,trim(col4) as col4,trim(col5) as col5,trim(col6) as col6,'' as col7,trim(col8) as col8,trim(col9) as col9,trim(col10) as col10,trim(col11) as col11,trim(col12) as col12,trim(col13) as col13

When i print the variable it's printing the whole string.

But when i print the hive variable using SET SRC_CLMN; i see only till "trim(regexp_replace(col1,"

So this is throwing an error in hive query.

1
shellhivebeeline

1 Answers

0
votes

Output of SRC_CLMN variable:

trim(regexp_replace(col1, '[^a-zA-Z]', ' ')) as col1,trim(col2) as col2,trim(regexp_extract(col1,"(([^)]+))", 1)) as col3,trim(col4) as col4,trim(col5) as col5,trim(col6) as col6,'' as col7,trim(col8) as col8,trim(col9) as col9,trim(col10) as col10,trim(col11) as col11,trim(col12) as col12,trim(col13) as col13

as you can see in the output there is single quotations '[^a-zA-Z]', ' ')) you have to use skip character to make them available in your string

simply that's how the variable save it SRC_CLMN='trim(regexp_replace(col1, '[^a-zA-Z]', ' ')) as col1,trim(col2) as col2,trim(regexp_extract(col1,"(([^)]+))", 1)) as col3,trim(col4) as col4,trim(col5) as col5,trim(col6) as col6,'' as col7,trim(col8) as col8,trim(col9) as col9,trim(col10) as col10,trim(col11) as col11,trim(col12) as col12,trim(col13) as col13

if you check the Bold Italic part i marked, you will see it starts with a quotations and ends with one, so for the variable thats the string

you need to handle all the quotations with skip characters, you can use sed to do so to replace all ' and add skip character \'