remove surrounding quotes from fields while loading data into hive

3
votes

I want to load a table with input data into hive. I have data in the following format. 

"153662";"0002241447";"0"
"153662";"000647036X";"0"
"153662";"0020434901";"0"
"153662";"0020973403";"0"
"153662";"0028604202";"0"
"153662";"0030437512";"0"

I want to load this data into a table with two varchar columns and one int column.But the surrounding double quotes trouble me. I have created the following table.

CREATE EXTERNAL TABLE Table(A varchar(50),B varchar(50),C varchar(50))
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\;'
LINES TERMINATED BY '\n'

STORED AS TEXTFILE

but the quotes around the field also become part of field as shown below.

"276725"    "034545104X"    "0"
"276726"    "0155061224"    "5"

I want to ignore them. Also I want the third field to be read as INT. Currently it becomes NULL when I provide third field as INT while making table.

2
hadoophive

2 Answers

7
votes

You will have to use Csv-Serde for this.

CREATE TABLE Table(A varchar(50),B varchar(50),C varchar(50))
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
WITH SERDEPROPERTIES 
(
    "separatorChar" = ";",
    "quoteChar"     = "\""
)  
STORED AS TEXTFILE;
2
votes

Multiple ways to achieve this:

  1. Use CSV serde
  2. Use regex serde- regex "\"(.*)\"\;\"(.*)\"\;\"(.*)\""
  3. Load data to external table then remove double quotes:

CREATE EXTERNAL TABLE source( a string, b String, c String) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\;' LOCATION 'xyz';

CREATE TABLE destination AS SELECT REGEXP_REPLACE(a,'"',''), REGEXP_REPLACE(b,'"',''), CAST ( REGEXP_REPLACE(c,'"','') AS BIGINT) FROM source;