Sqoop + S3 + Parquet results in Wrong FS error

1
votes

When trying to import data to S3 in Parquet format using Sqoop, as follows:

bin/sqoop import --connect 'jdbc:[conn_string]' --table [table] --target-dir s3a://bucket-name/ --hive-drop-import-delims --as-parquetfile

... I get the following error:

ERROR tool.ImportTool: Imported Failed: Wrong FS: s3a://bucket-name/, expected: hdfs://localhost:9000

I have no problem importing non-parquet data or working with s3a directly through HDFS. Seems like this issue, but it was supposedly fixed many versions ago.

2
hadoopamazon-s3parquetsqoop
Which version of Sqoop are you using? – Suraj Nayak
Is there a ticket against sqoop for this? – spazm

2 Answers

3
votes

Sqoop import in parquet format currently doesn't support s3 as target directory, work around you can use is to import data to hdfs and then use distcp to copy the from hdfs to s3

0
votes

Sqoop doesn't support import to parquet file if you are using S3 as a target directory.