HDinsight hive activity pipeline not creating output in Azure data lake

Question

Trying to run a hive activity using data factory, the pipeline finished fine and table with data is created inside cluster, but output dataset is not creating file in Azure data lake store, is this by intention?

just trying to learn so be gentle.

Input dataset:

Standard input csv file containing the data

{
"name": "dlsinput",
"properties": {
    "published": false,
    "type": "AzureDataLakeStore",
    "linkedServiceName": "dls",
    "typeProperties": {
        "fileName": "output.csv",
        "folderPath": "data/output/",
        "format": {
            "type": "TextFormat",
            "columnDelimiter": ","
        }
    },
    "availability": {
        "frequency": "Day",
        "interval": 1
    },
    "external": true,
    "policy": {}
}
}

Pipeline:

Pipeline referring to hdinsight cluster

{
"name": "HiveActivitySamplePipeline",
"properties": {
    "activities": [
        {
            "type": "HDInsightHive",
            "typeProperties": {
                "scriptPath": "scripts/hive.hql",
                "scriptLinkedService": "sta"
            },
            "inputs": [
                {
                    "name": "dlsinput"
                }
            ],
            "outputs": [
                {
                    "name": "dlsoutput"
                }
            ],
            "scheduler": {
                "frequency": "Day",
                "interval": 1
            },
            "name": "HiveActivitySample",
            "linkedServiceName": "hdi"
        }
    ],
    "start": "2018-04-05T12:20:00Z",
    "end": "2018-04-10T23:59:59Z",
    "isPaused": false,
    "hubName": "adf",
    "pipelineMode": "Scheduled"
}
}

Output:

Output with file i want to have created

{
"name": "dlsoutput",
"properties": {
    "published": false,
    "type": "AzureDataLakeStore",
    "linkedServiceName": "dls",
    "typeProperties": {
        "fileName": "myfile.csv",
        "folderPath": "data/output/",
        "format": {
            "type": "TextFormat",
            "rowDelimiter": "\n",
            "columnDelimiter": ","
        }
    },
    "availability": {
        "frequency": "Day",
        "interval": 1
    }
}
}

Hive.hql

DROP TABLE IF EXISTS temp;
CREATE EXTERNAL TABLE IF NOT EXISTS temp (
Name STRING,
Road STRING,
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS TEXTFILE
 LOCATION 'adl://dls.azuredatalakestore.net/data/output/';

Shawn Xiao Shawn Xiao · Accepted Answer · 2018-04-20T11:12:56

Looks like it's because the Hive.hql only includes commands to create a table, and there is no data inserted to the table, so you didn't see any data file generated.

HDinsight hive activity pipeline not creating output in Azure data lake

1 Answers