How to create Azure HDInsight cluster with Hive metastore using Powershell?

Question

I was trying to create HDInsight cluster with Hive metastore. Cluster was created successfully but without Hive metastore. Also any error was not returned. I used Powershell script below.

$resourceGroupName = "ResourceGroup"
$storageAccountName = "myStorage"
$containerName="myContainer"

$clusterName = "myCluster"
$location = "West Europe"
$clusterNodes = 2
$clusterType = "Hadoop"
$OSType = "Windows"
$hdVersion = "3.2"

$hadoopUserName = "user"
$hadoopUserPassword = "password"
$hadoopUserPW = ConvertTo-SecureString -String $hadoopUserPassword -AsPlainText -Force
$clusterCreds = New-Object System.Management.Automation.PSCredential($hadoopUserName,$hadoopUserPW)

$sqlDatabaseServerName = "mydbserver"
$metaStoreDBName = "HiveMetaStoreDB"
$sqlDatabaseLogin = "myDBLogin"
$sqlDatabasePassword = "myDBPassword"
$sqlDatabaseSecurePassword = ConvertTo-SecureString -String $sqlDatabasePassword -AsPlainText -Force
$sqlServerCred = New-Object System.Management.Automation.PSCredential ($sqlDatabaseLogin, $sqlDatabaseSecurePassword)


# Create a new HDInsight cluster

    $config = New-AzureRmHDInsightClusterConfig -ClusterType Hadoop `
        | Add-AzureRmHDInsightMetastore `
            -SqlAzureServerName "$sqlDatabaseServerName.database.windows.net" `
            -DatabaseName $metaStoreDBName `
            -Credential $sqlServerCred `
            -MetastoreType HiveMetastore 

$config.DefaultStorageAccountName = "$storageAccountName.blob.core.windows.net"
$config.DefaultStorageAccountKey = $storageAccountKey           

New-AzureRmHDInsightCluster `
        -config $config `
        -OSType $OSType `
        -clustername $clusterName `
        -HttpCredential $clusterCreds `
        -DefaultStorageContainer $containerName `
        -Location $location `
        -ResourceGroupName $resourceGroupName `
        -ClusterSizeInNodes $clusterNodes `
        -Version $hdVersion

Could somebody tell me the reason why Hive metastore was not created? Or what should I do to create it?

I had same problem with New-AzureRmHDInsightClusterConfig, but it works after Microsoft Azure PowerShell was updated (30th Januray 2016). — Tomas

Jonathan Gao Jonathan Gao · Accepted Answer · 2016-01-05T21:06:04

The cluster creation process doesn't create the Azure Database for you. You must create the database before you can use it as the Hive metastore.

How to create Azure HDInsight cluster with Hive metastore using Powershell?

1 Answers