Cannot create Azure HDInsight cluster with Hive metastore using Powershell

Question

I get error when I try to create Azure HDInsight cluster using powershell cmdlet:

New-AzureRmHDInsightClusterConfig  `
    | Add-AzureRmHDInsightMetastore `
        -SqlAzureServerName "$sqlDatabaseServerName.database.windows.net" `
        -DatabaseName $hiveMetaStoreDBName `
        -Credential $sqlServerCred `
        -MetastoreType HiveMetaStore `
    | New-AzureRmHDInsightCluster `
        -ResourceGroupName $resourceGroupName `
        -HttpCredential $clusterCreds `         
        -ClusterName $clusterName `
        -Location $location `
        -ClusterType $clusterType `
        -OSType $OSType `
        -Version "$hdVersion" `
        -SshCredential $clusterCreds `
        -DefaultStorageAccountName "$storageAccountName.blob.core.windows.net" `
        -DefaultStorageAccountKey $storageAccountKey `
        -ClusterSizeInNodes $clusterNodes

Looks like parameters are not recognized by powershell because it asks to input them (see below). I input required parameters (Location, ClusterName, ClusterSizeInNodes) and then error occurs.

cmdlet New-AzureRmHDInsightCluster at command pipeline position 3
Supply values for the following parameters:
(Type !? for Help.)
Location: West Europe
ClusterName: xxxxxxxxx
ClusterSizeInNodes: 1
New-AzureRmHDInsightCluster : BadRequest: ParameterNullOrEmpty,Parameter 'ASVAccount.AccountName' cannot be null or 
empty.
At line:117 char:11
+         | New-AzureRmHDInsightCluster `
+           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : NotSpecified: (:) [New-AzureRmHDInsightCluster], CloudException
+ FullyQualifiedErrorId : Hyak.Common.CloudException,Microsoft.Azure.Commands.HDInsight.NewAzureHDInsightClusterCom 
   mand

Does somebody know why its happens or what is wrong in smdlet?

Answer 1

From the error message, it seems like your $storageAccountName parameter to the New-AzureRmHDInsightCluster cmdlet is Null or Empty, you may want to inspect further on this.

Besides, I would also strongly recommend you to specify the -DefaultStorageContainer to the New-AzureRmHDInsightCluster cmdlet as well. This will ensures that the cmdlet will be able to resolve the FQDN of your storage account Uri.

E.g. asv://[e-mail]/

Hope this helps!

Answer 2

Use below command for Cluster with Hive Metastore.

Here is a working PowerShell script, to be used with Azure ARM PowerShell, 1.0.1 or later – you can install Azure RM PS via web platform installer or follow this blog https://azure.microsoft.com/en-us/blog/azps-1-0/ 


Add-AzureRmAccount
$MyClusterName = "clustername";
$MyClusterLocation = "East US 2";
$NumClusterNodes = 2;
$MyClusterVersion = "3.2";
$MyHDInsightUserName = ""
$MyHDInsightPwd = ""
$MySqlAzureUserName = ""
$MySqlAzurePwd = ""
$MySqlAzureServerName = "*.database.windows.net"
$MySqlAzureDbName = "Dbtest"
$MyDefaultContainerName = "tastoreps"
$clusterResourceGroupName = "dirg"
# Use the correct Azure Subscription!
$subid = ""
Select-AzureRmSubscription -SubscriptionId $subid
# Storage key
$primaryStorageAcctName = "toragesouth"
$primaryStorageResourceGroupName = "storagerg"
# you need to use an ARM based storage as the primary account , classic storage won’t work as a primary account, known issue
$storageAccountKey = Get-AzureRmStorageAccountKey -ResourceGroupName $primaryStorageResourceGroupName -Name $primaryStorageAcctName | %{ $_.Key1 }
# credentials
$HdInsightPwd = ConvertTo-SecureString $MyHDInsightPwd -AsPlainText -Force
$HdInsightCreds = New-Object System.Management.Automation.PSCredential ($MyHDInsightUserName, $HdInsightPwd)
$SqlAzurePwd = ConvertTo-SecureString $MySqlAzurePwd -AsPlainText -Force
$SqlAzureCreds = New-Object System.Management.Automation.PSCredential ($MySqlAzureUserName, $SqlAzurePwd)
$config = New-AzureRmHDInsightClusterConfig -ClusterType Hadoop | 
Add-AzureRmHDInsightMetastore -SqlAzureServerName $MySqlAzureServerName -DatabaseName $MySqlAzureDbName -Credential $SqlAzureCreds -MetastoreType HiveMetastore | 
Add-AzureRmHDInsightMetastore -SqlAzureServerName $MySqlAzureServerName -DatabaseName $MySqlAzureDbName -Credential $SqlAzureCreds -MetastoreType OozieMetastore 
$config.DefaultStorageAccountName="$StorageAcctName.blob.core.windows.net"
$config.DefaultStorageAccountKey=$storageAccountKey
#create cluster
New-AzureRmHDInsightCluster -config $config -OSType Windows -clustername $MyClusterName -HttpCredential $HdInsightCreds -DefaultStorageContainer $MyDefaultContainerName -Location $MyClusterLocation -ResourceGroupName $clusterResourceGroupName -ClusterSizeInNodes $NumClusterNodes -Version $MyClusterVersion