Azure Data Lake U-SQL job failing with error: JM_FinalMetadataOperationUserError

1
votes

I've been running U-SQL jobs on Azure Data Lake that update partitioned tables and they run successfully, but fail at the "Finalizing" phase with the error "JM_FinalMetadataOperationUserError", the stack trace is as follows:

Object ec2d8ab3-6df5-48eb-9169-daec48bf27a6.DeployedLogs.dbo.ConcurrentRecord__1__System.DateTime.FromBinary
(5246322962427387904 /* 2012-02-01T12:00:00.0000000Z */) 
already exists at Scope.MetadataService.Client.WcfBasedMetadataClient.ExecuteMetadataJob(MetadataJob mdJob) 
at MetaDataExecutor.RealMetaDataExecutor.Execute(MetadataJob job) 
at MetaDataExecutor.Committer.Commit(MetadataJob job)

The error message states:

Failed to write job meta-data due to user error

What can cause this problem? Is it due to the values being added/truncated from the table?

Would appreciate the help.

1
azure-data-lakeu-sql

1 Answers

5
votes

The reason for the error is that you ran two jobs at the same time to create the same partition. One script completed and the second one then failed when it detects that the partition has been created.

We are currently looking into how we can improve the error message and reduce the chances of such race conditions.

[Added] Here are the details from looking at the job history and the evidence for the race condition (anonymized the account name):

  1. This job compiled at 05/31/2016 00:56:06.100. At this time, the partition didn’t exist, so CreateEntityOperation for this partition included in the algebra. Logs: d,05/31/2016 00:56:06.100,ScopeMetadataService,GetMetaEntity,SrcFile="" SrcFunc="" SrcLine="0" Pid="17072" Tid="16548" TS="0x01D1BB11E2999697" String1="Begin to get information of meta entity ec2d8ab3-6df5-48eb-9169-daec48bf27a6.DeployedLogs.dbo.ConcurrentRecord__1__System.DateTime.FromBinary(5246322962427387904 /* 2012-02-01T12:00:00.0000000Z */)"

  2. The partition created by another job at 05/31/2016 01:07:53.766 (after compilation of failed job but before metadata operation execution of failed job). Job url: https://b******.azuredatalakeanalytics.net/Jobs/909c4680-fd90-452a-93b0-fcc7860de504?api-version=2015-02-12 Log: d,05/31/2016 01:07:53.766,ScopeMetadataService,CreateEntity,SrcFile="" SrcFunc="" SrcLine="0" Pid="17608" Tid="20028" TS="0x01D1BB138866CD29" String1="entity ec2d8ab3-6df5-48eb-9169-daec48bf27a6.DeployedLogs.dbo.ConcurrentRecord__1__System.DateTime.FromBinary(5246322962427387904 /* 2012-02-01T12:00:00.0000000Z */) created successfully.

  3. User job failed when executing the metadata operation to create the partition at 05/31/2016 02:17:57.241 because the partition created by another job. Logs: e,05/31/2016 02:17:57.241,ScopeMetadataService,SMS unexpected exception when commit metadata task,SrcFile="" SrcFunc="" SrcLine="0" Pid="17072" Tid="20972" TS="0x01D1BB1D51DE3815" String1="Scope.MetadataService.Contract.FaultContract.AlreadyExistException: Object ec2d8ab3-6df5-48eb-9169-daec48bf27a6.DeployedLogs.dbo.ConcurrentRecord__1__System.DateTime.FromBinary(5246322962427387904 /* 2012-02-01T12:00:00.0000000Z */) already exists.

Please contact me at mrys at Microsoft if you want to follow up in more details.