Azure Data factory copy activity failed mapping strings (from csv) to Azure SQL table sink uniqueidentifier field

Question

I have an Azure data factory (DF) pipeline that consists a Copy activity. The Copy activity uses HTTP connector as source to invoke a REST end-point and returns csv stream that sinks with Azure SQL Database table.

The Copy fails when CSV contains strings (such as 40f52caf-e616-4321-8ea3-12ea3cbc54e9) which are mapped to an uniqueIdentifier field in target table with error message The given value of type String from the data source cannot be converted to type uniqueidentifier of the specified target column.

I have tried to wrapped the source string with {} such as {40f52caf-e616-4321-8ea3-12ea3cbc54e9} with no success.

The Copy activity will work if I modified the target table field from uniqueIdentifier to nvarchar(100).

Jay Gong Jay Gong · Accepted Answer · 2018-07-19T09:52:55

I reproduce your issue on my side.

The reason is data types of source and sink are dismatch.You could check the Data type mapping for SQL server.

Your source data type is string which is mapped to nvarchar or varchar, and uniqueidentifier in sql database needs GUID type in azure data factory.

So,please configure sql server stored procedure in your sql server sink as a workaround.

Please follow the steps from this doc:

Step 1: Configure your Sink dataset:

Step 2: Configure Sink section in copy activity as follows:

Step 3: In your database, define the table type with the same name as sqlWriterTableType. Notice that the schema of the table type should be same as the schema returned by your input data.

    CREATE TYPE [dbo].[CsvType] AS TABLE(
    [ID] [varchar](256) NOT NULL
)

Step 4: In your database, define the stored procedure with the same name as SqlWriterStoredProcedureName. It handles input data from your specified source, and merge into the output table. Notice that the parameter name of the stored procedure should be the same as the "tableName" defined in dataset.

Create PROCEDURE convertCsv @ctest [dbo].[CsvType] READONLY
AS
BEGIN
  MERGE [dbo].[adf] AS target
  USING @ctest AS source
  ON (1=1)
  WHEN NOT MATCHED THEN
      INSERT (id)
      VALUES (convert(uniqueidentifier,source.ID));
END

Output:

Hope it helps you.Any concern,please free feel to let me know.

Azure Data factory copy activity failed mapping strings (from csv) to Azure SQL table sink uniqueidentifier field

1 Answers