How to Export Huge Table from Azure SQL Database to CSV using Logic Apps

0
votes

I work on proof of concept of Azure Logic Apps. I have a table with 70 columns and 10 million rows. I use Logic Apps to export data and upload to FTP.

The process works on a small dataset (3000 rows); however, the process fails on the whole table.

I get the error message below:

BadRequest. Http request failed as there is an error: 'Cannot write more bytes to the buffer than the configured maximum buffer size: 104857600.'

Logic App Steps:

  1. Recurrence
  2. SELECT statement - failure
  3. Create CSV Table
  4. Upload to FTP

Please advise. I could use a stored procedure and batch it in 5,000 rows but I'm not sure how to implement it in Logic Apps.

1
azureazure-sql-databaseazure-logic-apps
is your file more tan 50mb at the end ? may be you can trigger a data factory run that will export to csv in a blob storage then upload to ftp ? – Thomas

1 Answers

0
votes

Based on this documentation, there is a file a HTTP message size limit of 100 MB and a file size limit of 50 MB when processing FTP files. You can use chunking to workaround those limits, if the connector supports it.