how to Set File Expiry on file Azure Datalake gen1 file at creation time?

Question

Using Azure Data Analytics to run U-SQL queries to write csvs in Azure Data Factory, how can I get an expiry value set on the files?

I have a Azure Service Fabric app with a service that writes json files to azure data lakes. In this write process using the SDK, I'm able to apply an expiry value to those json files. I then go on to process those files further using ADLA with U-SQL in ADF. From what I read in older posts, U-SQL doesn't support setting the expire time. However is it possible to set the expire time through other activities in the Factory? or am I going to need my service to monitor and update expiry values?

Answer 1

Unfortunately, Azure Data Lake Gen1 don’t have ability to set expiration of files when the file is created and U-SQL doesn't support setting the expire time.

I would suggest you to vote up an idea submitted by another Azure customer.

https://feedback.azure.com/forums/270578-data-factory/suggestions/20308555-setting-expiration-in-azure-data-lake-store

https://feedback.azure.com/forums/327234-data-lake/suggestions/17579557-support-expiration-of-files-in-adls

All of the feedback you share in these forums will be monitored and reviewed by the Microsoft engineering teams responsible for building Azure.