Delta lake has transaction logs that contain information on the writes happening at deltalake. Is there some kind of audit logs for reading queries happening on Deltalake?
Something like: notebook X issued a read on Deltalake at time T.
Use Case: This information will help me understand the lineage between datasets and associated notebooks involved in the lineages.
P.S: My deltalake is stored in ADLS (mounted in databricks)
As the definitive record of every change ever made to a table, the Delta Lake transaction log offers users a verifiable data lineage that is useful for governance, audit and compliance purposes. It can also be used to trace the origin of an inadvertent change or a bug in a pipeline back to the exact action that caused it.
Users can run DESCRIBE HISTORY to see metadata around the changes that were made.
For more details, refer to Diving Into Delta Lake: Unpacking The Transaction Log.
