Skip to main content

Fabric platform

New

Subscribe to OneLake data events using Azure EventGrid or Service Bus

Vote (1) Share
Jens Vestergaard's profile image

Jens Vestergaard on 03 Dec 2024 20:57:58

As announced per December 2nd 2024, there is a new capability in Real Time Hub to subscribe to data events from OneLake. See article: Unlocking the power of Real-Time Data with OneLake Events


I would like to be able to subscribe to these events from Azure, using either EventGrid or a Service Bus queue as an endpoint for the events.


Currently we are handling a lot of orchestration using EventGrid and Service Bus.

We use Event Grid to distribute messages from an Azure Storage Account (ADLS Gen2) that serves as our landing are in our ETL/ELT process.

We would have preferred to use only OneLake as our storage, but since we are not able to hook up to these events, and external (to Fabric) storage account was needed.

All of our data processing occurs in Fabric using Notebooks, but the orchestration is handled outside of Fabric because of the limitations to the Notebook queue in Fabric.

Sometimes we receive 100+ files, which each kick of a Blob Created event, which again triggers a number of processes, to identify, filter, and classify the source and destination for said file.

Some of these 100+ files have dependencies, some don't, and this is why we can't just bulk load them using the runMultiple functionality in Fabric - every file (event) needs to be handled individually.


Having the ability to subscribe to OneLake data events would allow any outside third party application to adopt to OneLake with little or no change.

In our situation, this would mean that our landing area could reside inside of OneLake, together with the rest of the data and not sit outside in an external Storage account