🏭🎗️🗒️Microsoft Fabric: Eventstreams [Notes]

Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)! 

Last updated: 8-Mar-2025

Real-Time Intelligence architecture [4]

[Microsoft Fabric] Eventstream(s)

• {def} feature in Microsoft Fabric's Real-Time Intelligence experience, that allows to bring real-time events into Fabric
• bring real-time events into Fabric, transform them, and then route them to various destinations without writing any code 
• ⇐ aka no-code solution
• {feature} drag and drop experience 
• gives users an intuitive and easy way to create your event data processing, transforming, and routing logic without writing any code 
• work by creating a pipeline of events from multiple internal and external sources to different destinations
• a conveyor belt that moves data from one place to another [1]
• transformations to the data can be added along the way [1]
• filtering, aggregating, or enriching
• {def} eventstream
• an instance of the Eventstream item in Fabric [2]
• {feature} end-to-end data flow diagram 
• provide a comprehensive understanding of the data flow and organization [2].
• {feature} eventstream visual editor
• used to design pipelines by dragging and dropping different nodes [1]
• sources
• where event data comes from 
• one can choose
• the source type
• the data format
• the consumer group
• Azure Event Hubs
• allows to get event data from an Azure event hub [1]
• allows to create a cloud connection  with the appropriate authentication and privacy level [1]
• Azure IoT Hub
• SaaS service used to connect, monitor, and manage IoT assets with a no-code experience [1]
• CDC-enabled databases
• software process that identifies and tracks changes to data in a database, enabling real-time or near-real-time data movement [1]
• Azure SQL Database 
• PostgreSQL Database
• MySQL Database
• Azure Cosmos DB
• Google Cloud Pub/Sub
• messaging service for exchanging event data among applications and services [1]
• Amazon Kinesis Data Streams
• collect, process, and analyze real-time, streaming data [1]
• Confluent Cloud Kafka
• fully managed service based on Apache Kafka for stream processing [1]
• Fabric workspace events
• events triggered by changes in Fabric Workspace
• e.g. creating, updating, or deleting items. 
• allows to capture, transform, and route events for in-depth analysis and monitoring within Fabric [1]
• the integration offers enhanced flexibility in tracking and understanding workspace activities [1]
• Azure blob storage events
• system triggers for actions like creating, replacing, or deleting a blob [1]
• these actions are linked to Fabric events
• allowing to process Blob Storage events as continuous data streams for routing and analysis within Fabric  [1]
• support streamed or unstreamed events [1]
• custom endpoint
• REST API or SDKs can be used to send event data from custom app to eventstream [1]
• allows to specify the data format and the consumer group of the custom app [1]
• sample data
• out-of-box sample data
• destinations
• where transformed event data is stored. 
• in a table in an eventhouse or a lakehouse [1]
• redirect data to 
• another eventstream for further processing [1]
• an activator to trigger an action [1]
• Eventhouse
• offers the capability to funnel your real-time event data into a KQL database [1]
• Lakehouse
• allows to preprocess real-time events before their ingestion in the lakehouse
• the events are transformed into Delta Lake format and later stored in specific lakehouse tables [1]
• facilitating the data warehousing needs [1]
• custom endpoint
• directs real-time event traffic to a bespoke application [1]
• enables the integration of proprietary applications with the event stream, allowing for the immediate consumption of event data [1]
• {scenario} aim to transfer real-time data to an independent system not hosted on the Microsoft Fabric [1]
• Derived Stream
• specialized destination created post-application of stream operations like Filter or Manage Fields to an eventstream
• represents the altered default stream after processing, which can be routed to various destinations within Fabric and monitored in the Real-Time hub [1]
• Fabric Activator
• enables to use Fabric Activator to trigger automated actions based on values in streaming data [1]
• transformations
• filter or aggregate the data as is processed from the stream [1]
• include common data operations
• filtering
• filter events based on the value of a field in the input
• depending on the data type (number or text), the transformation keeps the values that match the selected condition, such as is null or is not null [1]
• joining
• transformation that combines data from two streams based on a matching condition between them [1]
• aggregating
• calculates an aggregation every time a new event occurs over a period of time [1]
• Sum, Minimum, Maximum, or Average
• allows renaming calculated columns, and filtering or slicing the aggregation based on other dimensions in your data [1]
• one can have one or more aggregations in the same transformation [1]
• grouping
• allows to calculate aggregations across all events within a certain time window [1]
• one can group by the values in one or more fields [1]
• allows for the renaming of columns
• similar to the Aggregate transformation 
• ⇐ provides more options for aggregation and includes more complex options for time windows [1]
• allows to add more than one aggregation per transformation [1]
• allows to define the logic needed for processing, transforming, and routing event data [1]
• union
• allows to connect two or more nodes and add events with shared fields (with the same name and data type) into one table [1]
• fields that don't match are dropped and not included in the output [1]
• expand
• array transformation that allows to create a new row for each value within an array [1]
• manage fields
• allows to add, remove, change data type, or rename fields coming in from an input or another transformation [1]
• temporal windowing functions 
• enable to analyze data events within discrete time periods [1]
• way to perform operations on the data contained in temporal windows [1]
• e.g. aggregating, filtering, or transforming streaming events that occur within a specified time period [1]
• allow analyzing streaming data that changes over time [1]
• e.g. sensor readings, web-clicks, on-line transactions, etc.
• provide great flexibility to keep an accurate record of events as they occur [1]
• {type} tumbling windows
• divides incoming events into fixed and nonoverlapping intervals based on arrival time [1]
• {type} sliding windows 
• take the events into fixed and overlapping intervals based on time and divides them [1]
• {type} session windows 
• divides events into variable and nonoverlapping intervals that are based on a gap of lack of activity [1]
• {type} hopping windows
• are different from tumbling windows as they model scheduled overlapping window [1]
• {type} snapshot windows 
• group event stream events that have the same timestamp and are unlike the other windowing functions, which require the function to be named [1]
• one can add the System.Timestamp() to the GROUP BY clause [1]
• {type} window duration
• the length of each window interval [1]
• can be in seconds, minutes, hours, and even days [1]
• {parameter} window offset
• optional parameter that shifts the start and end of each window interval by a specified amount of time [1]
• {concept} grouping key
• one or more columns in an event data use to group the data               by [1]
• aggregation function
• one or more of the functions applied to each group of events in each window [1]
• where the counts, sums, averages, min/max, and even custom functions become useful [1]
• see the event data flowing through the pipeline in real-time [1]
• handles the scaling, reliability, and security of event stream automatically [1]
• no need to write any code or manage any infrastructure [1]
• {feature} eventstream editing canvas
• used to 
• add and manage sources and destinations [1]
• see the event data [1]
• check the data insights [1]
• view logs for each source or destination [1]
• {feature} Apache Kafka endpoint on the Eventstream item
• {benefit} enables users to connect and consume streaming events through the Kafka protocol [2]
• application using the protocol can send or receive streaming events with specific topics [2]
• requires updating the connection settings to use the Kafka endpoint provided in the Eventstream [2]
• {feature} support runtime logs and data insights for the connector sources in Live View mode [3]
• allows to examine detailed logs generated by the connector engines for the specific connector [3]
• help with identifying failure causes or warnings [3]
• ⇐ accessible in the bottom pane of an eventstream by selecting the relevant connector source node on the canvas in Live View mode [3]
• {feature} support data insights for the connector sources in Live View mode [3]
• {feature} integrates eventstreams CI/CD tools
• {benefit} developers can efficiently build and maintain eventstreams from end-to-end in a web-based environment, while ensuring source control and smooth versioning across projects [3]
• {feature} REST APIs
•  allow to automate and manage eventstreams programmatically
• {benefit} simplify CI/CD workflows and making it easier to integrate eventstreams with external applications [3]
• {recommendation} use event streams feature with at least SKU: F4 [2]
• {limitation} maximum message size: 1 MB [2]
• {limitation} maximum retention period of event data: 90 days [2]

Previous Post  <<||>> Next Post 

References:

[1] Microsoft Learn (2024) Microsoft Fabric: Use real-time eventstreams in Microsoft Fabric [link]
[2] Microsoft Learn (2025) Microsoft Fabric: Fabric Eventstream - overview [link]
[3] Microsoft Learn (2024) Microsoft Fabric: What's new in Fabric event streams? [link]

[4] Microsoft Learn (2025) Real Time Intelligence L200 Pitch Deck [link]

[5] Microsoft Learn (2025) Use real-time eventstreams in Microsoft Fabric [link]

Resources:

[R1] Microsoft Learn (2024) Microsoft Fabric exercises [link]
[R2] Microsoft Fabric Updates Blog (2024) CI/CD – Git Integration and Deployment Pipeline [link]

[R3] Microsoft Learn (2025) Fabric: What's new in Microsoft Fabric? [link]

Acronyms:

API - Application Programming Interface
CDC - Change Data Capture
CI/CD  - Continuous Integration/Continuous Delivery
DB - database
IoT - Internet of Things
KQL - Kusto Query Language
RTI - Real-Time Intelligence

SaaS - Software-as-a-Service
SDK - Software Development Kit
SKU - Stock Keeping Unit

⛁DBMS: Event Streaming Databases (More of a Kafka’s Story)

Event streaming architectures are architectures in which data are generated by different sources, and then processed, stored, analyzed, and acted upon in real-time by the different applications tapped into the data streams. An event streaming database is then a database that assures that its data are continuously up-to-date, providing specific functionality like management of connectors, materialized views and running queries on data-in-motion (rather than on static data). 

Reading about this type of technologies one can easily start fantasizing about the Web as a database in which intelligent agents can process streams of data in real-time, in which knowledge is derived and propagated over the networks in an infinitely and ever-growing flow in which the limits are hardly perceptible, in which the agents act as a mind disconnected in the end from the human intent. One is stroke by the fusing elements of realism and the fantastic aspects, more like in a Kafka’s story in which the metamorphosis of the technologies and social aspects can easily lead to absurd implications.

The link to Kafka was somehow suggested by Apache Kafka, an open-source distributed event streaming platform, which seems to lead the trends within this new-developing market. Kafka provides database functionality and guarantees the ACID (atomicity, concurrency, isolation, durability) properties of transactions while tapping into data streams. 

Data streaming is an appealing concept though it has some important challenges like data overload or over-flooding, the complexity derived from building specific (business) and integrity rules for processing the data, respectively for keeping data consistency and truth within the ever-growing and ever-changing flows. 

Data overload or over-flooding occurs when applications are not able to keep the pace with the volume of data or events fired with each change. Imagine the raindrops falling on a wide surface in which each millimeter or micrometer has its own rules for handling the raindrops and this at large scale. The raindrops must infiltrate into the surface to be processed and find their way to the beneath water flows, aggregating up to streams that could nurture seas or even oceans. Same metaphor can be applied to the data events, in which the data pervade applications accumulating in streams processed by engines to derive value. However heavy rains can easily lead to floods in which water aggregates at the surface. 

Business applications rely on predefined paths in which the flow of data is tidily linked to specific actions found themselves in processual sequences that reflect the material or cash flow. Any variation in the data flow from expectations will lead to inefficiencies and ultimately to chaos. Some benefit might be derived from data integrations between the business applications, however applications must be designed for this purpose and handle extreme behaviors like over-flooding. 

Data streams are maybe ideal for social media networks in which one broadcasts data through the networks and any consumer that can tap to the network can further use the respective data. We can see however the problems of nowadays social media – data, better said information, flow through the networks being changed as fit for purposes that can easily diverge from the initial intent. Moreover, information gets denatured, misused, overused to the degree that it overloads the networks, being more and more difficult to distinguish between reliable and non-reliable information. If common sense helps in the process of handling such information, not the same can be said about machines or applications. 

It will be challenging to deal with the vastness, vagueness, uncertainty, inconsistency, and deceit of the networks of the future, however data streaming more likely will have a future as long it can address such issues by design. 

🎯Hubert Dulay - Collected Quotes

"A data fabric is a pattern that is very similar to a data mesh in that both provide solutions encompassing data governance and self-service: discovery, access, security, integration, transformation, and lineage. [...] In simple terms, a data fabric is a metadriven means of connecting disparate sets of data and related tools to provide a cohesive data experience and to deliver data in a self-service manner." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"A data fabric is an architectural approach to provide data access across multiple technologies and platforms, and is based on a technology solution. One key contrast is that a data mesh is much more than just technology: it is a pattern that involves people and processes. Instead of taking ownership of an entire data platform, as in a data fabric, the data mesh allows data producers to focus on data production, allows data consumers to focus on consumption, and allows hybrid teams to consume other data products, blend other data to create even more interesting data products, and publish these data products - with some data governance considerations in place." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"A domain has two main roles: data product engineer (or just data engineer) and the data product owner (or data product manager, or data steward). These roles can be the same or dedicated people in the domain. Data product owners must have a deep understanding of who their data consumers are, how the data is used, and what methods are used to consume the data. This will help ensure that the data products meet the needs of their use cases. Data product engineers are responsible for creating data products that are high quality, reliable, and usable by consumers. It should be possible to extend existing domain roles to include these domain roles with minimal effort." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"Consumability is a very important requirement because it will directly affect the experience domain consumers will have in a streaming data mesh. If other domains cannot easily consume streaming data products, then they may opt out of the streaming data mesh and decide to build their own integrations by hand, bypassing any issues they encounter with the data mesh. Some factors to consider when ingesting data derivatives that will affect the consumability of other domains are as follows: (*) Lack of scalability (*) Lack of interoperability" (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"Data governance creates access controls between the data product producer and consumer and provides metadata like schema definitions and lineages. In some cases, mastered data along with reference data may be relevant to the implementation. Data governance allows us to create appropriate access controls for these resources as well." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"Data governance is a set of policies, standards, processes, roles, and responsibilities that collectively ensure accountability and ownership of data across the business. Policies are the rules and regulations surrounding data defined by the business itself or, more importantly, externally by laws that, if broken, could cost a business a massive amount in fines. These policies also include enforcement of standards that enable interoperability and consumability of data between domains, especially in a decentralized data platform like a streaming data mesh. These policies are implemented as processes and controls on data by authorizing, authenticating, and safeguarding private or personal data. Policies are implemented using roles that represent groups, people, or systems to create access controls around data." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"Data lineage is the path the data took from its source origin, the stops it made along the way, and its destination. This includes information on all the systems it passed through, how it was cleansed, what it was enriched with, and how it was secured. Capturing all that metadata is difficult because many of those systems and applications don’t share information. It’s up to you to assemble the data’s path by pulling metadata from all those systems/applications and assembling them in hopes that you find the path your data took from its current location (destination) to its source system. Lineage is probably the hardest piece of metadata to acquire for either streaming or batching data pipelines." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"Data lineage provides the entire history of the data: origin, transformations, enrichments, users who engineered the transformations, etc. Consumers of the data need to trust that the data they will be using is the correct data. Data lineage provides a perspective that creates trust. It does so by mapping out the steps for policies, standards, processes, roles, and responsibilities that were involved with the sourcing, transformation, enrichment, and cleansing of the data." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"Data mesh is not completely decentralized. The data is decentralized in domains, but the mesh part of data mesh is not. Data governance is critical in building the mesh in a data mesh." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"Data tags are a simple way of providing the consuming domains with more information about the streaming data product: how it was built and what to expect when consuming it. Many of the streaming data characteristics are hard to measure, like quality and security, so it’s sometimes tough to provide that important information to the consuming domain. Instead of providing a number or a score, we can provide tags that represent levels of quality and security." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"Domain-driven design (DDD) is the methodology that helps us understand complex domain models by connecting the data model itself to core business concepts. The understanding that emerges from DDD creates a foundation to designing distributed, microservice-based, client-facing applications. DDD connects the implementation of software and its components to an evolving and ever-changing data model. The domain is the world of the business you are working with and the problems you are trying to solve. This typically involves rules, processes, and existing systems that need to be integrated as part of your solution." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"In a data mesh, data is decentralized, while in a data fabric, centralization of data is allowed. And with data centralization like data lakes, you get the monolithic problems that come with it. Data mesh tries to apply a microservices approach to data by decomposing data domains into smaller and more agile groups." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"Since domains are used to create data products, and sharing data products across many domains ultimately builds a mesh of data, we need to ensure that the data being served follows some guidelines. Data governance involves creating and adhering to a set of global rules, standards, and policies applied to all data products and their interfaces to ensure a collaborative and interoperable data mesh community. These guidelines must be agreed upon among the participating data mesh domains." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"The best approach to building self-services for domains is to follow what many SaaS services do. They follow a serverless model that is easier for their users to understand and utilize in their applications. The intention of the serverless SaaS providers is to not require their users to worry about the 'servers' that are allocated on their behalf. Users can focus more on their business rather than managing and tuning servers. This should be the same model for self-services: to make a streaming data mesh serverless so domains need to focus only on their business."(Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"To overcome ambiguous domain challenges, each domain boundary must be distinct and explicit. Business area, processes, and data that belong together need to stay together. Additionally, each data domain should belong to one, and only one, Agile or DevOps team. Data integration points within a data domain should be manageable and understood by all team members." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"We recommend making domain boundaries concrete and immutable. This helps avoid lengthy discussions about who owns what data, and also prohibits teams from freely interpreting domain boundaries to suit their own needs. Creating a domain-oriented structure is a transition - not only for data, but for people and resources. When creating domain boundaries, resources may eventually align with other teams, disrupting and evolving the current team structure. The entire concept of data mesh is just as much about resource alignment as it is about data, so the realignment of resources should not be considered a roadblock as you go through this process." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"When building a data mesh, it is necessary to enable existing engineers in a domain to perform the tasks required. Domains have to capture data from their operational stores, transform (join or enrich, aggregate, balance) that data, and publish their data products to the data mesh. Self-service services are the “easy buttons” necessary to make data mesh easy to adopt with high usability. In summary, the selfservices enable the domain engineers to take on many of the tasks the data engineer was responsible for across all lines of the business. A data mesh not only breaks up the monolithic data lake, but also breaks up the monolithic role of the data engineer into simple tasks the domain engineers can perform." (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)

"While a data mesh seeks to solve many of the same problems that a data fabric addresses - namely, the ability to address data in a single, composite data environment—the approach is different. While a data fabric enables users to create a single, virtual layer on top of distributed data, a data mesh further empowers distributed groups of data producers to manage and publish data as they see fit. Data fabrics allow for a low-to-no-code data virtualization experience by applying data integration within APIs that reside within the data fabric. The data mesh, however, allows for data engineers to write code for APIs with which to interface further. Without clearly defined boundaries, domains appear to be too interconnected, and ownership becomes either political or subject to interpretation. For instance, a large retailer most likely has multiple domains. [...]" (Hubert Dulay & Stephen Mooney, "Streaming Data Mesh", 2023)