With a nice foundation of Event Publishers & Handlers, we can expect a vast amount of new ones being added that will be available out of the box in the future.
Currently Azure Event Grid only supports events in JSON, but in the future they will support other data formats such as binary.
Let’s have a look at what I’d like to see being added.
High-Level Event Grid Monitoring
I’m curious to see what the operations & monitoring story will be for Azure Event Grid.
In the following sections I will refer to other Azure services/technologies that provide similar features but I’d love to see Azure Event Grid expose those capabilities out of the box instead of integrating with Azure Application Insights for example.
This would allow us to have one centralized monitoring experience for everything related to Azure Event Grid instead of having this in another service (aka “Dependency”), since this service is running as an infrastructure service supporting other services.
High-Level Component Overview
Personally, I think it would be great to have a dashboard that shows me all the Event Publishers, Topics, Subscriptions & Handlers that are connected with each other.
My vision on this is comparable with the monitoring experience that Azure Data Factory provides:
While this is only the pipeline overview, it clearly indicates how each data set, service and pipeline are connected which each other. If you go to the Monitoring-dasbhoard it also provides you an overview of all processed data slices.
It would be nice to have a similar experience for Azure Event Grid where every failed event is listed so we can view the body of the event. This also enables us to troubleshoot why the event failed and if it’s related to the content of the event or not. That said, since Azure Event Grid is a high-volume service I’m not hoping for this one. However, it would be nice to have as a premium feature at least.
Another interesting feature to have is a real-time sense of the throughput of all the events in the grid, something similar to Netflix’ Vizceral (GitHub).
Performance metrics per Event Handler
Next to the high-level component overview it would be great to have some integrated performance gauges.
These gauges allow us to gain insights on the processing performance of Event Handlers allowing us to pinpoint scalability problems.
This can be comparable to what Azure Application Insights “Application Map” offers where you can see the amount of requests, success rate and failures:
Adding metadata to events
Introducing a metadata-node in the event payload would be great as well. This would allow us to specify additional context information about the event, while it’s not business specific information.
By doing this; we can add telemetry information, such as correlation ids, allowing us to correlate all our telemetry across all the Event Publishers, Topics, Handlers and downstream systems.
Taking it a step further, it would be nice to use “Application Insights Analytics” (aka Kusto). This provides us to search for these events and correlate the route it took in Azure Event Grid.
Integration with Azure Data Factory
Thé Event Handler I’m looking forward to is Azure Data Factory. As of today, Azure Data Factory only supports a slicing model where it will trigger your pipeline every 1 hour, day, week, etc while in several scenarios this is not the best use-case.
It would be good if we could use Azure Event Grid to forward events for newly uploaded blobs and send that to the Data Factory Handler to trigger your pipeline. This can not only make the data processing flow feel more natural and also the performance could increase while we divide the processing in smaller pieces instead of running one big pipeline.
Summary
While Azure Event Grid is still in preview, it’s always good to think about ways it can be improved and how we will operate this service. We’ve talked about a few features I’d like to see being added which are mainly focused to monitoring the whole infrastructure and how we can correlate this back to our other telemetry.
My biggest wish is having a high-level overview of the Event Grid components and how they are connected (which Azure Functions also lacks).
My second request would be to have an out of the box centralized monitoring experience and not being forced to use Azure Application Insights. This would mean that we are fully depending on Application Insights which adds an unnecessary dependency; which is also not that cheap, certainly not with the telemetry this service will generate.
Does this mean that I don’t want to have integration with Azure Application Insights? No! Just not as the built-in way to operate Azure Event Grid.
This is of course early thinking, my vision on this can change once I use this more.
Thanks for reading,
Tom Kerkhove.
Subscribe to our RSS feed