Microsoft Azure Event Hubs is a managed platform component of Azure IoT services that provides a telemetry data ingestion at cloud scale with low latency and high reliability.
For your Internet of Things (IoT) scenarios, you can think of Event Hubs as the loosely-coupled beginning of an event pipeline that sits between event publishers like sensors and event consumers like Azure Stream Analytics. Unlike queues, Event Hubs implement partitions (shards) to support massive horizontal scale for the processing of millions of events per second. Consumer Groups provide consuming applications an independent view of the Event Hub from which to read the telemetry streams that can lead to complex event processing, storage or other downstream services.
Now that you have a brief summary of this event ingestion technology, it’s time to step through the creation of your own Event Hub so you can start bringing your IoT scenarios to life.
Go to your Azure Portal and click the Service Bus icon on the left side of the page as shown below:
If you have an existing Service Bus namespace, then you can reuse it. Otherwise, click Create a New Namespace.
The Create a Namespace dialog will pop up on your screen as shown below:
In this dialog you will enter a unique Namespace Name, select a Region, select a Subscription to bill against, choose Messaging as the Type in order to support Event Hubs and choose Standard as the Messaging Tier. This allows you to support a sufficient number of Brokered connections (AMQP) into the Event Hub and up to 20 consumer groups leading out of the Event Hub. Click the checkbox when you’re done.
With your Service Bus namespace created, click on the appropriate highlighted row as shown below:
Click Event Hubs from one of the choices across the top of the page to bring up the page shown below:
Click Create a New Event Hub.
Select Quick Create to which should be sufficient for most IoT scenarios.
Enter a unique Event Hub Name, select the same Region as your Service Bus Namespace, select a Subscription to bill against, select the Service Bus Namespace you previously created and then click the Create a New Event Hub checkbox.
With your Event Hub created, click on the appropriate highlighted row as shown below:
Click Configure from one of the choices across the top of the page to bring up the page shown below:
The Message Retention text box allows you to configure the number of days you’d like to have your messages retained in the Event Hub with a default of one day. The Event Hub State combo box allows you to enable or disable your Event Hub. Following the Quick Create path gave you a Partition Count of 16. This value is not changeable once it’s been set so you might consider a Custom Create of your Event Hub if you need a different value. Partitions refer to a scale unit where each one supports message ingress of 1 MB/sec and an egress of 2 MB/sec. You can set the number of Event Hub throughput units on your Service Bus Scale page. The default value is set to one.
In your next configuration step, you will create two shared access policies to facilitate security on your message ingress and egress as shown below:
Click into the Name textbox and enter an ingress name then select the Permissions combo box and select Send. Repeat the process on the newly created row below by adding an egress name and then select Manage, Send, and Listen from the combo box. Click the Save icon at the bottom of the page and then you’ll notice that shared access keys are generated for both your message ingress and egress policies. Those keys will be used to create the connection strings used by your IoT devices, gateways and event consumers like Azure Stream Analytics.
To view and use those connection strings, click Dashboard at the top of the page and then click the Connection Information key icon at the bottom of the page to bring up the Access connection information dialog as shown below:
This is where you will go to copy the Shared Access Signature (SAS) key connection strings into your code to authenticate access to entities within the namespace. The authentication and security model ensures that only devices that present valid credentials can send data to an Event Hub. It also ensures that one device cannot impersonate another device. Lastly, it prevents a rogue device from sending data to an Event Hub by blocking it. Of course, all communication between devices and Event Hubs occurs over TLS.
To wrap things up, click Consumer Groups from one of the choices across the top of the page to bring up the page shown below:
Rather than using the $Default Consumer Group, it’s a good idea to specify one or more of them yourself to create views of the Event Hub that will be used by things Steam Analytics. This is a simple process that starts with clicking the + Create icon at the bottom of the page.
The Create a Consumer Group dialog will pop up on your screen as shown below:
Type in a meaningful name in the Consumer Group Name textbox and then click the checkbox to save and exit.
Some of you may be wondering why do you need to use Event Hubs for event ingestion when you’ve been uploading data from disparate clients to servers using SOAP + XML and REST + JSON for more than a decade. The answer has to do with wire protocol efficiency and reliability. By default, Event Hubs use the Advanced Message Queuing Protocol (AMQP) which is an OASIS standard. This is a binary, peer-to-peer, wire protocol designed for the efficient, reliable exchange of business messages that got its start on Wall Street. If it’s good enough for the critical financial transactions between the world’s largest investment banks and stock exchanges, I’m pretty sure it’s good enough for the rest of us.
At this point, your Event Hub should be up and running. The next step is to get a device sending telemetry into your Event Hub so you can see it working. See you next time.