Microsoft Azure Event Hubs is a managed platform component of Azure #IoT services that provides a telemetry data ingestion at cloud scale with low latency and high reliability.
For your Internet of Things (IoT) scenarios, you can think of Event Hubs as the loosely-coupled beginning of an event pipeline that sits between event publishers like sensors and event consumers like Azure Stream Analytics. With industry analysts predicting tens of billions of “Things” sending telemetry over the Internet in the coming years, most data ingestion solutions won’t be able to handle the onslaught of information. Event Hubs and Azure are designed for this very scenario. Unlike queues, Event Hubs implement partitions (shards) to support massive horizontal scale for the processing of a million events per second. Consumer Groups provide consuming applications an independent view of the Event Hub from which to read the telemetry streams that can lead to complex event processing, storage or other downstream services.
Now that you have a brief summary of this event ingestion technology, it’s time to step through the creation of your own Event Hub so you can start bringing your IoT scenarios to life.
Go to your Azure Portal and click the Service Bus icon on the left side of the page as shown below:
If you have an existing Service Bus namespace, then you can reuse it. Otherwise, click Create a New Namespace.
The Create a Namespace dialog will pop up on your screen as shown below:
In this dialog you will enter a unique Namespace Name, select a Region, select a Subscription to bill against, choose Messaging as the Type in order to support Event Hubs and choose Standard as the Messaging Tier. This allows you to support a sufficient number of Brokered connections (AMQP) into the Event Hub and up to 20 consumer groups leading out of the Event Hub. Click the checkbox when you’re done.
With your Service Bus namespace created, click on the appropriate highlighted row as shown below:
Click Event Hubs from one of the choices across the top of the page to bring up the page shown below:
Click Create a New Event Hub.
Select Quick Create to which should be sufficient for most IoT scenarios.
Enter a unique Event Hub Name, select the same Region as your Service Bus Namespace, select a Subscription to bill against, select the Service Bus Namespace you previously created and then click the Create a New Event Hub checkbox.
With your Event Hub created, click on the appropriate highlighted row as shown below:
Click Configure from one of the choices across the top of the page to bring up the page shown below:
The Message Retention text box allows you to configure the number of days you’d like to have your messages retained in the Event Hub with a default of one day. The Event Hub State combo box allows you to enable or disable your Event Hub. Following the Quick Create path gave you a Partition Count of 16. This value is not changeable once it’s been set so you might consider a Custom Create of your Event Hub if you need a different value. Partitions refer to a scale unit where each one supports message ingress of 1 MB/sec and an egress of 2 MB/sec. You can set the number of Event Hub throughput units on your Service Bus Scale page. The default value is set to one.
In your next configuration step, you will create two shared access policies to facilitate security on your message ingress and egress as shown below:
Click into the Name textbox and enter an ingress name then select the Permissions combo box and select Send. Repeat the process on the newly created row below by adding an egress name and then select Manage, Send, and Listen from the combo box. Click the Save icon at the bottom of the page and then you’ll notice that shared access keys are generated for both your message ingress and egress policies. Those keys will be used to create the connection strings used by your IoT devices, gateways and event consumers like Azure Stream Analytics.
To view and use those connection strings, click Dashboard at the top of the page and then click the Connection Information key icon at the bottom of the page to bring up the Access connection information dialog as shown below:
This is where you will go to copy the Shared Access Signature (SAS) key connection strings into your code to authenticate access to entities within the namespace. The authentication and security model ensures that only devices that present valid credentials can send data to an Event Hub. It also ensures that one device cannot impersonate another device. Lastly, it prevents a rogue device from sending data to an Event Hub by blocking it. Of course, all communication between devices and Event Hubs occurs over TLS.
To wrap things up, click Consumer Groups from one of the choices across the top of the page to bring up the page shown below:
Rather than using the $Default Consumer Group, it’s a good idea to specify one or more of them yourself to create views of the Event Hub that will be used by things Steam Analytics. This is a simple process that starts with clicking the + Create icon at the bottom of the page.
The Create a Consumer Group dialog will pop up on your screen as shown below:
Type in a meaningful name in the Consumer Group Name textbox and then click the checkbox to save and exit.
Some of you may be wondering why do you need to use Event Hubs for event ingestion when you’ve been uploading data from disparate clients to servers using SOAP + XML and REST + JSON for more than a decade. The answer has to do with wire protocol efficiency and reliability. By default, Event Hubs use the Advanced Message Queuing Protocol (AMQP) which is an OASIS standard. This is a binary, peer-to-peer, wire protocol designed for the efficient, reliable exchange of business messages that got its start on Wall Street. If it’s good enough for the critical financial transactions between the world’s largest investment banks and stock exchanges, I’m pretty sure it’s good enough for the rest of us.
At this point, your Event Hub should be up and running. The next step is to get a device sending telemetry into your Event Hub so you can see it working. To test this out, I’ll walk you through the creation of a simple Windows console application.
To get started, create a new C# Console Application in Visual Studio 2013 and call it ContosoIoTConsole as shown below:
In the Solution Explorer, right-click on References and select Manage NuGet Packages…
In the Search Online box type Azure Service Bus.
Install Microsoft Azure Service Bus version 2.6.1 or later.
After that, right-click on References again and add a reference to System.Configuration so your application can read from configuration files.
In the Solution Explorer, open the App.config file. You’ll notice that it’s already filled with various Service Bus extensions. I want you to scroll down to the appSettings section at the bottom where you’ll see the beginnings of a Service Bus connection string waiting to be filled-in with your specific Event Hub data as shown below:
Replace [your namespace] with the name of the Service Bus Namespace you created in the Azure portal. I called my namespace ContosoIoT.
As you slide across to the right, you’ll see SharedAccessKeyName=. I want you to replace RootManageSharedAccessKey with the name of the data ingress shared access policy you created in your Event Hub. I named mine TelemetrySender.
In order to replace [your secret] with the correct value, go to the Dashboard page of your Event Hub and click the Connection Information key at the bottom of the page. A dialog containing access connection information with connection strings will appear. Copy the connection string from the data ingress shared access policy you created and paste it into notepad because it contains too much information. Just copy the SharedAccessKey value at the end of the connection string into [your secret] and then save and close the file.
Hopefully along the way you noticed that you can just paste the entire connection string into the value to get the same result as the direction above.
Keep in mind that when you deploy your individual devices to production, they won’t all be using this same key like you’re doing now for this test scenario. SAS tokens based on the shared access policies must be created and used by each device sending data to Event Hubs.
Now it’s time to jump in and write some code. Open Program.cs and add:
using Microsoft.ServiceBus.Messaging; using System.Configuration;
with all the other using statements found above the namespace.
The actual three lines of code needed to send the IoT equivalent of “Hello World” to your Event Hub is shown below:
First you grab the connection string you created in App.config. Next, you create an EventHubClient based on the connection string and the name of your Event Hub. Lastly, you call the Send method to pass along encoded event data as AMQP. In this scenario you’re only sending a simple string but you can send classes as well.
Run this console app several times and then wait a few minutes before checking the Event Hub dashboard in your browser since it doesn’t update in real time. Verify that your “Hello IoT” messages made it to their destination. Congratulations!
You’re now up and running with the basics of high-speed, high-scale telemetry ingestion in Azure for all your IoT and M2M scenarios. Now it’s time to move from a simple “Hello IoT” example to something more real-world like a street parking scenario found in a smart city.
One feature of Smart Cities is to help drivers find free parking spaces on city streets using their smartphones, tablets or in-car navigation apps. This is accomplished by embedding low-power, magnetic sensors in the streets near the curbs where free or metered parking spots are available. These sensors detect the absence or presence of a large metal object above them and relay this Boolean (Yes/No) state via a low-power, 6LoWPAN mesh network to a nearby field gateway that’s probably mounted on a street light.
Modelling this data via your existing console app is trivial and only requires the addition of a class + minimal code to hydrate an object with data and serialize it for transport. To get started, return to your ContosoIoTConsole solution in Visual Studio, right-click on References and add a reference to Newtonsoft.Json to support serializing your new class as JSON.
Next up, right-click on your existing ContosoIoTConsole project and add a public class called StreetParking that looks like the code shown below:
For this example, you’re just going to model a single street block and GPS coordinates with four available parking spaces to choose from.
Jumping back to the Program class from the previous Hello World example, you’ll be re-using the connectionString and EventHubClient code at the top and bottom of Main() below:
Since you’ll be serializing your StreetParking class as JSON, add using Newtonsoft.Json; above the namespace with all the other using statements.
The new code you’ll add above includes instantiating a new StreetParking object, hydrating all its properties with data, serializing the object as a JSON string and then sending the data to your Event Hub. With these code additions made, run your console app a few times to verify that your street parking event arrived in the Event Hub.
Sharing my knowledge and helping others never stops, so connect with me on my blog at https://robtiffany.com , follow me on Twitter at https://twitter.com/RobTiffany and on LinkedIn at https://www.linkedin.com/in/robtiffany