Getting Started with Azure IoT services: Event Hubs

Microsoft Azure Event Hubs is a managed platform component of Azure IoT services that provides a telemetry data ingestion at cloud scale with low latency and high reliability.

For your Internet of Things (IoT) scenarios, you can think of Event Hubs as the loosely-coupled beginning of an event pipeline that sits between event publishers like sensors and event consumers like Azure Stream Analytics. Unlike queues, Event Hubs implement partitions (shards) to support massive horizontal scale for the processing of millions of events per second. Consumer Groups provide consuming applications an independent view of the Event Hub from which to read the telemetry streams that can lead to complex event processing, storage or other downstream services.

Event Hub Graphic

Now that you have a brief summary of this event ingestion technology, it’s time to step through the creation of your own Event Hub so you can start bringing your IoT scenarios to life.

Go to your Azure Portal and click the Service Bus icon on the left side of the page as shown below:

Create Service Bus Namespace

If you have an existing Service Bus namespace, then you can reuse it. Otherwise, click Create a New Namespace.

The Create a Namespace dialog will pop up on your screen as shown below:

Create Namespace Dialog

In this dialog you will enter a unique Namespace Name, select a Region, select a Subscription to bill against, choose Messaging as the Type in order to support Event Hubs and choose Standard as the Messaging Tier. This allows you to support a sufficient number of Brokered connections (AMQP) into the Event Hub and up to 20 consumer groups leading out of the Event Hub.  Click the checkbox when you’re done.

With your Service Bus namespace created, click on the appropriate highlighted row as shown below:

Service Bus Created

Click Event Hubs from one of the choices across the top of the page to bring up the page shown below:

Create Event Hub

Click Create a New Event Hub.

Select Quick Create to which should be sufficient for most IoT scenarios.

Create Event Hub Quick Create

Enter a unique Event Hub Name, select the same Region as your Service Bus Namespace, select a Subscription to bill against, select the Service Bus Namespace you previously created and then click the Create a New Event Hub checkbox.

With your Event Hub created, click on the appropriate highlighted row as shown below:

Event Hub Created

Click Configure from one of the choices across the top of the page to bring up the page shown below:

EventH ub Configure

The Message Retention text box allows you to configure the number of days you’d like to have your messages retained in the Event Hub with a default of one day.  The Event Hub State combo box allows you to enable or disable your Event Hub.  Following the Quick Create path gave you a Partition Count of 16.  This value is not changeable once it’s been set so you might consider a Custom Create of your Event Hub if you need a different value.  Partitions refer to a scale unit where each one supports message ingress of 1 MB/sec and an egress of 2 MB/sec.  You can set the number of Event Hub throughput units on your Service Bus Scale page.  The default value is set to one.

In your next configuration step, you will create two shared access policies to facilitate security on your message ingress and egress as shown below:

SharedAccessPoliciesClick into the Name textbox and enter an ingress name then select the Permissions combo box and select Send.  Repeat the process on the newly created row below by adding an egress name and then select Manage, Send, and Listen from the combo box.  Click the Save icon at the bottom of the page and then you’ll notice that shared access keys are generated for both your message ingress and egress policies.  Those keys will be used to create the connection strings used by your IoT devices, gateways and event consumers like Azure Stream Analytics.

To view and use those connection strings, click Dashboard at the top of the page and then click the Connection Information key icon at the bottom of the page to bring up the Access connection information dialog as shown below:

Connection Strings

This is where you will go to copy the Shared Access Signature (SAS) key connection strings into your code to authenticate access to entities within the namespace. The authentication and security model ensures that only devices that present valid credentials can send data to an Event Hub. It also ensures that one device cannot impersonate another device. Lastly, it prevents a rogue device from sending data to an Event Hub by blocking it. Of course, all communication between devices and Event Hubs occurs over TLS.

To wrap things up, click Consumer Groups from one of the choices across the top of the page to bring up the page shown below:

Consumer Groups

Rather than using the $Default Consumer Group, it’s a good idea to specify one or more of them yourself to create views of the Event Hub that will be used by things Steam Analytics.  This is a simple process that starts with clicking the + Create icon at the bottom of the page.

The Create a Consumer Group dialog will pop up on your screen as shown below:

Parking Group

 

Type in a meaningful name in the Consumer Group Name textbox and then click the checkbox to save and exit.

Some of you may be wondering why do you need to use Event Hubs for event ingestion when you’ve been uploading data from disparate clients to servers using SOAP + XML and REST + JSON for more than a decade.  The answer has to do with wire protocol efficiency and reliability.  By default, Event Hubs use the Advanced Message Queuing Protocol (AMQP) which is an OASIS standard.  This is a binary, peer-to-peer, wire protocol designed for the efficient, reliable exchange of business messages that got its start on Wall Street.  If it’s good enough for the critical financial transactions between the world’s largest investment banks and stock exchanges, I’m pretty sure it’s good enough for the rest of us.

At this point, your Event Hub should be up and running. The next step is to get a device sending telemetry into your Event Hub so you can see it working. See you next time.

Rob

Microsoft Office is Now Available for the iPad

Today we made the highly anticipated move to bring Microsoft Word, Excel and PowerPoint to the Apple iPad.

All three products can be downloaded for free from the Apple App Store and used to view existing documents.  Via an in-app purchase mechanism, iPad users can unlock the full functionality of the products by subscribing to Office 365 for $99 per year.

ExcelExcel for iPad

Word

Word for iPad

PowerPoint

PowerPoint for iPad

Take it for a spin.

Rob

Happy Mobile New Year 2014

It’s a new year and therefore it’s time to throw some rapid-fire mobile architecture concepts at you to help improve your mobile enterprise systems.

iPaq

  • For starters, developers and IT Pros involved in mobility need to start thinking holistically about backend systems, data transports, and mobile endpoints.  Working in the mobile enterprise space doesn’t mean you just get to be a “device guy.”  Remember, your CIO wants to empower her employees by moving data from any backend system out to any device.
  • Here’s my obligatory statement on keeping things simple.  Large enterprise mobile systems will get complicated enough on their own without you contributing to the chaos.
  • Take the idea of Mobile SOA seriously and wrap your enterprise in a Web API.  Your various backend ERP, CRM, and custom systems use a variety of proprietary wire protocols to communicate.  You need to make all those backend systems “mobile friendly” by having them all speak a device-agnostic language.  You all know I’ve been a REST guy for more than a decade so it’s no surprise that I want you to map those proprietary APIs to RESTful APIs.  REST is lighter then SOAP and works with all mobile devices.  For your coarse-grained APIs that send larger payloads of information, make sure to serialize that data as JSON.  JSON is lighter than XML and works with all mobile devices.  The other litmus test here is around simplicity.  If a webpage with JavaScript can consume your REST/JSON API without needing an SDK to interpret your wire-protocol, then native apps will find your API simpler to work with than a system that requires an SDK.  Remember, you’re not just building this API for mobile devices.  You’re building it for developers.  If it’s not super-simple to work with, they may not use it.
  • Some backend systems that you’re building Web APIs for are designed for serious web-scale, but most are not.  If a particular backend package already has the performance and scalability to handle millions of devices, then have your middleware Web API servers sit directly in front of them and map REST API calls directly to their proprietary APIs.  For the the majority of backend systems that are wholly incapable  of handling an onslaught of millions of devices, you need to give them some help.  You’ll need to create a server facade in front of them via a database server that includes EAI adapter capabilities along with the ability to scale out via replication.  You’ll use an adapter designed specifically to interface with the backend system in question to prefetch data from the proprietary APIs and put that data in staging tables in the database.  You’ll then shard this database via replication to scale either complete databases, tables, or parts of tables to n number of “shared-nothing” database nodes on different servers.  In this scenario, your middleware Web API servers will make calls to the various database shards to return cached data to mobile devices in order to solve your scalability problem for backend systems that need help in this department.
  • Common sense tells us that if you make your backend systems faster, then the user experience on the mobile devices will improve because users won’t be waiting around for data to arrive to fill their screen.  The first and easiest thing I want you to do is switch all your spinning hard drives to solid state drives (SSDs).  Moving all your backend systems, database servers, middleware servers, and web servers to lightning-fast SSDs will instantly boost the performance of your entire system without writing or changing a line of code.  This is a no-brainer.  The next thing I want you to do is take better advantage of cheap RAM and in-memory operations.  I want every mapped API call between your middleware Web API servers and backend systems to be cached in-memory.  If you’re using a server facade in front of backend systems, ensure the EAI/ETL capabilities of the database support in-memory data movement operations.  Likewise, I want you to use database servers that support in-memory OLTP.  All the Web API calls between your mobile devices and the middleware servers also need to be cached in-memory as well.  Last but not least, the native and web apps on your mobile devices need to cache data offline in a local data store so that users can keep working without needing to constantly call your Web APIs.
  • Implement a lightweight Enterprise Mobility Management (EMM) system in your organization.  It needs to provide things like an enterprise app store, security policy enforcement, and things to make an employee’s life easier like the simple provisioning of Wi-Fi, VPN, and email profiles.  In order for mobile apps to access the Web API you’ve built, your EMM should also support a secure gateway to access internal corporate servers via single sign-on (SSO).
  • Finally, keep investigating HTML5.  It’s the world’s most important shared technology that isn’t owned by any one company.  It works with the browsers on every mobile device and makes things like app deployment and updating easier than equivalent processes with native apps.  It supports offline operations and even includes two options of local data storage.  Web pages no longer submit/post data to themselves or other pages but use JavaScript instead.  Mobile web apps can make AJAX calls to you Web API and easily work with the JSON data that’s returned.  Mobile browsers keep getting faster and use optimizations like hardware acceleration and JIT-compiling of JavaScript to blur the lines with native apps.

Happy New Year!
Rob

Keeping Windows 8 Tablets in Sync with SQL Server 2012

I’m pleased to announce that my newest book, “Keeping Windows 8 Tablets in Sync with SQL Server 2012,” is now available for sale on Amazon.

Spending a decade travelling the globe to help the world’s largest companies design and build mobile solutions had taught me a few things.  Large organizations are not interested in constantly running on the new technology hamster wheel.  They prefer to leverage existing investments, skills, and technologies rather than always chasing the next big thing.  Don’t believe me?  Take mobile and the cloud for example:

  • In 2003 I was building Pocket PC solutions for large companies that wirelessly connected apps on those devices to SAP.  I assumed mobile was going mainstream that year.  I was wrong.  I was early.  Mobile apps wouldn’t explode until the end of the decade with the iPhone 3G.
  • In 2004, my partner Darren Flatt and I launched the first cloud-based mobile device management (MDM) company to facilitate software distribution and policy enforcement on early smartphones and handhelds.  Early again.  MDM didn’t get big until the end of the decade.
  • At PDC in 2008, my company launched our cloud offering called Azure.  We skipped directly to the developer Nirvana called Platform as a Service (PaaS).  I spent a few years doing nothing but speaking and writing about Windows Phones communicating with Web Roles.  Turns outs companies wanted to take smaller steps to the cloud by uploading their existing servers as VMs.

Being early over and over again taught me how the real world of business operates outside of Redmond and Silicon Valley.  Businesses need to make money doing what they do best.  Where appropriate, they will use technology to help them improve their processes and give them a competitive advantage.  So let’s cut to the chase and talk about why I wrote my new book:

  • Tablets and Smartphones are taking over the world of business and outselling laptops and desktops.  This is a well-known fact and not speculation on my part.
  • There are 1.3 billion Windows laptops, tablets, and desktops being used all over the world.  Windows 7 is in first place with Windows XP in second.
  • Companies run their businesses on Microsoft Office combined with tens of millions of Win32 apps they created internally over the last 2 decades.  Intranet-based web apps also became a huge force starting in the late 90s.
  • Tools like Visual Basic, Access, PowerBuilder, Java, and Delphi made it easy to rapidly build those Win32 line of business apps in the 90s and helped ensure the success of Windows in the enterprise.
  • Many of those developers moved to VB and C# in the 2000s to build .NET Windows Forms (WinForms) apps that leveraged their existing Visual Basic skills from the 90s.
  • Some businesses built Service Oriented Architecture (SOA) infrastructures of Web Services based on SOAP and XML over the last decade in order to connect mobile devices to their servers.  Most business did not, and instead opted for out-of-the-box solutions that didn’t require them to write a lot of code so they could get to market faster.
  • While the “white collar” enterprise recently started building business apps for the iPhone and iPad, the “blue collar” enterprise has been building WinForms apps for rugged Windows Mobile devices using the .NET Compact Framework and a mobile database called SQL Server Compact for over a decade.
  • Most businesses run servers in their own data centers.  Many of them are using virtualization technologies like Hyper-V and VMware to help them create a private cloud.
  • Of the businesses that have dipped their collective toes in the public cloud for internal apps, most of them are following the Infrastructure as a Service (IaaS) model where they upload their own servers in a VM.  Just look at the success of Amazon and the interest in Azure Infrastructure Services.

So the goal of my new book is to help businesses transition to the tablet era in a way that respects their existing investments, skills, technologies, enterprise security requirements, and appetite for risk.

Windows 8 Book Front

Since I’ve been involved in countless mobile projects where companies used the Microsoft data sync technologies already baked into SQL Server and SQL Server Compact, I decided to illustrate how to virtualize this sync infrastructure with Hyper-V.  With an eye towards existing trends that are widely embraced, this gives businesses the flexibility to use this proven technology in a private, public, or hybrid cloud.  Companies authenticate their employees against the same Active Directory they’ve used for over a decade.  I’m deadly serious about security and you’ll be glad to know the technology in this book handles it at every tier of your solution with Domain credentials plus encrypted data-at-rest and data-in-transit.  You also have the option of synchronizing mobile data with any edition of SQL Server 2005, 2008 or 2012 using Microsoft sync technologies that takes care of all data movement plumbing.  Your development team avoids writing thousands of lines of code to create web services, sync logic, change tracking, error handling, and retry logic.  With Microsoft lowering risk to your project by taking care of the server backend, security, and data sync technologies, your team can focus on building the best possible Windows 8 tablet app for the enterprise.

Speaking of tablet app development, it’s important to show you a path that doesn’t force you to learn all-new tools or programming languages, frameworks, or paradigms.  As a developer, you get to keep using Visual Studio along with the Desktop WinForms skills you’ve mastered over the last decade.  Better still, you can accomplish everything using the free version of Visual Studio 2012.  While you might be thinking Windows 8 tablet solutions must be created via Windows Store apps, this is not the case.  Instead, I show you how to apply Modern UI principles to Desktop WinForms apps that are full-screen and touch-first.  Concepts like content over chrome, use of typography, and UI elements with large hit targets are all covered in detail.  I also respect your investment in Windows 7 laptops and tablets by ensuring your touch apps are backwards compatible and keyboard + mouse/trackpad friendly.

Windows 8 Book Back

If you’re looking to build a new Windows 8 tablet app using what you have and what you know, this book is for you.  If you’re looking to port an existing Windows XP or Windows Mobile WinForm app to a Windows 8 tablet, this book empowers you with the skills to make your porting effort a successful one.

The takeaway is you don’t have to scrap your existing investments to participate in the tablet revolution.  I purposely made the book low-cost, hands-on, short, and to-the-point so you can rapidly build mobile solutions for Windows 8 tablets instead of wasting your time with theory.  Take “Keeping Windows 8 Tablets in Sync with SQL Server 2012″ for a spin so you can start building mobile apps for the world’s first and only enterprise-class tablet today.

Stay in Sync!

-Rob

Windows Azure Infrastructure Services are Live

I’m pleased to announce that Windows Azure fully supports Infrastructure as a Service (IaaS).

This new service now makes it possible for companies to move their existing servers and applications into the cloud.  We understand that customers don’t want to rip and replace their current infrastructure to benefit from the cloud; they want the strengths of their on-premises investments and the flexibility of the cloud.  It’s not only about Infrastructure as a Service (IaaS) or Platform as a Service (PaaS), it’s about Infrastructure Services and Platform Services and hybrid scenarios.  The cloud should be an enabler for innovation and Windows Azure can now be an extension of your organization’s IT fabric.

The Windows Azure Virtual Machines and Windows Azure Virtual Network are now available to help you meet your changing business needs by providing an on-demand, scalable infrastructure.  Not only can these VMs support up to 8 CPU cores, but we’ve added higher memory instances that include up to 56 GB of RAM.  These infrastructure services allow you to extend your data centers and business-critical workloads into the cloud while leveraging your existing skills and investments.

Today we are also announcing a commitment to match Amazon Web Services prices for commodity services such as compute, storage and bandwidth.  This starts with reducing our GA prices on Virtual Machines and Cloud Services by 21-33%.  Windows Azure is now your most price-competitive cloud option.  At the same time, Microsoft provides you a financially backed 99.95% monthly SLA when you deploy multiple instances of Virtual Machines.

Not only are prebuilt Linux images such as Ubuntu, CentOS, and Suse Linux Enterprise Server available through the Windows Azure Virtual Machine Image Gallery, but so is Windows Server 2012, SQL Server 2012, BizTalk Server 2013 and SharePoint Server 2013.  We also provide server support for Dynamics GP 2013, Dynamics NAV 2013, Forefront Identity Manager 2010 R2 SP1, Project Server 2013, System Center 2012 SP1, and Team Foundation Server 2012.

On a personal note, I’m happy to see this breathe new life into all the mobile data sync solutions that have been deployed in data centers all over the world.  You’ll now be able to take advantage of Windows Azure VMs so all your mobile devices running SQL Server Compact can synchronize business data with SQL Server in the cloud.

-Rob