As you can see from the picture above:

1. For the Management Tools Critical Capability, there is no Cloud-based device management solution, policy-enforcement, or software distribution solution from Microsoft for Android. As I mentioned in last week’s post, consumer software distribution comes from the Android Market and the enterprise equivalent is facilitated via internal web servers and user-clickable URLs. Since Android is a wide-open system, competing markets and app stores are on the rise from Amazon and others.
2. For both the Client and Server Integrated Development Environment (IDE) and Multichannel Tool Critical Capability, Android uses Visual Studio. Endpoint development consists of HTML5, ECMAScript 5, and CSS3 delivered by ASP.NET via Web Roles. WCF REST + JSON Web services can also be created and consumed via Ajax calls from the browser. On the Cloud side of things, the Windows Azure SDK plugs into Visual Studio and provides Android developers with everything they need to build Cloud applications. It includes a Cloud emulator to simulate all aspects of Windows Azure and AppFabric on their development computer. In scenarios where native development is required by the customers, the Windows Azure Toolkit for Android can be used to allow Java via Eclipse to securely communicate with the Microsoft cloud.
3. For the cross-platform Application Client Runtime Critical Capability, Android uses the WebKit browser called Chrome to provide HTML5 + CSS3 + ECMAScript5 capabilities. Offline storage is important to keep potentially disconnected Android smartphones and tablets working and this is facilitated by Web Storage which is accessible via JavaScript.
4. For the Security Critical Capability, Android 3.0 and higher provides hardware encryption based on the user’s device passcode for data-at-rest. Data-in-transit is secured via SSL and VPN. LDAP API support allows it to access corporate directory services. Auth in the Microsoft cloud is handled via the Windows Azure AppFabric Access Control Service (ACS).
5. For the Enterprise Application Integration Tools Critical Capability, Android can reach out to servers directly via Web Services or indirectly through the Cloud via the Windows Azure AppFabric Service Bus to connect to other enterprise packages.
6. The Multichannel Server Critical Capability to support any open protocol is handled automatically by Windows Azure. Cross-Platform wire protocols riding on top of HTTP are exposed by Windows Communication Foundation (WCF) and include SOAP, REST and Atompub. Cross-Platform data serialization is also provided by WCF including XML, JSON, and OData. These Multichannel capabilities support thick clients making web service calls as well as thin web clients making Ajax calls. Distributed caching to dramatically boost the performance of any client is provided by Windows Azure AppFabric Caching.
7. As you might imagine, the Hosting Critical Capability is handled by Windows Azure. Beyond providing the most complete solution of any Cloud provider, Windows Azure Connect provides an IPSec-protected connection with your On-Premises network and SQL Azure Data Sync can be used to move data between SQL Server and SQL Azure. This gives you the Hybrid Cloud solution you might be looking for.
8. For the Packaged Mobile Apps or Components Critical Capability, Android runs cross-platform mobile apps including Skype, Bing, MSN, Tag, Hotmail, and of course the critical ActiveSync component that makes push emails, contacts, calendars, and device management policies possible.

While Android 3.0 and higher meets many of Gartner’s Critical Capabilities, it doesn’t fare very well when it comes to cloud-based device management.  While other mobile device platforms also come up short in this department, I’m sure this will change in the coming year.  The tidal wave of CoIT means that device management in the future will look very different from how it did 5 years ago.  Expect a clear separation between corporate apps/data and personal apps/data to be managed.

Best Regards,

Rob

As you can see from the picture above:

1. For the Management Tools Critical Capability, Android uses Microsoft Exchange for On-Premise policy enforcement via Exchange ActiveSync (EAS) but has no private software distribution equivalent to System Center Configuration Manager 2007 from Microsoft today. Instead, in-house apps are hosted and APKs distributed via a web server over wireless by having a user click on a URL or through a variety of app stores. In the future, System Center Configuration Manager 2012 will be able to better manage Android devices.
2. For both the Client and Server Integrated Development Environment (IDE) and Multichannel Tool Critical Capability, Android uses Visual Studio. While the Server/EAI development functionality is the same as every other platform, endpoint development will consist of HTML5, ECMAScript 5, and CSS3 delivered by ASP.NET. WCF REST + JSON Web services can also be created and consumed via Ajax calls from the browser.
3. For the cross-platform Application Client Runtime Critical Capability, we will rely on Android’s WebKit browser to provide HTML5 + CSS3 + ECMAScript5 capabilities. Offline storage is important to keep potentially disconnected Android working and this is facilitated by Web Storage which is accessible via JavaScript.
4. For the Security Critical Capability, Android 3.0 and higher provides hardware encryption based on the user’s device passcode for data-at-rest. Data-in-transit is secured via SSL and VPN. LDAP API support allows it to access corporate directory services.
5. For the Enterprise Application Integration Tools Critical Capability, Android can reach out to servers directly via Web Services or indirectly via SQL Server (JDBC) or BizTalk using SSIS/Adapters to connect to other enterprise packages.
6. The Multichannel Server Critical Capability to support any open protocol directly, via Reverse Proxy, or VPN is facilitated by ISA/TMG/UAG/IIS. Cross-Platform wire protocols riding on top of HTTP are exposed by Windows Communication Foundation (WCF) and include SOAP, REST and Atompub. Cross-Platform data serialization is also provided by WCF including XML, JSON, and OData. These Multichannel capabilities support thick clients making web service calls as well as thin web clients making Ajax calls. Distributed caching to dramatically boost the performance of any client is provided by Windows Server AppFabric Caching.
7. While the Hosting Critical Capability may not be as relevant in an on-premises scenario, Windows Azure Connect provides an IPSec-protected connection to the Cloud and SQL Azure Data Sync can be used to move data between SQL Server and SQL Azure.
8. For the Packaged Mobile Apps or Components Critical Capability, Android runs cross-platform mobile apps including Skype, Bing, MSN, Tag, Hotmail, and of course the critical ActiveSync component that makes push emails, contacts, calendars, and device management policies possible.

Newer versions of Android (3.x/4.0) are beginning to meet more of Gartner’s Critical Capabilities. It’s really improved in the last year in areas of encryption, but device fragmentation makes this improvement uneven.  The app story is still the ‘Wild West’ since the Android Market is an un-vetted free-for-all. This big ‘red flag’ has given rise to curated app stores like the one from Amazon.  As you can see from the picture, the big gap is with the client application runtime critical capability. Native development via Java/Eclipse is where Google wants to steer you and Microsoft doesn’t make native tools, runtimes or languages for this platform. You can definitely perform your own due diligence on Mono for Android from our friend Miguel de Icaza and his colleagues in order to reuse your existing .NET and C# skills. From a Microsoft perspective though, you’re definitely looking at HTML5 delivered via ASP.NET.

Next week, I’ll cover how Android connects to the Cloud.

Best Regards,

Rob

As you can see from the picture above:

1. For the Management Tools Critical Capability, iOS uses Microsoft Exchange for On-Premise policy enforcement via Exchange ActiveSync (EAS) but has no private software distribution equivalent to System Center Configuration Manager 2007 from Microsoft today. Instead, in-house apps are hosted and distributed via a web server over wireless by having a user click on a URL.  In the future, System Center Configuration Manager 2012 will be able to better manage iOS devices.
2. For both the Client and Server Integrated Development Environment (IDE) and Multichannel Tool Critical Capability, iOS uses Visual Studio. While the Server/EAI development functionality is the same as every other platform, endpoint development will consist of HTML5, ECMAScript 5, and CSS3 delivered by ASP.NET.  WCF REST + JSON Web services can also be created and consumed via Ajax calls from the browser.
3. For the cross-platform Application Client Runtime Critical Capability, we will rely on iOS’s WebKit browser called Safari to provide HTML5 + CSS3 + ECMAScript5 capabilities. Offline storage is important to keep potentially disconnected iPhones and iPads working and this is facilitated by Web Storage which is accessible via JavaScript.
4. For the Security Critical Capability, iOS provides AES 256 hardware encryption as well as Data Protection based on the user’s device passcode for data-at-rest. Data-in-transit is secured via SSL, VPN, and 802.1X.  Built-in LDAP support allows it to access corporate directory services.
5. For the Enterprise Application Integration Tools Critical Capability, iOS can reach out to servers directly via Web Services or indirectly via SQL Server or BizTalk using SSIS/Adapters to connect to other enterprise packages.
6. The Multichannel Server Critical Capability to support any open protocol directly, via Reverse Proxy, or VPN is facilitated by ISA/TMG/UAG/IIS. Crosss-Platform wire protocols riding on top of HTTP are exposed by Windows Communication Foundation (WCF) and include SOAP, REST and Atompub. Cross-Platform data serialization is also provided by WCF including XML, JSON, and OData. These Multichannel capabilities support thick clients making web service calls as well as thin web clients making Ajax calls. Distributed caching to dramatically boost the performance of any client is provided by Windows Server AppFabric Caching.
7. While the Hosting Critical Capability may not be as relevant in an on-premises scenario, Windows Azure Connect provides an IPSec-protected connection to the Cloud and SQL Azure Data Sync can be used to move data between SQL Server and SQL Azure.
8. For the Packaged Mobile Apps or Components Critical Capability, iOS runs cross-platform mobile apps including OneNote, Bing, Tag, and of course the critical ActiveSync component that makes push emails, contacts, calendars, and device management policies possible.

As you can see, iOS meets many of Gartner’s Critical Capabilities.  It’s really improved over the years in areas of security and device management.  As you can see from the picture, the big gap is with the client application runtime critical capability.  Native development via Xcode/Objective-C is where Apple wants to steer you and Microsoft doesn’t make native tools, runtimes or languages for this platform.  You can certainly kick the tires and perform your own due diligence on MonoTouch from our friend Miguel de Icaza and his colleagues in order to reuse your existing .NET and C# skills.  From a Microsoft perspective though, you’re definitely looking at HTML5 delivered via ASP.NET.

Next week, I’ll cover how iOS connects to the Cloud.

Best Regards,

Rob

If you need a refresher on the Gartner MEAP Critical Capabilities, check out: http://robtiffany.com/meap/consumerization-of-it-collides-with-meap-windows-on-premises 

In this week’s scenario, I’ll use the picture below to illustrate how Mobile versions of Windows 7 in the form of slates, laptops, and tablets utilize some or all of Gartner’s Critical Capabilities to connect to Microsoft’s Cloud infrastructure:

As you can see from the picture above:

1. For the Management Tools Critical Capability, Windows 7 uses Windows Intune for Cloud-based device management and software distribution.
2. For both the Client and Server Integrated Development Environment (IDE) and Multichannel Tool Critical Capability, Windows 7 uses Visual Studio. The Windows Azure SDK plugs into Visual Studio and provides developers with everything they need to build Cloud applications.  It even includes a Cloud emulator to simulate all aspects of Windows Azure on their development computer.
3. For the cross-platform Application Client Runtime Critical Capability, Windows 7 uses .NET (Silverlight/WPF/WinForms) for thick clients. For thin clients, it uses Internet Explorer 9 to provide HTML5 + CSS3 + ECMAScript5 capabilities. Offline storage is important to keep potentially disconnected mobile clients working and this is facilitated by SQL Server Compact + Isolated Storage for thick clients and Web Storage for thin clients.
4. For the Security Critical Capability, Windows 7 provides security for data at rest via Bitlocker, data in transit via SSL, & Authorization/Authentication via the Windows Azure AppFabric Access Control Serivce (ACS).
5. For the Enterprise Application Integration Tools Critical Capability, Windows 7 can reach out to servers directly via Web Services or indirectly through the Cloud via the Windows Azure AppFabric Service Bus to connect to other enterprise packages.
6. The Multichannel Server Critical Capability to support any open protocol is handled automatically by Windows Azure. Crosss-Platform wire protocols riding on top of HTTP are exposed by Windows Communication Foundation (WCF) and include SOAP, REST and Atompub. Cross-Platform data serialization is also provided by WCF including XML, JSON, and OData. Cross-Platform data synchronization if provided by the Sync Framework. These Multichannel capabilities support thick clients making web service calls as well as thin web clients making Ajax calls. Distributed caching to dramatically boost the performance of any client is provided by Windows Azure AppFabric Caching.
7. As you might imagine, the Hosting Critical Capability is knocked out of the park with Windows Azure.  Beyond providing the most complete solution of any Cloud provider, Windows Azure Connect provides an IPSec-protected connection with your On-Premises network and SQL Azure Data Sync can be used to move data between SQL Server and SQL Azure.  This gives you the Hybrid Cloud solution you might be looking for.
8. For the Packaged Mobile Apps or Components Critical Capability, Windows 7 runs cross-platform mobile apps include Office/Lync/IE/Outlook/Bing.

As you can see from this and last week’s article, Windows 7 meets all of Gartner’s Critical Capabilities whether it’s connecting to Microsoft’s On-Premises or Cloud servers and infrastructure.  They great takeaway from the picture above, is Windows 7 only needs to know how to integrate its apps with WCF in the exact same way as is does in the On-Premises scenario.  Windows developers can focus on Windows without having to concern themselves with the various options provided by Windows Azure.  Cloud developers just need to provide a WCF interface to the mobile clients.

When an employee walks in the door with a wireless Windows 7 Slate device, you can rest assured that you can make them productive via Windows Azure without sacrificing any of the Gartner Critical Capabilities.

Next week, I’ll cover how Windows Phone connects to an On-Premises Microsoft infrastructure.

Best Regards,

Rob

Gartner specifies the following Critical Capabilities that must be addressed in order for a given product or stack of products to be considered a Mobile Enterprise Application Platform (MEAP):

• Integrated Development Environment
  

  A dedicated environment or plug-in for composing backend server and client side logic, including UI and UX

• Application Client Runtime
  

  The client runtime logic for the application, either in native format or packaged within a container.

• Enterprise Application Integration Tools
  

  Tools for integration of mobile server with back end systems, both bespoke and purchased apps or application suites.

• Packaged Mobile Apps or Components
  

  Self standing mobile applications or components.

• Multichannel Tools or Servers
  

  Tools that allow for “write once, run anywhere” thick or rich mobile clients, cross compilers or environments or platforms that allow business logic to be supported across thin, thick, and rich mobile architectures.

• Management Tools
  

  Tools for provisioning, supporting, debugging, updating or decommissioning mobile applications.

• Security 
  

  Tools for ensuring the security and privacy of enterprise data on board the device, while transiting through wired or wireless networks, through peripherals, and with backend systems and integration packages.

• Hosting
  

  The ability to host all development, provisioning, management functions, and optionally corporate data.

In this first scenario, I’ll use the picture below to illustrate how Mobile versions of Windows 7 in the form of slates, laptops, and tablets utilize some or all of Gartner’s Critical Capabilities to connect to an On-Premise Microsoft infrastructure:

As you can see from the picture above, Windows 7:

1. For the Management Tools Critical Capability, Windows 7 uses System Center Configuration Manager (SCCM) 2007 for on-premises device management and software distribution.
2. For both the Client and Server Integrated Development Environment (IDE) and Multichannel Tool Critical Capability, Windows 7 uses Visual Studio.
3. For the cross-platform Application Client Runtime Critical Capability, Windows 7 uses .NET (Silverlight/WPF/WinForms) for thick clients.  For thin clients, it uses Internet Explorer 9 to provide HTML5 + CSS3 + ECMAScript5 capabilities.  Offline storage is important to keep potentially disconnected mobile clients working and this is facilitated by SQL Server Compact + Isolated Storage for thick clients and Web Storage for thin clients.
4. For the Security Critical Capability, Windows 7 provides security for data at rest via Bitlocker, data in transit via SSL+VPN, data in the database via RSA/AES, & Authorization/Authentication via Active Directory.
5. For the Enterprise Application Integration Tools Critical Capability, Windows 7 can reach out to servers directly via Web Services or indirectly via SQL Server or BizTalk using SSIS/Adapters/Sync to connect to other enterprise packages.
6. The Multichannel Server Critical Capability to support any open protocol directly, via Reverse Proxy, or VPN is facilitated by ISA/TMG/UAG/IIS.  Crosss-Platform wire protocols riding on top of HTTP are exposed by Windows Communication Foundation (WCF) and include SOAP, REST and Atompub. Cross-Platform data serialization is also provided by WCF including XML, JSON, and OData. Cross-Platform data synchronization if provided by the Sync Framework.  These Multichannel capabilities support thick clients making web service calls as well as thin web clients making Ajax calls.  Distributed caching to dramatically boost the performance of any client is provided by Windows Server AppFabric Caching.
7. While the Hosting Critical Capability may not be as relevant in an on-premises scenario, Windows Azure Connect provide an IPSec-protected connection to the Cloud and SQL Azure Data Sync can be used to move data between SQL Server and SQL Azure.
8. For the Packaged Mobile Apps or Components Critical Capability, Windows 7 runs cross-platform mobile apps include Office/Lync/IE/Outlook/Bing.

It should come as no surprise that Windows 7 has a compelling and complete MEAP story to address the issues surrounding the Consumerization of IT (CoIT) when an employee walks in the door with a wireless Windows 7 Slate device.

Next week, I’ll cover how Windows 7 connects to the Cloud.

Best Regards,

Rob

There’s lots of folks out there that think they don’t have an actual connection to the end user because they develop on-premise and cloud server APIs.  You know who I’m talking about.  Server developers who are building Web Services using WCF to expose functionality to be consumed by clients.  Web developers that are creating dynamic ASP.NET web pages that make AJAX requests.  Well, I’m one of those people too. 

Just because you don’t build client UI’s that people can see and touch, doesn’t mean you don’t have an important impact on UX.

Once a user gets past the “skin-deep” beauty of the UI you typically think of, they’re often more concerned about accomplishing the task that the mobile app was designed for.  If a user taps on a button and has to wait over a minute to retrieve and display the data they’re requesting, it’s fair to say that they just had a poor user experience. 

In our new world where wirelessly-connected smartphones and tablets rule the day, a number of factors that you may not have thought of, come into play that impact user experience.  After the user clicks the button, and before data is retrieved, quite a few steps are taken through a very long path:

1. Your Request is transmitted from the device to the nearest cell tower while competing for bandwidth with hundreds or thousands of other nearby mobile devices all doing the same thing.  Oh, and those 3G and psuedo-4G speeds you read about are only achieved in perfect conditions with a limited number of wireless devices connected at any given time.
2. Your Request travels down the tower to the base station.
3. Your Request travels from base station to the mobile operator’s backhaul network where it competes with millions of other requests for bandwidth across fiber and other types of circuits.  Keep in mind that despite providing fast wireless speeds in perfect conditions, the amount of bandwidth in the various mobile operators backhaul networks varies widely.
4. Your Request jumps on to the public Internet and starts hopping through routers.
5. Your Request hits the outer firewall of the Domain you’re trying to reach.
6. Your Request might hit a reverse proxy.
7. Your Request might go through a back firewall as it leaves the DMZ to enter a corporate network.
8. Your Request finally gets to the intended server, and depending on the amount of traffic it’s experiencing, it will wait in an invisible queue created by the server operating system until it gets processed.
9. Your Response travels back through all those same hoops to get back to your wireless, mobile device.

Wow, I’m already tired just thinking about that!

So it should be clear that despite the great convenience provided by wireless data networks, there are a lot of hoops built into the system that work against you to diminish the user experience.  Doctors take an oath to “do no harm,” and I think server developers should take that oath to heart.  You have a lot of options to consider to make this wireless journey as fast as possible in order to put a smile on the user’s face.  Besides fast servers, lots of processor cores, fast SANs, caching, queuing, scaling out, data sharding, using the best SQL query plan, using faster code algorithms and such, I want you to optimize the stuff that’s actually going over the wireless network.  I want to make sure you “do no harm,” and that means not using fat, slow transports and wire protocols to move your data between devices and servers.

If you’ve been to my Tech Ed sessions or read past blog posts from me, you know that speed and efficiency mean a lot to me.  In the past, I’ve demonstrated 4 different ways to return a list of 8 delivery drivers from SQL Azure.  Depending on the choice the server developer made, the resulting user experience could be bad or good.

1. Worst:  Using OData with all it’s helpful metadata, the list of drivers used 8.54 kb of data.
2. Not as bad:  Using SOAP + DataSets, the list of drivers used 3 kb of data.
3. Much better:  Using REST + XML, the list shrank to 1.24 kb of data.
4. Best:  Using REST + JSON, the list dropped all the way down to 639 bytes.

Each one of the examples above returned the data my user was asking for, but they diverged in how much bandwidth they ate along the way.  Guess what happens if you take advantage of the built-in gzip + deflate compression capabilities found in IIS 7?  Through the use of the URL Rewrite Module, tweaks to your web.config file, and specifying an Accept-Encoding header like Nick Randolph figured out, you can shrink the 639 bytes to a fraction of that size.  Now we’re talking great user experience here!

Now you’re on the right track with efficient, compressed, REST + JSON Web Services, so let’s move on to the mobile web. 

Those giant, flashy websites you’ve been building since the 90′s aren’t going to cut it on mobile devices.  It doesn’t matter that your iPhone or Windows Phone 7 browser can render the Wall Street Journal in all its glory.  It’s not a good user experience.  Have you ever heard the phrase, “just because you can do something, doesn’t mean you should do that thing?”  Just because my mobile browser can flawlessly render the New York Times, doesn’t mean my user is interested in waiting till the next ice age of it to download and fully render.  Oh, and then you have to pinch and zoom to actually find anything that your eyes can read.  Do your users a favor and view my Tech Ed Europe session on the mobile web and download the mobile web best practices cards to build web sites for small screens and slow wireless networks with lots of latency.

Mobile web sites should be displayed in a single column, heavy on text, light on pictures and graphics, and weigh-in at under 20 kb in size.  Yes, I just said under 20 kb.  Just in case you’ve heard otherwise, the secret to a successful mobile web site is not HTML 5.  Remember, the mobile web is all about reaching as many users as possible with your site or web application.  I’ll have more to say about this in my forthcoming book on the subject.

So what’s the big takeaway here?

Cloud and on-premise server developers have a big role to play in UX even if you can’t always see what they’re doing!

Keep coding,

-Rob

We’ve come a long way and you’re probably wondering what qualifies this new embedded database to take on the Internet:

• Native support for x64 Windows Servers
• Virtual memory usage has been optimized to ensure the database can support up to 256 open connections – (Are you actually using 256 pooled connections with your “Big” database today?)
• Supports databases up to 4 GB in size – (Feel free to implement your own data sharding scheme)
• Developed, stress-tested, and tuned to support ASP.NET web applications
• Avoids the interprocess communications performance hit by running in-process with your web application
• Row-level locking to boost concurrency
• Step up to Government + Military grade security SHA2 algorithm to secure data with FIPS compliance
• Enhanced data reliability via true atomicity, consistency, isolation, and durability (ACID) support
• Transaction support to commit and roll back grouped changes
• Full referential integrity with cascading deletes and updates
• Support ADO.NET Entity Framework 4 – (Do I hear WCF Data Services?)
• Paging queries are supported via T-SQL syntax to only return the data you actually need

Wow, that’s quite a list!  SQL Server Compact 4.0 databases are easily developed using the new WebMatrix IDE or through Visual Studio 2010 SP1.  I’m loving the new ASP.NET Web Pages.  It reminds me of the good old days of building web applications with Classic ASP back in the 90′s with Visual InterDev and Homesite.

What about Mobility?

Since SQL Server Compact owes its heritage to mobile and embedded versions of Windows, you might be wanting to know what our story is there.  The good news is that you can build and deploy v4.0 databases on Windows XP, Windows Vista, and Windows 7.  If you want to implement an occasionally-connected solution that utilizes the Sync Framework, Remote Data Access (RDA), or Merge Replication, you’ll need to stick with SQL Server Compact 3.5 SP2.  Time and resource-constraints prevented the Compact team from enabling these features.  Luckily, single-user WPF/WinForms database applications running on Windows Slates, laptops and Windows Embedded Handheld devices will work just fine with the v3.5 SP2 runtime.  Get a jumpstart with this by pickup up “Enterprise Data Synchronization with Microsoft SQL Server 2008 and SQL Server Compact 3.5 Mobile Merge Replication” at   http://www.amazon.com/Enterprise-Synchronization-Microsoft-Compact-Replication/dp/0979891213/ref=sr_1_1?s=books&ie=UTF8&qid=1281715114&sr=1-1 to start building those MEAP solutions.

With the tidal wave of Windows Slates hitting the market, a secure, powerful mobile database that allows users to work offline and syncs with SQL Server is definitely going to be a hot item!

So run, don’t walk to the Microsoft Download site to download the Next-Gen database for the web:

http://www.microsoft.com/downloads/en/details.aspx?FamilyID=033cfb76-5382-44fb-bc7e-b3c8174832e2

If you need to support occasionally-connected mobile applications with sync capabilities on muliple Windows platforms, download SQL Server Compact 3.5 SP2:

http://www.microsoft.com/downloads/en/details.aspx?FamilyID=e497988a-c93a-404c-b161-3a0b323dce24

Keep Syncing,

Rob

Our favorite mobile database, SQL Server Compact 4.0 finds it’s way to the web with this tool providing a simple way to give your mobile web site a database.  It’s been beefed up and tuned for the stress of providing data services to Internet and supports 256 concurrent connections.  Since it’s a file-based database, you just copy it along with your web pages to your on-premise server, web hosting provider or Azure.

Last but not least, you get IIS Express which is a welcome replacement for the Cassini development web server currently used by Visual Studio.  This gives all developers the power of IIS 7.x without needing Administrator access to their box, even if they’re running on Windows XP.

The lightweight, inline-code nature of developing with WebMatrix makes it easy to build low-bandwidth sites that follow XHTML Basic 1.1 recommendations so you can target any mobile web browser.  From there, it’s up to you to determine if you want to support more advanced features found in mobile browsers like IE Mobile, Opera, or Webkit (iPhone, Android, webOS or Blackberry).

- Rob

Active Directory

• Since your clients will be passing in their Domain\username + password credentials when they sync, both IIS and SQL Server will make auth requests of the Domain Controller. Ensure that you have at least a primary and backup Domain Controller, that the NTDS.dit disk drives are big enough to handle the creation of a large number of new AD DS objects (mobile users and groups), and that your servers have enough RAM to cache all those objects in memory.

Database Schema

• Ensure your schema is sufficiently de-normalized so that you never have to perform more than a 4-way JOIN across tables. This affects server-side JOIN filters as well as SSCE performance.
• To ensure uniqueness across all nodes participating in the sync infrastructure, use GUIDs for your primary keys so that SQL Server doesn’t have to deal with the overhead of managing Identity ranges. Make sure to mark your GUIDs as ROWGUIDCOL for that table so that Merge won’t try to add an additional Uniqueidentifier column to the table.  Don’t create clustered indexes when using GUIDs as primary keys because they will suffer horrible fragmentation that will rapidly degrade performance.  Just use a normal index.
• Create clustered indexes for your primary keys when using Indentity columns, Datetime, or other natural keys.  Ensure that every column in every table that participates in a WHERE clause is indexed.

Distributor

• If your network connection is fast and reliable like Wi-Fi or Ethernet, your SSCE client has more than 32 MB of free RAM, and SQL Server isn’t experiencing any deadlocks due to contention with ETL operations or too many concurrent Merge Agents, create a new Merge Agent Profile based on the High Volume Server-to-Server Profile so that SQL Server will perform more work per round-trip and speed up your synchronizations.
• If you’re using a 2G/3G Wireless Wide Area Network connection, create a Merge Agent Profile based on the Default Profile so that SQL Server will perform less work and use fewer threads per round-trip during synchronization than the High Volume Server to Server Profile which will help to reduce server locking contention and perform less work per round trip which will make your synchronizations more likely to succeed.
• In order to prevent SQL Server from performing Metadata Cleanup every time a Subscriber synchronizes, set the –MetadataRetentionCleanup parameter to 0.
• As SQL Server has to scale up to handle a higher number of concurrent users in the future, locking contention will increase due to more Merge Agents trying to perform work at the same time.  When this happens, adjust the parameters of the Default Profile so that both  –SrcThreads and –DestThreads are equal to 1.

Publication

• When defining the Articles you’re going to sync, only check the minimum tables and columns needed by the Subscriber to successfully perform its work.
• For Lookup/Reference tables that aren’t modified by the Subscriber, mark those as Download-only to prevent change-tracking metadata from being sent to the Subscriber.
• Despite the fact the column-level tracking sends less data over the air, stick with row-level tracking so SQL Server won’t have to do as much work to track the changes.
• Use the default conflict resolver where the “Server wins” unless you absolutely need a different manner of picking a winner during a conflict.
• Use Static Filters to reduce the amount of server data going out to all Subscribers.
• Make limited use of Parameterized Filters which are designed to reduce and further specify the subset of data going out to a particular Subscriber based on a HOST_NAME() which creates data partitions.  This powerful feature slows performance and reduces scalability with each additional filter, so it must be used sparingly.
• Keep filter queries simple and don’t use IN clauses, sub-selects or any kind of circular logic.
• Strive to always create “well-partitioned” Articles where all changes that are uploaded/downloaded are mapped to only the single partition ID for best performance and scalability.
  • When using Parameterized Filters, always create non-overlapping data partitions where each row from a filtered table only goes to a single Subscriber instead of more than one which will avoid the use of certain Merge metadata tables.
  • Each Article in this scenario can only be pubished to a single Publication
  • A Subscriber cannot insert rows that do not belong to its partition ID.
  • A Subscriber cannot update columns that are involved in filtering.
  • In a join filter hierarchy, a regular article cannot be the parent of a “well-partitioned” article.
  • The join filter in which a well-partitioned article is the child must have the join_unique_key set to a value of 1 which relates to the Unique key check box of the Add Join dialog.  This means there’s a one-to-one or one-to-many relationship with the foreign key.
  • Each “well-partitioned” Article can have only one subset or join filter. The article can have a subset filter and be the parent of a join filter, but cannot have a subset filter and be the child of a join filter.
• Never extend a filter out to more than 4 joined tables.
• Do not filter tables that are primarily lookup/reference tables, small tables, and tables with data that does not change.
• Schedule the Snapshot Agent to run once per day to create an unfiltered schema Snapshot.
• Set your Subscriptions to expire as soon as possible to keep the amount change-tracking metadata SQL Server has to manage to an absolute minimum. Normally, set the value to 2 to accommodate 3-day weekends since 24 hours are automatically added to the time to account for multiple time zones. If server-side change tracking isn’t needed and Subscribers are pulling down a new database every day and aren’t uploading data, then set the expiration value to 1.
• Set Allow parameterized filters equal to True.
• Set Validate Subscribers equal to HOST_NAME().
• Set Precompute partitions equal to True to allow SQL Server to optimize synchronization by computing in advance which data rows belong in which partitions.
• Set Optimize synchronization equal to False if Precompute partitions is equal to True.  Otherwise set it to True to optimize filtered Subscriptions by storing more metadata at the Publisher.
• Set Limit concurrent processes equal to True.
• Set Maximum concurrent processes equal to the number of SQL Server processor cores.  If exceesive locking contention occurs, reduce the number of concurrent processes until the problem is fixed.
• Set Replicate schema changes equal to True.
• Check Automatically define a partition and generate a snapshot if needed when a new Subscriber tries to synchronize. This will reduce Initialization times since SQL Server creates and applies snapshots using the fast BCP utility instead of a series of slower SELECT and INSERT statements.
• Add data partitions based on unique HOST_NAMEs and schedule the Snapshot Agent to create those filtered Snapshots nightly or on the weekend so they’ll be built using the fast BCP utility and waiting for new Subscribers to download in the morning.
• Ensure that SQL Server has 1 processor core and 2 GB of RAM for every 100 concurrent Subscribers utilizing bi-directional sync. Add 1 core and 2 GB of RAM server for every additional 100 concurrent Subscribers you want to add to the system.  Never add more Subscribers and/or IIS servers without also adding new cores and RAM to the Publisher.
• Turn off Hyperthreading in the BIOS of the SQL Server as it has been known to degrade SQL Server performance.
• Do not add your own user-defined triggers to tables on a Published database since Merge places 3 triggers on each table already.
• Add one or more Filegroups to your database to contain multiple, secondary database files spread out across many physical disks.
• Limit use of large object types such as text, ntext, image, varchar(max), nvarchar(max) or varbinary(max) as they require a significant memory allocation and will negatively impact performance.
• Set SQL Servers’s minimum and maximum memory usage to within 2 GB of total system memory so it doesn’t have to allocate more memory on-demand.
• Always use SQL Server 2008 R2 and Windows Server 2008 R2 since they work better together because they take advantage of the next generation networking stack which dramatically increases network throughput. They can also scale up as high as 256 cores.
• Due to how Merge Replication tracks changes with triggers, Merge Agents, and tracking tables, it will create locking contention withDML/ ETL operations.  This contention degrades server performance which negatively impacts sync times with devices.  This contention should be mitgated by performing large INSERT/UPDATE/DELETE DML/ETL operations during a nightly maintenance window when Subscribers aren’t synchronizing.
• Since Published databases result in slower DML/ETL operations, perform changes in bulk by using XML Stored Procedures to boost performance.
• To improve the performance of pre-computed partitions when DML/ETL operations result in lots of data changes, ensure that changes to a Parent table in a join filter are made before corresponding changes in the child tables.  This means that when DML/ETL operations are pushing new data into SQL Server, they must add master data to the parent filter table first, and then add detail data to all the related child tables second, in order for that data to be pre-computed and optimized for sync.
• Create filter partitions based on things that don’t change every day.  Partitions that are added and deleted from SQL Server and Subscribers that move from one partition to another is very disruptive to the performance of Merge Replication.
• Always perform initializations and re-initializations over Wi-Fi or Ethernet when the device is docked because this is the slowest operation where the entire database must be downloaded instead of just deltas.  To determine rough estimates for initialization, multiply the size of the resulting SSCE .sdf file x the bandwidth speed available to the device.  A file copy over the expected network will also yield estimates for mininum sync times.  These times don’t include the work SQL Server and IIS must perform to provide the data or data INSERT times on SSCE.
• If your SQL Server Publisher hits a saturation point with too many concurrent mobile Subscribers, you can scale it out creating a Server/Push Republishing hierarchy. Put the primary SQL Server Publisher at the top of the pyramid and have two or more SQL Servers subscribe to it. These can be unfiltered Subscriptions where all SQL Servers get the same data or the Subscribers can filter their data feeds by region for example. Then have the Subscribing SQL Servers Publish their Subscription for consumption by mobile SSCE clients.
• Create just a single Publication.

Internet Information Services

• Use the x64 version of the SQL Server Compact 3.5 SP2 Server Tools with Windows Server 2008 R2 running inside IIS 7.5.
• Use a single Server Agent in a single Virtual Directory.
• Ensure the IIS Virtual Directory where the Server Agent resides is on a fast solid-state drive that’s separate from the disk where Windows Server is installed to better support file I/O.
• Use a low-end server with 2 processor cores and 2 GB of RAM to support 400 concurrent Subscribers queued at the same time.
• Set the MAX_THREADS_PER_POOL Server Agent registry key equal to 2 to match the IIS processor cores and RAM. Do not set this value to a higher number than the number of cores.
• Set the MAX_PENDING_REQUEST Server Agent registry key equal to 400 which means the Server Agent will queue up to 400 concurrent Subscribers waiting for one of the 2 worker threads to become available to sync with SQL Server.
• Set the IIS Connection Limits property to 400 to prevent an unlimited number of connections reaching the Server Agent.
• Add a new load-balanced IIS server for every additional 400 concurrent Subscribers you want to add to the system.

Subscriber

• Use the appropriate x64, x86 or ARM version of SQL Server Compact 3.5 SP2 to take advantage of the PostSyncCleanup property of the SqlCeReplication object that can reduce the time it takes to perform an initial synchronization. Set the PostSyncCleanup property equal to 3 where neither UpdateStats nor CleanByRetention are performed.
• Increase the Max Buffer Size connection string parameter to 1024 on a phone and 4096 on a PC to boost both replication and SQL query processing performance. If you have more RAM available, set those values even higher until you reach the law of diminishing returns.
• Keep your SSCE database compact and fast by setting the Autoshrink Threshold connection string parameter to 10 so it starts reclaiming empty data pages once the database has become 10% fragmented.
• Replication performance testing must be performed using actual PDAs to observe how available RAM, storage space and CPU speed affect moving data into the device’s memory area and how quickly this data is inserted into the SSCE database tables.  Since the SSCE database doubles in size during replication, the device must have enough storage available or the operation will fail.  Having plenty of available RAM is important so that SSCE can utilize its memory buffer to complete a Merge Replication operation more quickly.  With plenty of available RAM and storage, a fast CPU will make all operations faster.
• The PDA must have at least an extra 32 MB of available free RAM that can be used by the .NET Compact Framework (NETCF) application.  If additional applications are running on the device at the same time, even more RAM is needed.  If a NETCF application has insufficient RAM is will discard its compiled code and run in interpreted mode which will slow the application down.  If the NETCF app is still under memory pressure after discarding compiled code, Windows Mobile will first tell the application to return free memory to the operating system and then will terminate the app if needed.
• Set the CompressionLevel property of the SqlCeReplication object to 0 for fast connections and increment it from 1 to 6 on slower connections like GPRS to increase speed and reduce bandwidth consumption.
• Tune the ConnectionRetryTimeout, ConnectTimeout, ReceiveTimeout and SendTimeout properties of the SqlCeReplication object based on expected bandwidth speeds:

• You can decrease potentially slow SSCE file I/O by adjusting the Flush Interval connection string parameter to write committed transactions to disk less often than the default of every 10 seconds.  Test longer intervals between flushes like 20 or 30 seconds. Keep in mind that these transactions can be lost if the disk or system fails before flushing occurs so be careful.
• When replicating data that has been captured in the field by the device, perform Upload-only syncs to shorten the duration.

Storage

• Use a Fibre Channel SAN with 15k RPM or solid-state disks for best I/O performance.
• Databases should reside on a RAID 10, unshared LUN comprised of at least 6 disks.
• Database logs should reside on a RAID 10, unshared LUN comprised of at least 6 disks.
• Tempdb should reside on a RAID 10, unshared LUN comprised of at least 6 disks.
• The Tempdb log should reside on a RAID 10, unshared LUN comprised of at least 6 disks.
• The Snapshot share should reside on a RAID 10, unshared LUN comprised of at least 6 disks.  This disk array should be large enough to accommodate a growing number of filtered Snapshots. Snapshot folders for Subscribers that no longer use the system must be manually deleted.
• Merge Replication metadata tables should reside on a RAID 10, unshared LUN comprised of at least 6 disks.
• Increase your Host Bus Adapter (HBA) queue depths to 64.
• Your Publication database should be broken up into half the number of files as the SQL Server has processor cores. Each file must be the same size.
• Tempdb should be pre-sized with an auto-growth increment of 10%. It should be broken up into the same number of files as the SQL Server has processor cores. Each file must be the same size.

High Availability

• Load-balance the IIS servers to scale them out. Enable Server Affinity (stickiness) since the Replication Session Control Blocks that transmit data between the Server Agent and SSCE are stateful. Test to ensure that your load-balancer is actually sending equal amounts of Subscriber sync traffic to each IIS server.  Some load-balancers can erroneously send all traffic to a single IIS server if not properly configured.
• Implement Windows Clustering so that SQL Server can failover to a second node.
• Using SQL Server Mirroring so that your Published database will failover to a standby server.
• Make a second SQL Server into an unfiltered Subscriber to your Publisher so that it can take over Merge Replication duties for mobile clients as a Republisher if the primary SQL Server fails. SSCE clients would just have to reinitialize their Subscriptions to begin synchronizing with the new Republisher.

Ongoing Maintenance

• Use the Replication Monitor to have a real-time view of the synchronization performance of all your Subscribers.
• Use the web-based SQL Server Compact Server Agent Statistics and Diagnostics tools to monitor the health and activity of the Server Agent running on IIS.
• Create a SQL Job to execute the sp_MSmakegeneration stored procedure after large DML operations. Regular execution after INSERTING/UPDATING/DELETING data from either DML/ETL operations or after receiving lots of changes from Subscribers will maintain subsequent sync performance. Executing this stored procedure from the server-side is preferable to having it executed as a result of a Subscriber sync which would block all other Subscribers.
• During your nightly maintenance window, rebuild the indexes and update the statistics of the following Merge Replication metadata tables:
  • MSmerge_contents
  • MSmerge_tombstone
  • MSmerge_genhistory
  • MSmerge_current_partition_mappings
  • MSmerge_past_partition_mappings
  • MSmerge_generation_partition_mappings
• If you notice performance degradation during the day due to a large number of Subscribers making large changes to the database, you can updates the statistics (with fullscan) of the Merge Replication metadata tables more frequently throughout the day to force stored proc recompiles to get a better query plan.
  • UPDATE STATISTICS MSmerge_generation_partition_mappings WITH FULLSCAN
  • UPDATE STATISTICS MSmerge_genhistory WITH FULLSCAN
• Rebuild/defrag indexes on your database tables and Merge Replication metadata tables throughout the day to reduce locking contention and maintain performance.
• Use the Missing Indexes feature of SQL Server to tell you which indexes you could add that would give your system a performance boost. Do not add recommended indexes to Merge system tables.
• Use the Database Engine Tuning Advisor to give you comprehensive performance tuning recommendations that cover every aspect of SQL Server.
• Monitor the performance of the following counters:
  • Processor Object: % Processor Time: This counter represents the percentage of processor utilization. A value over 80% is a CPU bottleneck.
  • System Object: Processor Queue Length: This counter represents the number of threads that are delayed in the processor Ready Queue and waiting to be scheduled for execution. A value over 2 is bottleneck and shows that there is more work available than the processor can handle. Remember to divide the value by the number of processor cores on your server.
  • Memory Object: Available Mbytes: This counter represents the amount of physical memory available for allocation to a process or for system use. Values below 10% of total system RAM indicate that you need to add additional RAM to your server.
  • PhysicalDisk Object: % Disk Time: This counter represents the percentage of time that the selected disk is busy responding to read or write requests. A value greater than 50% is an I/O bottleneck.
  • PhysicalDisk Object: Average Disk Queue Length: This counter represents the average number of read/write requests that are queued on a given physical disk. If your disk queue length is greater than 2, you’ve got an I/O bottleneck with too many read/write operations waiting to be performed.
  • PhysicalDisk Object: Average Disk Seconds/Read and Disk Seconds/Write: These counters represent the average time in seconds of a read or write of data to and from a disk. A value of less than 10 ms is what you’re shooting for in terms of best performance. You can get by with subpar values between 10 – 20 ms but anything above that is considered slow. Times above 50 ms represent a very serious I/O bottleneck.
  • PhysicalDisk Object: Average Disk Reads/Second and Disk Writes/Second: These counters represent the rate of read and write operations against a given disk. You need to ensure that these values stay below 85% of a disk’s capacity by adding disks or reducing the load from SQL Server. Disk access times will increase exponentially when you get beyond 85% capacity.
• A limited number of database schema changes can be made and synchronized down to SSCE Subscribers without any code changes which makes it easier to update your system as it evolves over time.
• Use a Merge Replication Test Harness to stress test the entire system.  The ability to simulate hundreds or thousands of concurrent synchronizing Subscribers allows you to monitor performance and the load on the system.  This is helpful in properly configuring and tuning SQL Server, IIS, and the Windows Mobile devices.  It will tell you where you’re having problems and it will let you predict how much server hardware you will need to support growing numbers of Subscribers over time.  It’s also very important to simulate worst-case scenarios that you never expect to happen.

I hope this information sufficiently empowers you to take on the largest MEAP solutions that involve SQL Server Merge Replication and SQL Server Compact.  If you need a deeper dive, go check out my book on Enterprise Data Synchronization http://www.amazon.com/Enterprise-Synchronization-Microsoft-Compact-Replication/dp/0979891213/ref=sr_1_1?ie=UTF8&s=books&qid=1271964573&sr=1-1 over at Amazon.  Now go build a fast and scalable solution for your company or your customers.

Best Regards,

Rob

P.S.  If your solution doesn’t require all the advanced features found in Merge Replication, I highly recommend you use Remote Data Access (RDA).  This is a much simpler sync technology that’s extremely fast, scalable, and easier to manage.

With the tidal wave of mobile and wireless technologies sweeping across both the consumer and enterprise landscapes, I believe MEAP offerings give us a glimpse of a new standard for designing all future infrastructures.

-Rob

1. When there are 3 or more mobile applications
2. When there are 3 or more targeted operating systems or platforms
3. When they involve the integration of 3 or more back-end systems

Leaders in this space have included Sybase iAnywhere, Antenna, Dexterra, Syclo and Spring Wireless.  Microsoft goes from a large Mobile General Store with myriad solutions to a player in this space with a MEAP solution of our own:  Visual Studio is used to build the mobile logic and UI.  Merge Replication provides occasionally-connected data synchronization between SQL Server Compact on the mobile device and SQL Server in the data center.  SQL Server Business Intelligence Development Studio is used to visually create connections to back-end systems like SAP or databases like Oracle.  Data in transit is secured via SSL or VPN, data at rest is encrypted via device encryption, SQL Server Compact, BitLocker or programmatically through the Crypto API.  Integration packages that communicate with back-end systems are encrypted and digitally signed. 

We already have the best mobile email, calendaring, and contacts product in the business where Exchange Active Sync keeps Outlook and Outlook Mobile always up to date with Exchange Server.  Server-to-device as well as peer-to-peer device notifications are facilitated through WCF Store and Forward on Exchange.  Software and patch distribution along with device settings and policy management is accompished via System Center Configuration Manager.  ISA Server provides both VPN and Reverse Proxy access to roaming applications on the Internet on any platform.

When you put this stack in place and resuse it for multiple mobile applications instead of going with point solutions, ROI savings increase as the need for POCs, Pilots and training are reduced and the need for extra client access licenses is eliminated.  That’s Gartner’s first requirement.  We hit Gartner’s second requirement by uniformly supporting 3 mobile operating systems in the form of Windows, Windows CE, and Windows Mobile.  Last but not least, our SQL Server Integration Services technology combined with dozens of connectors mean we can connect your mobile devices with almost any back-end package or database.

Yes, Microsoft does have a Mobile Enterprise Application Platform that’s already proven to scale to tens of thousands of devices and it will definitely save you time and money.

- Rob

With the world’s largest organizations rolling out tens of thousands of Windows® phones, laptops, tablets and Netbooks to empower their respective mobile workforces, the ability to create mobile line of business solutions that support large numbers of users is absolutely critical. In my fourth book on mobile infrastructure and development, I show you how to take the SQL Server data you use to run your organization and make it available to all of your mobile employees.

Step-by-step, I’ll walk you through the process of building a secure, performant, n-tier, mobile enterprise application platform architecture designed to scale to thousands of users. You’ll also learn how to create occasionally-connected .NET applications designed to thrive in unreliable wireless conditions.

• Learn how to “Mobilize” your organization by making your enterprise data available to employees carrying Windows® phones, laptops, Netbooks and tablets in the field.
• Learn how to build an N-Tier Mobile Sync infrastructure that will scale to thousands of users.
• Learn how to create occasionally-connected .NET applications designed to thrive in unreliable wireless conditions.
• Learn best practices in security, reliability, performance, load-balancing, reverse proxy and hardware configuration.
• Learn how to implement this technology in real world scenarios like supply chain management, retail, sales force automation, healthcare and emergency management.

Keep in mind that the knowledge you gain from this book didn’t come from me dreaming this stuff up in an Ivory Tower.  It came from building some of the worlds largest and most complex data synchronization systems for the world’s largest companies.  In addition to the hands-on experience that went into this book, I’d also like to thank some of my colleagues for their invaluable contributions:

• Liam Cavanagh is a Senior Lead Program Manager for Microsoft’s Sync Framework and Cloud Data Services and he wrote the forward.
• Catherine Wyatt is the Managing Editor for Hood Canal Press who made the publishing of the book possible.
• Darren Shaffer is the CEO of Handheld Logic and he wrote the Chapter on building the Mobile Subscriber. 
• Michael Jimenez is a Mobility Architect at Microsoft and he wrote the Appendix that shows you how to create an ISA Server 2006 Reverse Proxy to publish your sync infrastructure to the Internet.

It’s my sincere hope that this book will encourage you to un-tether your workforce from their desktop computers and boost your organization’s agility by pushing out critical business functions to the point of activity where employees are empowered to make timely decisions and perform tasks that best serve the interests of their customers and their company.

This repudiation of the traditional “connected” software application model increases customer satisfaction, boosts worker efficiency, reduces “missed opportunities” and results in cost savings as “un-wired” employees get their jobs done wherever they happen to be.

Best Regards,

Rob

Besides synchronizing the tables, rows and columns of a complete database between mobile clients and servers, the use of message queuing should be considered for many scenarios due to its high-reliability by ensuring that a critical message arrives at its destination.  Products like MSMQ, MQ Series, Tibco and JMS are used all over the world in the most mission-critical environments to ensure a high level of availability and reliability.  They’re asynchronous by nature and use store and forward mechanisms so that messages get from point A to B to C.  A typical queue message includes the Destination which tells the messge where to go, a Label which describes the message, a Body which contains the message, and a Body Length so the receiver can verify that it received everything.

So how does any of this relate to Windows phones?  A number of years ago, an MSMQ client was made available for download and installation on Windows phones.  Additionally, the .NET Compact Framework 2.0 included classes to work with MSMQ.  Unfortunately, the installation of MSMQ was far from seamless which inhibited its adoption by customers.  More recently, functionality was included in the .NET Compact Framework 3.5 that facilitated store and forward messaging using Exchange 2007 as a transport.  This is a good solution for customers running the newest version of Exchange, have an unlimited data plan for their phones, and don’t mind running line of business applications over their email infrastructure.

So what do we do for customers that have found the Windows Mobile MSMQ client too much of a hassle, don’t have Exchange 2007 or don’t want to use it as a mobile message queue server?  I think the answer has been under our noses all along.  Burned in the ROM of every Windows Mobile 6.x device is SQL Server Compact + a lightweight data sync solution called Remote Data Access (RDA).  For those of you running Windows Mobile 5, XP, Vista or 7, you can easily download these bits to your mobile client.  SQL Server Compact is you local queue, RDA is your transport and SQL Server in the cloud or data center is your message queue server.  So let’s break this down and see how it will work.

A mobile application that captures data in the field would want to drop that info in a local queue.  SQL Server Compact becomes that local queue and the message format is actually a table with the following structure:

This table would be created inside a MessgeQueue database in SQL Server and RDA would pull it down to SQL Server Compact.  In the Pull method you call from .NET on the client, you would add “WHERE 1=0” to the SQL statement.  This filter has the effect of pulling down an empty shell of the table without retrieving any data to the client since that’s all you want.  It also means that when you insert local data into the table and call the Push method, the data will be removed from the client at the completion of a successful sync.

So you’re probably wondering, what makes this so special and message queue-like vs. anything else?  The secret is that unlike other sync technologies, RDA can wrap the upload of data into a transaction.  As the data is being uploaded over wireless, if any of the INSERTs into SQL Server fail for whatever reason, everything gets rolled-back and the original data remains in the local SQL Server Compact queue.  This is the kind of guaranteed commit that you expect from a message queuing system.  It’s an “all or nothing” success or rollback. 

It actually gets better.  A property of both RDA and Merge Replication is called ConnectionRetryTimeout.  This feature is designed to help you with unreliable wireless coverage where you have signal one minute and then lose it the next.  Let’s say you have this timeout value set to 2 minutes and you begin your Push upload of queued data.  Everything is working fine for the first few seconds but then you lose wireless coverage from your mobile operator.  If you regain coverage before the 2 minute time-out, the upload will resume where it left off.  Since both RDA and Merge send and receive data between the SQL Server Compact and IIS in tiny blocks, you never have to worry about running out of memory and you can pick up where you left off in case of a network dropout.

So the big takeaway here is that we do in fact have a Mobile Message Queue solution hidden on our Windows phones.  We have a message format that lets us drop text/xml/whatever data into the body, a label that a server process or SQL trigger can key off of to perform an action, and a transactional upload mechanism that ensures your critical data will cross the wireless chasm and make it to the other side intact.

So what’s next?  Now that you can capture data in a local queue and safely upload it, you might be wondering how queued messages from someone else can be pushed to your device.  Don’t worry, that will be in my next post.  Also, this isn’t just an article on how to solve a big problem in the mobile space, I’m actually building the necessary client and server pieces as well.  We’re all looking for a reliable and unified way to connect mobile devices to corporate assets and this just might be the simple answer we’re looking for.

- Rob

Before I dove head first into the world of Merge Replication, I always used RDA to get my customers up and running quickly.  Mobilizing an organization’s workforce quickly and easily is what it’s all about so they can start reaping the benefits.  In addition to a speedy time to market, there’s no faster or more scalable mobile sync technology on the market anywhere. 

So why wouldn’t I always use RDA?  Here’s a quick list:

1. You’re using Identity columns.
2. You want to replicate schema changes to the client.
3. You want change tracking on both the client and server to perform diffs of each of the tables during a sync instead of re-downloading the entire table.
4. You want to automatically resolve conflicts that arise when 2 people update the same data.
5. You want referential integrity constraints to be pushed down to the client database from SQL Server.
6. You don’t want to write code to perform synchronization or filter data.

If anything on the above list applied to you, you would shift to Merge Replication because it could manage ranges of Identity columns, push down schema changes, only sync data differences, resolve conflicts and push down a database’s referential integrity constraints.  Merge requires almost no code to get started and tables and columns are filtered visually via a wizard.

So why might you choose to use RDA?  Here’s another list:

1. Your Primary Keys use GUIDs instead of Identity columns.
2. Users don’t overwrite each other’s data so you don’t need conflict resolution.  The rule of “Last in Wins” works for you.
3. While you want indexes to be pushed down, you don’t care if your local SSCE database has referential integrity constraints applied.
4. You want to wrap the changes you upload to SQL Server in a transaction so that all changes are applied or none of them are.
5. Change tracking on the client is good enough and re-downloading updated server tables doesn’t take too long.
6. You developers don’t mind writing some sync code.
7. Be able to execute SQL and Stored Procedures directly against SQL Server via IIS.
8. You’re downloading read-only data.

If your solution meets the criteria in the list above, you’re probably a good candidate for using RDA instead of Merge.  Are there any other choices out there?

Back at MEDC 2007, we announced a new data replication technology for devices called Occasionally Connected Sync that would sit somewhere between RDA and Merge.  OCS as it used to be called was renamed Sync Services for ADO.NET and then was eventually merged into the Sync Framework. 

The Sync Framework is a developer-focused technology:

1. Supports conflict resolution.
2. Change tracking on the server as well as the client so that only data differences are exchanged.
3. Peer to Peer sync in the forthcoming v2 of Sync Framework.
4. Sync with databases other than SQL Server.
5. Best suited for SSCE running on a desktop or laptop.

The clearest differentiation that the Sync Framework has over Merge is its provider model which allows it to sync with other ADO.NET databases like Oracle or DB2.  SQL Server supports built-in P2P Transactional replication and v2 of the Sync Framework will allow you to do this via WCF.  If you development team doesn’t mind writing lots of sync code and needs to support scenarios like synchronizing with other databases from SSCE on the desktop, then the Sync Framework might be the way to go for you.  I wouldn’t yet recommend the Sync Framework for device sync since its wire protocol is currently based on the DataSet which may cause out of memory errors on Windows phones with limited working sets.

So where does this leave RDA?

The reason I’m writing this blog post is because time and time again I run into customer sync scenarios that don’t always need the power of Merge or the extra flexibility of the Sync Framework.  Most field service applications follow the same kind of pattern:

1. Lots of download-only lookup/reference tables that aren’t changed by the user.
2. Tables that are pushed down to the device that tell a user where to go and what to do.
3. Tables (sometimes empty) that are used to capture data from the user in the field that are upload-only.

These kinds of schemas don’t require conflict resolvers or server change tracking and are therefore well suited for RDA. 

What’s the big benefit of using RDA if a sync scenario meets its criteria?

1. You won’t modify SQL Server’s schema with GUIDs and Triggers.
2. You won’t degrade the performance of SQL Server by having it track changes and maintain extra metadata.
3. You will have the fastest and most scalable sync solution with least amount of hardware.
4. Time to market is shorter.

The big takeaway here is that I want you to consider your sync solution carefully before choosing a technology.  If your customer’s needs are met by RDA, then you should use it and reap the benefits of developing and deploying a simpler solution with fewer moving parts.

Remember Occam’s Razor.

-Rob