A separate 1099 for Hood Canal Press tells me that my SQL Server Merge Replication books are still killing it!  Our world of disconnected devices require efficient data sync now more than ever.  Despite a variety of sync technologies out there, Merge is still the best!

As you can see from the picture above:

1. For the Management Tools Critical Capability, there is no Cloud-based device management solution, policy-enforcement, or software distribution solution from Microsoft for Android. As I mentioned in last week’s post, consumer software distribution comes from the Android Market and the enterprise equivalent is facilitated via internal web servers and user-clickable URLs. Since Android is a wide-open system, competing markets and app stores are on the rise from Amazon and others.
2. For both the Client and Server Integrated Development Environment (IDE) and Multichannel Tool Critical Capability, Android uses Visual Studio. Endpoint development consists of HTML5, ECMAScript 5, and CSS3 delivered by ASP.NET via Web Roles. WCF REST + JSON Web services can also be created and consumed via Ajax calls from the browser. On the Cloud side of things, the Windows Azure SDK plugs into Visual Studio and provides Android developers with everything they need to build Cloud applications. It includes a Cloud emulator to simulate all aspects of Windows Azure and AppFabric on their development computer. In scenarios where native development is required by the customers, the Windows Azure Toolkit for Android can be used to allow Java via Eclipse to securely communicate with the Microsoft cloud.
3. For the cross-platform Application Client Runtime Critical Capability, Android uses the WebKit browser called Chrome to provide HTML5 + CSS3 + ECMAScript5 capabilities. Offline storage is important to keep potentially disconnected Android smartphones and tablets working and this is facilitated by Web Storage which is accessible via JavaScript.
4. For the Security Critical Capability, Android 3.0 and higher provides hardware encryption based on the user’s device passcode for data-at-rest. Data-in-transit is secured via SSL and VPN. LDAP API support allows it to access corporate directory services. Auth in the Microsoft cloud is handled via the Windows Azure AppFabric Access Control Service (ACS).
5. For the Enterprise Application Integration Tools Critical Capability, Android can reach out to servers directly via Web Services or indirectly through the Cloud via the Windows Azure AppFabric Service Bus to connect to other enterprise packages.
6. The Multichannel Server Critical Capability to support any open protocol is handled automatically by Windows Azure. Cross-Platform wire protocols riding on top of HTTP are exposed by Windows Communication Foundation (WCF) and include SOAP, REST and Atompub. Cross-Platform data serialization is also provided by WCF including XML, JSON, and OData. These Multichannel capabilities support thick clients making web service calls as well as thin web clients making Ajax calls. Distributed caching to dramatically boost the performance of any client is provided by Windows Azure AppFabric Caching.
7. As you might imagine, the Hosting Critical Capability is handled by Windows Azure. Beyond providing the most complete solution of any Cloud provider, Windows Azure Connect provides an IPSec-protected connection with your On-Premises network and SQL Azure Data Sync can be used to move data between SQL Server and SQL Azure. This gives you the Hybrid Cloud solution you might be looking for.
8. For the Packaged Mobile Apps or Components Critical Capability, Android runs cross-platform mobile apps including Skype, Bing, MSN, Tag, Hotmail, and of course the critical ActiveSync component that makes push emails, contacts, calendars, and device management policies possible.

While Android 3.0 and higher meets many of Gartner’s Critical Capabilities, it doesn’t fare very well when it comes to cloud-based device management.  While other mobile device platforms also come up short in this department, I’m sure this will change in the coming year.  The tidal wave of CoIT means that device management in the future will look very different from how it did 5 years ago.  Expect a clear separation between corporate apps/data and personal apps/data to be managed.

Best Regards,

Rob

As you can see from the picture above:

1. For the Management Tools Critical Capability, Android uses Microsoft Exchange for On-Premise policy enforcement via Exchange ActiveSync (EAS) but has no private software distribution equivalent to System Center Configuration Manager 2007 from Microsoft today. Instead, in-house apps are hosted and APKs distributed via a web server over wireless by having a user click on a URL or through a variety of app stores. In the future, System Center Configuration Manager 2012 will be able to better manage Android devices.
2. For both the Client and Server Integrated Development Environment (IDE) and Multichannel Tool Critical Capability, Android uses Visual Studio. While the Server/EAI development functionality is the same as every other platform, endpoint development will consist of HTML5, ECMAScript 5, and CSS3 delivered by ASP.NET. WCF REST + JSON Web services can also be created and consumed via Ajax calls from the browser.
3. For the cross-platform Application Client Runtime Critical Capability, we will rely on Android’s WebKit browser to provide HTML5 + CSS3 + ECMAScript5 capabilities. Offline storage is important to keep potentially disconnected Android working and this is facilitated by Web Storage which is accessible via JavaScript.
4. For the Security Critical Capability, Android 3.0 and higher provides hardware encryption based on the user’s device passcode for data-at-rest. Data-in-transit is secured via SSL and VPN. LDAP API support allows it to access corporate directory services.
5. For the Enterprise Application Integration Tools Critical Capability, Android can reach out to servers directly via Web Services or indirectly via SQL Server (JDBC) or BizTalk using SSIS/Adapters to connect to other enterprise packages.
6. The Multichannel Server Critical Capability to support any open protocol directly, via Reverse Proxy, or VPN is facilitated by ISA/TMG/UAG/IIS. Cross-Platform wire protocols riding on top of HTTP are exposed by Windows Communication Foundation (WCF) and include SOAP, REST and Atompub. Cross-Platform data serialization is also provided by WCF including XML, JSON, and OData. These Multichannel capabilities support thick clients making web service calls as well as thin web clients making Ajax calls. Distributed caching to dramatically boost the performance of any client is provided by Windows Server AppFabric Caching.
7. While the Hosting Critical Capability may not be as relevant in an on-premises scenario, Windows Azure Connect provides an IPSec-protected connection to the Cloud and SQL Azure Data Sync can be used to move data between SQL Server and SQL Azure.
8. For the Packaged Mobile Apps or Components Critical Capability, Android runs cross-platform mobile apps including Skype, Bing, MSN, Tag, Hotmail, and of course the critical ActiveSync component that makes push emails, contacts, calendars, and device management policies possible.

Newer versions of Android (3.x/4.0) are beginning to meet more of Gartner’s Critical Capabilities. It’s really improved in the last year in areas of encryption, but device fragmentation makes this improvement uneven.  The app story is still the ‘Wild West’ since the Android Market is an un-vetted free-for-all. This big ‘red flag’ has given rise to curated app stores like the one from Amazon.  As you can see from the picture, the big gap is with the client application runtime critical capability. Native development via Java/Eclipse is where Google wants to steer you and Microsoft doesn’t make native tools, runtimes or languages for this platform. You can definitely perform your own due diligence on Mono for Android from our friend Miguel de Icaza and his colleagues in order to reuse your existing .NET and C# skills. From a Microsoft perspective though, you’re definitely looking at HTML5 delivered via ASP.NET.

Next week, I’ll cover how Android connects to the Cloud.

Best Regards,

Rob

As you can see from the picture above:

1. For the Management Tools Critical Capability, iOS uses Microsoft Exchange for On-Premise policy enforcement via Exchange ActiveSync (EAS) but has no private software distribution equivalent to System Center Configuration Manager 2007 from Microsoft today. Instead, in-house apps are hosted and distributed via a web server over wireless by having a user click on a URL.  In the future, System Center Configuration Manager 2012 will be able to better manage iOS devices.
2. For both the Client and Server Integrated Development Environment (IDE) and Multichannel Tool Critical Capability, iOS uses Visual Studio. While the Server/EAI development functionality is the same as every other platform, endpoint development will consist of HTML5, ECMAScript 5, and CSS3 delivered by ASP.NET.  WCF REST + JSON Web services can also be created and consumed via Ajax calls from the browser.
3. For the cross-platform Application Client Runtime Critical Capability, we will rely on iOS’s WebKit browser called Safari to provide HTML5 + CSS3 + ECMAScript5 capabilities. Offline storage is important to keep potentially disconnected iPhones and iPads working and this is facilitated by Web Storage which is accessible via JavaScript.
4. For the Security Critical Capability, iOS provides AES 256 hardware encryption as well as Data Protection based on the user’s device passcode for data-at-rest. Data-in-transit is secured via SSL, VPN, and 802.1X.  Built-in LDAP support allows it to access corporate directory services.
5. For the Enterprise Application Integration Tools Critical Capability, iOS can reach out to servers directly via Web Services or indirectly via SQL Server or BizTalk using SSIS/Adapters to connect to other enterprise packages.
6. The Multichannel Server Critical Capability to support any open protocol directly, via Reverse Proxy, or VPN is facilitated by ISA/TMG/UAG/IIS. Crosss-Platform wire protocols riding on top of HTTP are exposed by Windows Communication Foundation (WCF) and include SOAP, REST and Atompub. Cross-Platform data serialization is also provided by WCF including XML, JSON, and OData. These Multichannel capabilities support thick clients making web service calls as well as thin web clients making Ajax calls. Distributed caching to dramatically boost the performance of any client is provided by Windows Server AppFabric Caching.
7. While the Hosting Critical Capability may not be as relevant in an on-premises scenario, Windows Azure Connect provides an IPSec-protected connection to the Cloud and SQL Azure Data Sync can be used to move data between SQL Server and SQL Azure.
8. For the Packaged Mobile Apps or Components Critical Capability, iOS runs cross-platform mobile apps including OneNote, Bing, Tag, and of course the critical ActiveSync component that makes push emails, contacts, calendars, and device management policies possible.

As you can see, iOS meets many of Gartner’s Critical Capabilities.  It’s really improved over the years in areas of security and device management.  As you can see from the picture, the big gap is with the client application runtime critical capability.  Native development via Xcode/Objective-C is where Apple wants to steer you and Microsoft doesn’t make native tools, runtimes or languages for this platform.  You can certainly kick the tires and perform your own due diligence on MonoTouch from our friend Miguel de Icaza and his colleagues in order to reuse your existing .NET and C# skills.  From a Microsoft perspective though, you’re definitely looking at HTML5 delivered via ASP.NET.

Next week, I’ll cover how iOS connects to the Cloud.

Best Regards,

Rob

When it comes to delivering server solutions with Windows Server and SQL Server, speed is your friend (as we used to say in the submarine service). More speed means more things can happen in a given period of time. If more things can happen in a given period of time, then you can derive greater scalability. Duh, Winning!

Okay, this stuff is obvious so let’s move on.

Have you ever noticed when you’ve finished installing your shiny new Windows Server 2008 R2 box, the default Power Plan is set to “Balanced?” Guess what kind of performance and scalability you get when you decide to “Go Green” and save the world with a “Balanced” power plan? Needless to say, you’re not making the most of the high-powered CPUs you just paid big money for.

So how does this relate to SQL Server and reducing I/O contention?

Would it surprise you to know that the amount of time your CPU’s spend processing your queries could actually double with a Balanced plan? If it takes more CPU time to execute a query, then imagine all those Merge Agent processes locking and blocking each other as they try to perform DML operations on the change tracking tables.

So what’s the takeaway here?

Set your Windows Server 2008 R2 power plan to High Performance! If you’re part of a Windows Domain and you need to make this setting stick, have your sys admin enforce this setting on all your SQL Servers via Group Policy.

Go fast or go home because your users care about performance.

-Rob

To pull this off, SQL Server has to use a more advanced set of tracking tables + additional trigger logic than simpler sync systems.  Since it does more work in advance, it goes without saying that the Merge Agents are busily creating extra contention through locking and blocking of all those extra tracking tables.  Luckily, you have some control over how many extra tracking tables are used depending on the partition choices you make.  The fewer tables you have to use, while still getting the functionality you need is one of the keys to reducing SQL Server contention.

Let’s take a quick look at the tracking tables to give you some context:

• MSmerge_contents (A row is inserted for each user table insert or update)
• MSmerge_tombstone (A row is inserted for user table delete)
• MSmerge_genhistory (A grouping of the above inserts called a generation)
• MSmerge_partition_groups (One row for each distinct partition defined)
• MSmerge_current_ partition_mappings (One row for each unique combination of rows in MSmerge_contents and MSmerge_partition_groups)
• MSmerge_past_partition_mappings (One row for each row that no longer belongs in a given partition)

Wow, that’s a bunch of tracking tables!

So imagine a bunch of Merge Agents performing SELECTS, INSERTS, UPDATES, and DELETES against all these tables in order to make the magic happen for the end user.  The more Merge Agents you having running concurrently, the more locking and blocking the system will experience which degrades performance.

What if I told you that you could reduce contention by eliminating 2 tracking tables from the equation?

Anyone who has setup Merge Replication might recognize the Add Filter dialog below:

You will see one of these dialog boxes for every filter you create.  For the purposes of this article, I just want you to focus on the 2 radio buttons at the bottom.  You get to choose that a row from a table will sync with multiple subscriptions or just one subscription.  If multiple users share the same distinct partition, you select the top one, because it’s an overlapping partition and it will need to use all 6 of the tracking tables I mentioned previously.  This means multiple users send the exact same HOST_NAME() filter value when they sync.  This might be a scenario where a group of sales professionals share the same region and clients, but nothing specific to themselves.

On the other hand, if you know that only one user subscribes to a given partition, then you select the bottom radio button because it’s a non-overlapping partition, and only uses the top 4 tracking tables I mentioned above.  This is actually the most common scenario I see with my customers.  Imagine a delivery driver that syncs down the data he needs to tell him where to go, what to do, and then uploads data he’s captured about deliveries.

So the big takeaway here, is to always select the bottom radio button so that SQL Server will create more efficient, non-overlapping partitions and use fewer tracking tables.

But wait, there’s more!

Non-overlapping partitions are less likely to create conflicts that have to be resolved by the built-in conflict resolvers.  If you’re not updating other users rows, then you’re not crashing into each other which leads to faster, smoother operation.

Back at the beginning of the article, I mentioned that there’s a quite of lot of code executed by triggers and associated stored procedures to make this sophisticated machine work.  Depending on your schema, the amount of data in the tracking tables, and the types of filters you’ve created, a potentially slow 5-way join is executed by sp_MSenumchanges. You can turn that into a 3-way join and speed up the system by sticking to non-overlapping partitions.  That was easy.

It’s all about UX, and trust me, your users will thank you for caring about this stuff!

Keep Synching,

Rob

We’ve come a long way and you’re probably wondering what qualifies this new embedded database to take on the Internet:

• Native support for x64 Windows Servers
• Virtual memory usage has been optimized to ensure the database can support up to 256 open connections – (Are you actually using 256 pooled connections with your “Big” database today?)
• Supports databases up to 4 GB in size – (Feel free to implement your own data sharding scheme)
• Developed, stress-tested, and tuned to support ASP.NET web applications
• Avoids the interprocess communications performance hit by running in-process with your web application
• Row-level locking to boost concurrency
• Step up to Government + Military grade security SHA2 algorithm to secure data with FIPS compliance
• Enhanced data reliability via true atomicity, consistency, isolation, and durability (ACID) support
• Transaction support to commit and roll back grouped changes
• Full referential integrity with cascading deletes and updates
• Support ADO.NET Entity Framework 4 – (Do I hear WCF Data Services?)
• Paging queries are supported via T-SQL syntax to only return the data you actually need

Wow, that’s quite a list!  SQL Server Compact 4.0 databases are easily developed using the new WebMatrix IDE or through Visual Studio 2010 SP1.  I’m loving the new ASP.NET Web Pages.  It reminds me of the good old days of building web applications with Classic ASP back in the 90′s with Visual InterDev and Homesite.

What about Mobility?

Since SQL Server Compact owes its heritage to mobile and embedded versions of Windows, you might be wanting to know what our story is there.  The good news is that you can build and deploy v4.0 databases on Windows XP, Windows Vista, and Windows 7.  If you want to implement an occasionally-connected solution that utilizes the Sync Framework, Remote Data Access (RDA), or Merge Replication, you’ll need to stick with SQL Server Compact 3.5 SP2.  Time and resource-constraints prevented the Compact team from enabling these features.  Luckily, single-user WPF/WinForms database applications running on Windows Slates, laptops and Windows Embedded Handheld devices will work just fine with the v3.5 SP2 runtime.  Get a jumpstart with this by pickup up “Enterprise Data Synchronization with Microsoft SQL Server 2008 and SQL Server Compact 3.5 Mobile Merge Replication” at   http://www.amazon.com/Enterprise-Synchronization-Microsoft-Compact-Replication/dp/0979891213/ref=sr_1_1?s=books&ie=UTF8&qid=1281715114&sr=1-1 to start building those MEAP solutions.

With the tidal wave of Windows Slates hitting the market, a secure, powerful mobile database that allows users to work offline and syncs with SQL Server is definitely going to be a hot item!

So run, don’t walk to the Microsoft Download site to download the Next-Gen database for the web:

http://www.microsoft.com/downloads/en/details.aspx?FamilyID=033cfb76-5382-44fb-bc7e-b3c8174832e2

If you need to support occasionally-connected mobile applications with sync capabilities on muliple Windows platforms, download SQL Server Compact 3.5 SP2:

http://www.microsoft.com/downloads/en/details.aspx?FamilyID=e497988a-c93a-404c-b161-3a0b323dce24

Keep Syncing,

Rob

Uniqueness is a key factor when synchronizing data between SQL Server/Azure and multiple endpoints like Slates and Smartphones.  With data simultaneously created and updated on servers and clients, ensuring rows are unique to avoid key collisions is critical.  As you know, each row is uniquely identified by its Primary Key.

When creating Primary Keys, it’s common to use a compound key based on things like account numbers, insert time and other appropriate business items.  It’s even more popular to create Identity Columns for the Primary Key based on an Int or BigInt data type based on what I see from my customers.  When you designate a column(s) to be a Primary Key, SQL Server automatically makes it a Clustered Index.  Clustered indexes are faster than normal indexes for sequential values because the B-Tree leaf nodes are the actual data pages on disk, rather than just pointers to data pages.

While Identity Columns work well in most database situations, they often break down in a data synchronization scenario since multiple clients could find themselves creating new rows using the same key value.  When these clients sync their data with SQL Server, key collisions would occur.  Merge Replication includes a feature that hands out blocks of Identity Ranges to each client to prevent this. 

When using other Microsoft sync technologies like the Sync Framework or RDA, no such Identity Range mechanism exists and therefore I often see GUIDs utilized as Primary Keys to ensure uniqueness across all endpoints.  In fact, I see this more and more with Merge Replication too since SQL Server adds a GUID column to the end of each row for tracking purposes anyway.  Two birds get killed with one Uniqueidentifier stone. 

Using the Uniqueidentifier data type is not necessarily a bad idea.  Despite the tradeoff of reduced join performance vs. integers, the solved uniqueness problem allows sync pros to sleep better at night.  The primary drawback with using GUIDs as Primary Keys goes back to the fact that SQL Server automatically gives those columns a Clustered Index.

I thought Clustered Indexes were a good thing?

They are a good thing when the values found in the indexed column are sequential.  Unfortunately, GUIDs generated with the default NewId() function are completely random and therefore create a serious performance problem.  All those mobile devices uploading captured data means lots of Inserts for SQL Server.  Inserting random key values like GUIDs can cause fragmentation in excess of 90% because new pages have to be allocated with rows pushed to the new page in order to insert the record on the existing page.  This performance-killing, space-wasting page splitting wouldn’t happen with sequential Integers or Datetime values since they actually help fill the existing page.

 What about NEWSEQUENTIALID()?

Generating your GUIDs on SQL Server with this function will dramatically reduce fragmentation and wasted space since it guarantees that each GUID will be sequential.  Unfortunately, this isn’t bulletproof.  If your Windows Server is restarted for any reason, your GUIDs may start from a lower range.  They’ll still be globally unique, but your fragmentation will increase and performance will decrease.  Also keep in mind that all the devices synchronizing with SQL Server will be creating their own GUIDs which blows the whole NEWSEQUENTIALID() strategy out of the water.

Takeaway

If you’re going to use the Uniqueidentifier data type for your Primary Keys and you plan to sync your data with RDA, the Sync Framework or Merge Replication, ensure that Create as Clustered == No for better performance.  You’ll still get fragmentation, but it will be closer to the ~30% range instead almost 100%.

Keep synching

Rob

In my last Sync/Contention post, I beat up on a select group of SAN administrators who aren’t willing to go the extra mile to optimize the very heart of their organization, SQL Server.  You guys know who you are.

This time, I want to look at something more basic, yet often overlooked.

All DBAs know that Joining tables on non-indexed columns is the most expensive operation SQL Server can perform.  Amazingly, I run into this problem over and over with many of my customers.  Sync technologies like the Sync Framework, RDA and Merge Replication allow for varying levels of server-side filtering.  This is a popular feature used to reduce the size of the tables and rows being downloaded to Silverlight Isolated Storage or SQL Server Compact. 

It’s also a performance killer when tables and columns participating in a Join filter are not properly indexed.  Keeping rows locked longer than necessary creates undue blocking and deadlocking.  It also creates unhappy slate and smartphone users who have to wait longer for their sync to complete.

Do yourselft a favor and go take a look at all the filters you’ve created and makes sure that you have indexes on all those Joined columns.

Keep synching,

Rob

Sync technologies like Merge Replication and the Sync Framework track changes on SQL Server using triggers, stored procedures and special tracking tables.  The act of tracking changes made by each SQL Server Compact or Silverlight sync subscriber can cause a lot of locking and blocking on the server.  This diminishes performance and sometimes leads to deadlocks. 

Therefore, don’t listen to your SAN administrator when he says the RAID 5 will do.  RAID 1 or 10 must always be used for all databases, tempdb, and transaction logs.  Furthermore, each of these database objects must be placed on their own dedicated RAID arrays.  No sharing!  Remembers, as a DBA and sync expert, knowledge of SAN configuration must always be part of your skillset.

Keeping synching,

Rob