• If you care about speed & scalability, your web service will always check the distributed cache before querying the database.
• If you have to get your answer from your database, make sure to add the answer to your distributed cache concurrently with returning it to the user.
• Don’t ever call your primary database directly.  Call one of your horizontally-scaled relational or NoSQL replicas.
• Use download-only replication to create as many read-only replicas as you need.  Your replicas can contain complete copies of the primary database or just shards of data as appropriate.
• When sending data up from the clients to your servers, never let your web services insert, update, or delete directly against the primary database or the replicas.
• Have your web services drop that uploaded data in a queue and update the distributed cache while letting worker processes perform those DML operations against the primary database.
• Use peer-to-peer replication to maintain more than one read/write primary database at the top of your hierarchy.
• Remember to create and maintain more than one hierarchy in different geographies using the same peer-to-peer replication to provide even more availability, scalability, and performance to your mobile users all around the world.
• Your mobile users will be connecting to your service via slow, intermittent, unpredictable wireless data networks so always use small, efficient wire protocols like REST and data serialization like JSON.

-Rob

The new Sync Framework Toolkit found on MSDN builds on the existing Sync Framework 2.1′s ability to create disconnected applications, making it easier to expose data for synchronization to apps running on any client platform.  Where Sync Framework 2.1 required clients to be based on Windows, this free toolkit allows other Microsoft platforms to be used for offline clients such as Silverlight, Windows Phone 7, Windows Mobile, Windows Embedded Handheld, and new Windows Slates.   Additionally, non-Microsoft platforms such as iPhones, iPads, Android phones and tablets, Blackberries and browsers supporting HTML5 are all first-class sync citizens.  The secret is that we no longer require the installation of the Sync Framework runtime on client devices.  When coupled with use of an open protocol like OData for data transport, no platform or programming language is prevented from synchronizing data with our on-premise and cloud databases.  When the data arrives on your device, you can serialize it as JSON, or insert it into SQL Server Compact or SQLite depending on your platform preferences.

The Sync Framework Toolkit provides all the features enabled by theSync Framework 4.0 October 2010 CTP.  We are releasing the toolkit as source code samples on MSDN with the source code utilizing Sync Framework 2.1.  Source code provides the flexibility to customize or extend the capabilities we have provided to suit your specific requirements. The client-side source code in the package is released under the Apache 2.0 license and the server-side source code under the MS-LPL license.  The Sync Framework 2.1 is fully supported by Microsoft and the mobile-enabling source code is yours to use, build upon, and support for the apps you create.

Now some of you might be wondering why you would use a sync technology to move data rather than SOAP or REST web services.  The reason has to do with performance and bandwidth efficiency.  Using SOA, one would retrieve all the data needed to the device in order to see what has changed in SQL Server.  The same goes for uploading data.  Using the Sync Framework Toolkit, only the changes, or deltas, are transmitted over the air.  The boosts performance and reduces bandwidth usage which saves time and money in a world of congested mobile data networks with capped mobile data plans.  You also get a feature called batching, which breaks up the data sent over wireless networks into manageable pieces.  This not only prevents you from blowing out your limited bandwidth, but it also keeps you from using too much RAM memory both on the server and your memory-constrained mobile device.  When combined with conflict resolution and advanced filtering, I’m sold!

I think you’ll find the Sync Framework Toolkit to be an immensely valuable component of your MEAP solutions for the enterprise as well as the ones you build for consumers.

Keep Synching,

Rob

Since Windows Embedded Handheld uses an advanced version of the operating system kernel used by Windows Mobile 6.5.3, many of the techniques and best practices I’ve taugh customers and developers all over the world still apply.  While it still uses the slotted memory model found in Windows CE 5 with 32 processes and 32 MB of memory per process, you’ll find that numerous enhancements and tuning has taken place to give your line of business apps more of what they need.  I’m talking about more memory per process and improved performance.  Therefore, I’d like you to sit back and watch the video of a presentation I delivered at Tech Ed in Los Angeles a couple of years ago so you can better learn what this mobile platform has to offer in the form of better memory management and improved performance:

A recent Gartner report recommends that organizations should stay with Windows Embedded Handheld as the best mobile platform for enterprise line of business needs.  Great devices are available from OEMs like Intermec, Motorola, Psion, and Honeywell just to name a few.  I hope this video helps you with any memory management or performance issues you may need to deal with in your enterprise mobile apps.

Best Regards,
Rob

We’ve come a long way and you’re probably wondering what qualifies this new embedded database to take on the Internet:

• Native support for x64 Windows Servers
• Virtual memory usage has been optimized to ensure the database can support up to 256 open connections – (Are you actually using 256 pooled connections with your “Big” database today?)
• Supports databases up to 4 GB in size – (Feel free to implement your own data sharding scheme)
• Developed, stress-tested, and tuned to support ASP.NET web applications
• Avoids the interprocess communications performance hit by running in-process with your web application
• Row-level locking to boost concurrency
• Step up to Government + Military grade security SHA2 algorithm to secure data with FIPS compliance
• Enhanced data reliability via true atomicity, consistency, isolation, and durability (ACID) support
• Transaction support to commit and roll back grouped changes
• Full referential integrity with cascading deletes and updates
• Support ADO.NET Entity Framework 4 – (Do I hear WCF Data Services?)
• Paging queries are supported via T-SQL syntax to only return the data you actually need

Wow, that’s quite a list!  SQL Server Compact 4.0 databases are easily developed using the new WebMatrix IDE or through Visual Studio 2010 SP1.  I’m loving the new ASP.NET Web Pages.  It reminds me of the good old days of building web applications with Classic ASP back in the 90′s with Visual InterDev and Homesite.

What about Mobility?

Since SQL Server Compact owes its heritage to mobile and embedded versions of Windows, you might be wanting to know what our story is there.  The good news is that you can build and deploy v4.0 databases on Windows XP, Windows Vista, and Windows 7.  If you want to implement an occasionally-connected solution that utilizes the Sync Framework, Remote Data Access (RDA), or Merge Replication, you’ll need to stick with SQL Server Compact 3.5 SP2.  Time and resource-constraints prevented the Compact team from enabling these features.  Luckily, single-user WPF/WinForms database applications running on Windows Slates, laptops and Windows Embedded Handheld devices will work just fine with the v3.5 SP2 runtime.  Get a jumpstart with this by pickup up “Enterprise Data Synchronization with Microsoft SQL Server 2008 and SQL Server Compact 3.5 Mobile Merge Replication” at   http://www.amazon.com/Enterprise-Synchronization-Microsoft-Compact-Replication/dp/0979891213/ref=sr_1_1?s=books&ie=UTF8&qid=1281715114&sr=1-1 to start building those MEAP solutions.

With the tidal wave of Windows Slates hitting the market, a secure, powerful mobile database that allows users to work offline and syncs with SQL Server is definitely going to be a hot item!

So run, don’t walk to the Microsoft Download site to download the Next-Gen database for the web:

http://www.microsoft.com/downloads/en/details.aspx?FamilyID=033cfb76-5382-44fb-bc7e-b3c8174832e2

If you need to support occasionally-connected mobile applications with sync capabilities on muliple Windows platforms, download SQL Server Compact 3.5 SP2:

http://www.microsoft.com/downloads/en/details.aspx?FamilyID=e497988a-c93a-404c-b161-3a0b323dce24

Keep Syncing,

Rob

Uniqueness is a key factor when synchronizing data between SQL Server/Azure and multiple endpoints like Slates and Smartphones.  With data simultaneously created and updated on servers and clients, ensuring rows are unique to avoid key collisions is critical.  As you know, each row is uniquely identified by its Primary Key.

When creating Primary Keys, it’s common to use a compound key based on things like account numbers, insert time and other appropriate business items.  It’s even more popular to create Identity Columns for the Primary Key based on an Int or BigInt data type based on what I see from my customers.  When you designate a column(s) to be a Primary Key, SQL Server automatically makes it a Clustered Index.  Clustered indexes are faster than normal indexes for sequential values because the B-Tree leaf nodes are the actual data pages on disk, rather than just pointers to data pages.

While Identity Columns work well in most database situations, they often break down in a data synchronization scenario since multiple clients could find themselves creating new rows using the same key value.  When these clients sync their data with SQL Server, key collisions would occur.  Merge Replication includes a feature that hands out blocks of Identity Ranges to each client to prevent this. 

When using other Microsoft sync technologies like the Sync Framework or RDA, no such Identity Range mechanism exists and therefore I often see GUIDs utilized as Primary Keys to ensure uniqueness across all endpoints.  In fact, I see this more and more with Merge Replication too since SQL Server adds a GUID column to the end of each row for tracking purposes anyway.  Two birds get killed with one Uniqueidentifier stone. 

Using the Uniqueidentifier data type is not necessarily a bad idea.  Despite the tradeoff of reduced join performance vs. integers, the solved uniqueness problem allows sync pros to sleep better at night.  The primary drawback with using GUIDs as Primary Keys goes back to the fact that SQL Server automatically gives those columns a Clustered Index.

I thought Clustered Indexes were a good thing?

They are a good thing when the values found in the indexed column are sequential.  Unfortunately, GUIDs generated with the default NewId() function are completely random and therefore create a serious performance problem.  All those mobile devices uploading captured data means lots of Inserts for SQL Server.  Inserting random key values like GUIDs can cause fragmentation in excess of 90% because new pages have to be allocated with rows pushed to the new page in order to insert the record on the existing page.  This performance-killing, space-wasting page splitting wouldn’t happen with sequential Integers or Datetime values since they actually help fill the existing page.

 What about NEWSEQUENTIALID()?

Generating your GUIDs on SQL Server with this function will dramatically reduce fragmentation and wasted space since it guarantees that each GUID will be sequential.  Unfortunately, this isn’t bulletproof.  If your Windows Server is restarted for any reason, your GUIDs may start from a lower range.  They’ll still be globally unique, but your fragmentation will increase and performance will decrease.  Also keep in mind that all the devices synchronizing with SQL Server will be creating their own GUIDs which blows the whole NEWSEQUENTIALID() strategy out of the water.

Takeaway

If you’re going to use the Uniqueidentifier data type for your Primary Keys and you plan to sync your data with RDA, the Sync Framework or Merge Replication, ensure that Create as Clustered == No for better performance.  You’ll still get fragmentation, but it will be closer to the ~30% range instead almost 100%.

Keep synching

Rob

In my last Sync/Contention post, I beat up on a select group of SAN administrators who aren’t willing to go the extra mile to optimize the very heart of their organization, SQL Server.  You guys know who you are.

This time, I want to look at something more basic, yet often overlooked.

All DBAs know that Joining tables on non-indexed columns is the most expensive operation SQL Server can perform.  Amazingly, I run into this problem over and over with many of my customers.  Sync technologies like the Sync Framework, RDA and Merge Replication allow for varying levels of server-side filtering.  This is a popular feature used to reduce the size of the tables and rows being downloaded to Silverlight Isolated Storage or SQL Server Compact. 

It’s also a performance killer when tables and columns participating in a Join filter are not properly indexed.  Keeping rows locked longer than necessary creates undue blocking and deadlocking.  It also creates unhappy slate and smartphone users who have to wait longer for their sync to complete.

Do yourselft a favor and go take a look at all the filters you’ve created and makes sure that you have indexes on all those Joined columns.

Keep synching,

Rob

Sync technologies like Merge Replication and the Sync Framework track changes on SQL Server using triggers, stored procedures and special tracking tables.  The act of tracking changes made by each SQL Server Compact or Silverlight sync subscriber can cause a lot of locking and blocking on the server.  This diminishes performance and sometimes leads to deadlocks. 

Therefore, don’t listen to your SAN administrator when he says the RAID 5 will do.  RAID 1 or 10 must always be used for all databases, tempdb, and transaction logs.  Furthermore, each of these database objects must be placed on their own dedicated RAID arrays.  No sharing!  Remembers, as a DBA and sync expert, knowledge of SAN configuration must always be part of your skillset.

Keeping synching,

Rob

In this case, each ObservableCollection represents a table, each object represents a row, and each object property represents a column.  I had you create a Singleton class to hold all these objects in memory to serve as the database.  The fact that Silverlight supports Language Integrated Query (LINQ) means that you can use SQL-like statements to work with the multiple, ObservableCollections of objects. 

If you’re wondering why I have you cache everything in memory in a Singleton, there’s a few reasons.  For starters, it makes it easy to query everything with LINQ with the fastest performance possible for single and multi-table JOINs.  Secondly, I don’t represent a Microsoft product group and therefore wouldn’t engineer an unsupported provider that can query subsets of serialized data from files residing in Isolated Storage.  Finally, I don’t want you to accidentally find yourself with multiple instances of the same ObservableCollection when pulling data down from Azure or loading it from Isolated Storage.  Forcing everything into a Singleton prevents you wasting memory or updating objects in the wrong instance of an ObservableCollection.  An inconsistent database is not a good thing.  Don’t worry, you can control which tables are loaded into memory.

So what is this article all about and what are the “improvements” I’m talking about?

This time around, I’m going to focus on saving, loading and deleting the serialized ObservableCollections from Isolated Storage.  In that last article, I showed you how to serialize/de-serialize the ObservableCollections to and from Isolated Storage using the XmlSerializer.  This made it easy for you to save each table to its own XML file which sounds pretty cool.

So what’s wrong with this?

Saving anything as XML means that you’re using the largest, most verbose form of serialization.  After hearing me preach about the virtues of doing SOA with WCF REST + JSON, using the XmlSerializer probably seems out of place.  Luckily, the DataContractJsonSerializer supported by Silverlight on Windows Phone 7 gives you the most efficient wire protocol for data-in-transit can also be used to save those same .NET objects to Isolated Storage.  So the first improvement in this article comes from shrinking the size of the tables and improving the efficiency of the serialization/de-serializing operations to Isolated Storage using out-of-the-box functionality. 

While going from XML to JSON for your serializing might be good enough, there’s another improvement in the way you write the code that will make this much easier to implement for your own projects.  A look back to the previous article reveals a tight coupling between the tables that needed to be saved/loaded and the code needed to make that happen.  This meant that you would have to create a SaveTable and LoadTable method for each table that you wanted to retrieve from Azure.  The new code you’re about to see is generic and allows you to use a single SaveTable and LoadTable method even if you decide to download 100 tables.

Enough talk already, let’s see some code.  Launch your ContosoCloud solution in Visual Studio and open Database.cs.  I want you to overwrite the existing code with the code shown below:

using System;
using System.Net;
using System.Windows;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.IO.IsolatedStorage;
using System.Runtime.Serialization.Json;

namespace ContosoPhone
{
    sealed class Database
    {
        //Declare Instance
        private static readonly Database instance = new Database();

        //Private Constructor
        private Database() { }

        //The entry point into this Database
        public static Database Instance
        {
            get
            {
                return instance;
            }
        }

        //Serialize ObservableCollection to JSON in Isolated Storage
        public void SaveTable<T>(T tableToSave, string tableName)
        {
            if (tableToSave != null)
            {
                using (IsolatedStorageFile store = IsolatedStorageFile.GetUserStoreForApplication())
                {
                    using (IsolatedStorageFileStream stream = store.CreateFile(tableName + ".txt"))
                    {
                        DataContractJsonSerializer serializer = new DataContractJsonSerializer(typeof(T));
                        serializer.WriteObject(stream, tableToSave);
                    }
                }
            }
            else
            {
                throw new Exception("Table is empty");
            }
        }

        //Deserialize ObservableCollection from JSON in Isolated Storage
        public T LoadTable<T>(T tableToLoad, string tableName)
        {
            using (IsolatedStorageFile store = IsolatedStorageFile.GetUserStoreForApplication())
            {
                if (store.FileExists(tableName + ".txt"))
                {
                    using (IsolatedStorageFileStream stream = store.OpenFile(tableName + ".txt", System.IO.FileMode.Open))
                    {
                        DataContractJsonSerializer serializer = new DataContractJsonSerializer(typeof(T));
                        return (T)serializer.ReadObject(stream);
                    }
                }
                else
                {
                    throw new Exception("Table not found");
                }
            }
        }

        //Delete ObservableCollection from Isolated Storage
        public void DropTable(string tableName)
        {
            using (IsolatedStorageFile store = IsolatedStorageFile.GetUserStoreForApplication())
            {
                if (store.FileExists(tableName + ".txt"))
                {
                    store.DeleteFile(tableName + ".txt");
                }
                else
                {
                    throw new Exception("Table not found");
                }
            }
        }

       
        //Declare Private Table Variables
        private ObservableCollection<Customer> customerTable = null;

        //Customer Table
        public ObservableCollection<Customer> Customers
        {
            get { return customerTable; }
            set { customerTable = value; }
        }
    }
}

Looking from top to bottom, the first change you’ll notice is the new SaveTable method where you pass in the desired ObservableCollection and table name in order to serialize it as JSON using the DataContractJsonSerializer.  The next method down the list is LoadTable where you pass in the same parameters as SaveTable but you get back a de-serialized ObservableCollection.  The last new method in the Database Singleton is DropTable which simply deletes the serialized table from Isolated Storage if you don’t need it anymore.

So how do you call this code?

Bring up MainPage.xaml.cs, and find the click event for Save button.  Delete the existing XmlSerializer code and replace it with the following:

try
{
    Database.Instance.SaveTable<ObservableCollection<Customer>>(Database.Instance.Customers, "Customers");
}
catch (Exception ex)
{
    MessageBox.Show(ex.Message);
}

The code above shows you how to call the SaveTable method in the Singleton with the appropriate syntax to pass in the ObservableCollection type as well as actual ObservableCollection value and name.

Now find the click event for the Load button, delete the existing code and paste in the following:

try
{
    Database.Instance.Customers = Database.Instance.LoadTable<ObservableCollection<Customer>>(Database.Instance.Customers, "Customers");
}
catch (Exception ex)
{
    MessageBox.Show(ex.Message);
}

This code looks pretty much the same as the SaveTable code except that you set Database.Instance.Customers equal to the return value from the method.  For completeness sake, drop another button on MainPage.xaml and call it Drop.  In its click event, paste in the following code:

try
{
    Database.Instance.DropTable("Customers");
}
catch (Exception ex)
{
    MessageBox.Show(ex.Message);
}

For this code, just pass in the name of the table you want to delete from Isolated Storage and it’s gone.

It’s time to hit F5 so you can see how things behave.

When your app comes to life in the emulator, I want you to exercise the system by Getting, Adding, Updating and Deleting Customers.  In between, I want you to tap the Save button, close the app, reload the app and tap the Load button and then View Customers to ensure you’re seeing the list of Customers you expect.  Keep in mind that when you Save, you overwrite the previously saved table.  Likewise, when you Load, you overwrite the current in-memory ObservableCollection.  Additionally, Saving, Loading, and Dropping tables that don’t exist should throw an appropriate error message.

So what’s the big takeaway for these tweaks I’ve made to the in-memory database?

While switching serialization from XML to JSON is a great improvement in size and efficiency, I truly believe that making the SaveTable and LoadTable methods generic and reusable will boost developer productivity.  The new ease with which you can Save and Load 1, 10 or even 1,000 tables makes this more attractive to mobile developers that need to work with local data.

So where do we go from here?

You now have some of the basic elements of a database on Windows Phone 7.  You don’t have ACID support, indexes, stored procedures or triggers but you have a foundation to build on.  So what should be built next? 

To help ensure database consistency, I would add an AutoFlush feature next.  SQL Server Compact flushes its data to disk every 10 seconds and there’s nothing to prevent you from using the SaveTable method to do the same.  A timer set to fire at a user-specified interval that iterates through all the ObservableCollections and saves them will help keep your data safe from battery loss and unforeseen system failures.  The fact that your app can be tombstoned at any moment when a user taps the Back button makes an AutoFlush feature even more important.

Anything else?

At the beginning of this article I mentioned RDA which is a simple form of data synchronization.  It’s simple because it only tracks changes on the client but not the server.  To find out what’s new or changed on the server, RDA requires local tables on the device to be dropped and then re-downloaded from SQL Server.  With the system I’ve built and described throughout this series of articles, we already have this brute force functionality.  So what’s missing is client-side change tracking.  To do this, I would need to add code that fires during INSERTS, UPDATES, and DELETES and then writes the appropriate information to local tracking tables.  To push those changes back to SQL Azure, appropriate code would need to call WCF REST + JSON Services that execute DML code on Windows Azure.

I hope with the improvements I’ve made to the in-memory database in this article, you’ll feel even more empowered to build occasionally-connected Windows Phone 7 solutions for consumers and the enterprise.

Keep coding!

-Rob