• If you care about speed & scalability, your web service will always check the distributed cache before querying the database.
• If you have to get your answer from your database, make sure to add the answer to your distributed cache concurrently with returning it to the user.
• Don’t ever call your primary database directly.  Call one of your horizontally-scaled relational or NoSQL replicas.
• Use download-only replication to create as many read-only replicas as you need.  Your replicas can contain complete copies of the primary database or just shards of data as appropriate.
• When sending data up from the clients to your servers, never let your web services insert, update, or delete directly against the primary database or the replicas.
• Have your web services drop that uploaded data in a queue and update the distributed cache while letting worker processes perform those DML operations against the primary database.
• Use peer-to-peer replication to maintain more than one read/write primary database at the top of your hierarchy.
• Remember to create and maintain more than one hierarchy in different geographies using the same peer-to-peer replication to provide even more availability, scalability, and performance to your mobile users all around the world.
• Your mobile users will be connecting to your service via slow, intermittent, unpredictable wireless data networks so always use small, efficient wire protocols like REST and data serialization like JSON.

-Rob

The new Sync Framework Toolkit found on MSDN builds on the existing Sync Framework 2.1′s ability to create disconnected applications, making it easier to expose data for synchronization to apps running on any client platform.  Where Sync Framework 2.1 required clients to be based on Windows, this free toolkit allows other Microsoft platforms to be used for offline clients such as Silverlight, Windows Phone 7, Windows Mobile, Windows Embedded Handheld, and new Windows Slates.   Additionally, non-Microsoft platforms such as iPhones, iPads, Android phones and tablets, Blackberries and browsers supporting HTML5 are all first-class sync citizens.  The secret is that we no longer require the installation of the Sync Framework runtime on client devices.  When coupled with use of an open protocol like OData for data transport, no platform or programming language is prevented from synchronizing data with our on-premise and cloud databases.  When the data arrives on your device, you can serialize it as JSON, or insert it into SQL Server Compact or SQLite depending on your platform preferences.

The Sync Framework Toolkit provides all the features enabled by theSync Framework 4.0 October 2010 CTP.  We are releasing the toolkit as source code samples on MSDN with the source code utilizing Sync Framework 2.1.  Source code provides the flexibility to customize or extend the capabilities we have provided to suit your specific requirements. The client-side source code in the package is released under the Apache 2.0 license and the server-side source code under the MS-LPL license.  The Sync Framework 2.1 is fully supported by Microsoft and the mobile-enabling source code is yours to use, build upon, and support for the apps you create.

Now some of you might be wondering why you would use a sync technology to move data rather than SOAP or REST web services.  The reason has to do with performance and bandwidth efficiency.  Using SOA, one would retrieve all the data needed to the device in order to see what has changed in SQL Server.  The same goes for uploading data.  Using the Sync Framework Toolkit, only the changes, or deltas, are transmitted over the air.  The boosts performance and reduces bandwidth usage which saves time and money in a world of congested mobile data networks with capped mobile data plans.  You also get a feature called batching, which breaks up the data sent over wireless networks into manageable pieces.  This not only prevents you from blowing out your limited bandwidth, but it also keeps you from using too much RAM memory both on the server and your memory-constrained mobile device.  When combined with conflict resolution and advanced filtering, I’m sold!

I think you’ll find the Sync Framework Toolkit to be an immensely valuable component of your MEAP solutions for the enterprise as well as the ones you build for consumers.

Keep Synching,

Rob

Since Windows Embedded Handheld uses an advanced version of the operating system kernel used by Windows Mobile 6.5.3, many of the techniques and best practices I’ve taugh customers and developers all over the world still apply.  While it still uses the slotted memory model found in Windows CE 5 with 32 processes and 32 MB of memory per process, you’ll find that numerous enhancements and tuning has taken place to give your line of business apps more of what they need.  I’m talking about more memory per process and improved performance.  Therefore, I’d like you to sit back and watch the video of a presentation I delivered at Tech Ed in Los Angeles a couple of years ago so you can better learn what this mobile platform has to offer in the form of better memory management and improved performance:

A recent Gartner report recommends that organizations should stay with Windows Embedded Handheld as the best mobile platform for enterprise line of business needs.  Great devices are available from OEMs like Intermec, Motorola, Psion, and Honeywell just to name a few.  I hope this video helps you with any memory management or performance issues you may need to deal with in your enterprise mobile apps.

Best Regards,
Rob

We’ve come a long way and you’re probably wondering what qualifies this new embedded database to take on the Internet:

• Native support for x64 Windows Servers
• Virtual memory usage has been optimized to ensure the database can support up to 256 open connections – (Are you actually using 256 pooled connections with your “Big” database today?)
• Supports databases up to 4 GB in size – (Feel free to implement your own data sharding scheme)
• Developed, stress-tested, and tuned to support ASP.NET web applications
• Avoids the interprocess communications performance hit by running in-process with your web application
• Row-level locking to boost concurrency
• Step up to Government + Military grade security SHA2 algorithm to secure data with FIPS compliance
• Enhanced data reliability via true atomicity, consistency, isolation, and durability (ACID) support
• Transaction support to commit and roll back grouped changes
• Full referential integrity with cascading deletes and updates
• Support ADO.NET Entity Framework 4 – (Do I hear WCF Data Services?)
• Paging queries are supported via T-SQL syntax to only return the data you actually need

Wow, that’s quite a list!  SQL Server Compact 4.0 databases are easily developed using the new WebMatrix IDE or through Visual Studio 2010 SP1.  I’m loving the new ASP.NET Web Pages.  It reminds me of the good old days of building web applications with Classic ASP back in the 90′s with Visual InterDev and Homesite.

What about Mobility?

Since SQL Server Compact owes its heritage to mobile and embedded versions of Windows, you might be wanting to know what our story is there.  The good news is that you can build and deploy v4.0 databases on Windows XP, Windows Vista, and Windows 7.  If you want to implement an occasionally-connected solution that utilizes the Sync Framework, Remote Data Access (RDA), or Merge Replication, you’ll need to stick with SQL Server Compact 3.5 SP2.  Time and resource-constraints prevented the Compact team from enabling these features.  Luckily, single-user WPF/WinForms database applications running on Windows Slates, laptops and Windows Embedded Handheld devices will work just fine with the v3.5 SP2 runtime.  Get a jumpstart with this by pickup up “Enterprise Data Synchronization with Microsoft SQL Server 2008 and SQL Server Compact 3.5 Mobile Merge Replication” at   http://www.amazon.com/Enterprise-Synchronization-Microsoft-Compact-Replication/dp/0979891213/ref=sr_1_1?s=books&ie=UTF8&qid=1281715114&sr=1-1 to start building those MEAP solutions.

With the tidal wave of Windows Slates hitting the market, a secure, powerful mobile database that allows users to work offline and syncs with SQL Server is definitely going to be a hot item!

So run, don’t walk to the Microsoft Download site to download the Next-Gen database for the web:

http://www.microsoft.com/downloads/en/details.aspx?FamilyID=033cfb76-5382-44fb-bc7e-b3c8174832e2

If you need to support occasionally-connected mobile applications with sync capabilities on muliple Windows platforms, download SQL Server Compact 3.5 SP2:

http://www.microsoft.com/downloads/en/details.aspx?FamilyID=e497988a-c93a-404c-b161-3a0b323dce24

Keep Syncing,

Rob

Uniqueness is a key factor when synchronizing data between SQL Server/Azure and multiple endpoints like Slates and Smartphones.  With data simultaneously created and updated on servers and clients, ensuring rows are unique to avoid key collisions is critical.  As you know, each row is uniquely identified by its Primary Key.

When creating Primary Keys, it’s common to use a compound key based on things like account numbers, insert time and other appropriate business items.  It’s even more popular to create Identity Columns for the Primary Key based on an Int or BigInt data type based on what I see from my customers.  When you designate a column(s) to be a Primary Key, SQL Server automatically makes it a Clustered Index.  Clustered indexes are faster than normal indexes for sequential values because the B-Tree leaf nodes are the actual data pages on disk, rather than just pointers to data pages.

While Identity Columns work well in most database situations, they often break down in a data synchronization scenario since multiple clients could find themselves creating new rows using the same key value.  When these clients sync their data with SQL Server, key collisions would occur.  Merge Replication includes a feature that hands out blocks of Identity Ranges to each client to prevent this. 

When using other Microsoft sync technologies like the Sync Framework or RDA, no such Identity Range mechanism exists and therefore I often see GUIDs utilized as Primary Keys to ensure uniqueness across all endpoints.  In fact, I see this more and more with Merge Replication too since SQL Server adds a GUID column to the end of each row for tracking purposes anyway.  Two birds get killed with one Uniqueidentifier stone. 

Using the Uniqueidentifier data type is not necessarily a bad idea.  Despite the tradeoff of reduced join performance vs. integers, the solved uniqueness problem allows sync pros to sleep better at night.  The primary drawback with using GUIDs as Primary Keys goes back to the fact that SQL Server automatically gives those columns a Clustered Index.

I thought Clustered Indexes were a good thing?

They are a good thing when the values found in the indexed column are sequential.  Unfortunately, GUIDs generated with the default NewId() function are completely random and therefore create a serious performance problem.  All those mobile devices uploading captured data means lots of Inserts for SQL Server.  Inserting random key values like GUIDs can cause fragmentation in excess of 90% because new pages have to be allocated with rows pushed to the new page in order to insert the record on the existing page.  This performance-killing, space-wasting page splitting wouldn’t happen with sequential Integers or Datetime values since they actually help fill the existing page.

 What about NEWSEQUENTIALID()?

Generating your GUIDs on SQL Server with this function will dramatically reduce fragmentation and wasted space since it guarantees that each GUID will be sequential.  Unfortunately, this isn’t bulletproof.  If your Windows Server is restarted for any reason, your GUIDs may start from a lower range.  They’ll still be globally unique, but your fragmentation will increase and performance will decrease.  Also keep in mind that all the devices synchronizing with SQL Server will be creating their own GUIDs which blows the whole NEWSEQUENTIALID() strategy out of the water.

Takeaway

If you’re going to use the Uniqueidentifier data type for your Primary Keys and you plan to sync your data with RDA, the Sync Framework or Merge Replication, ensure that Create as Clustered == No for better performance.  You’ll still get fragmentation, but it will be closer to the ~30% range instead almost 100%.

Keep synching

Rob

In my last Sync/Contention post, I beat up on a select group of SAN administrators who aren’t willing to go the extra mile to optimize the very heart of their organization, SQL Server.  You guys know who you are.

This time, I want to look at something more basic, yet often overlooked.

All DBAs know that Joining tables on non-indexed columns is the most expensive operation SQL Server can perform.  Amazingly, I run into this problem over and over with many of my customers.  Sync technologies like the Sync Framework, RDA and Merge Replication allow for varying levels of server-side filtering.  This is a popular feature used to reduce the size of the tables and rows being downloaded to Silverlight Isolated Storage or SQL Server Compact. 

It’s also a performance killer when tables and columns participating in a Join filter are not properly indexed.  Keeping rows locked longer than necessary creates undue blocking and deadlocking.  It also creates unhappy slate and smartphone users who have to wait longer for their sync to complete.

Do yourselft a favor and go take a look at all the filters you’ve created and makes sure that you have indexes on all those Joined columns.

Keep synching,

Rob

Sync technologies like Merge Replication and the Sync Framework track changes on SQL Server using triggers, stored procedures and special tracking tables.  The act of tracking changes made by each SQL Server Compact or Silverlight sync subscriber can cause a lot of locking and blocking on the server.  This diminishes performance and sometimes leads to deadlocks. 

Therefore, don’t listen to your SAN administrator when he says the RAID 5 will do.  RAID 1 or 10 must always be used for all databases, tempdb, and transaction logs.  Furthermore, each of these database objects must be placed on their own dedicated RAID arrays.  No sharing!  Remembers, as a DBA and sync expert, knowledge of SAN configuration must always be part of your skillset.

Keeping synching,

Rob

In this case, each ObservableCollection represents a table, each object represents a row, and each object property represents a column.  I had you create a Singleton class to hold all these objects in memory to serve as the database.  The fact that Silverlight supports Language Integrated Query (LINQ) means that you can use SQL-like statements to work with the multiple, ObservableCollections of objects. 

If you’re wondering why I have you cache everything in memory in a Singleton, there’s a few reasons.  For starters, it makes it easy to query everything with LINQ with the fastest performance possible for single and multi-table JOINs.  Secondly, I don’t represent a Microsoft product group and therefore wouldn’t engineer an unsupported provider that can query subsets of serialized data from files residing in Isolated Storage.  Finally, I don’t want you to accidentally find yourself with multiple instances of the same ObservableCollection when pulling data down from Azure or loading it from Isolated Storage.  Forcing everything into a Singleton prevents you wasting memory or updating objects in the wrong instance of an ObservableCollection.  An inconsistent database is not a good thing.  Don’t worry, you can control which tables are loaded into memory.

So what is this article all about and what are the “improvements” I’m talking about?

This time around, I’m going to focus on saving, loading and deleting the serialized ObservableCollections from Isolated Storage.  In that last article, I showed you how to serialize/de-serialize the ObservableCollections to and from Isolated Storage using the XmlSerializer.  This made it easy for you to save each table to its own XML file which sounds pretty cool.

So what’s wrong with this?

Saving anything as XML means that you’re using the largest, most verbose form of serialization.  After hearing me preach about the virtues of doing SOA with WCF REST + JSON, using the XmlSerializer probably seems out of place.  Luckily, the DataContractJsonSerializer supported by Silverlight on Windows Phone 7 gives you the most efficient wire protocol for data-in-transit can also be used to save those same .NET objects to Isolated Storage.  So the first improvement in this article comes from shrinking the size of the tables and improving the efficiency of the serialization/de-serializing operations to Isolated Storage using out-of-the-box functionality. 

While going from XML to JSON for your serializing might be good enough, there’s another improvement in the way you write the code that will make this much easier to implement for your own projects.  A look back to the previous article reveals a tight coupling between the tables that needed to be saved/loaded and the code needed to make that happen.  This meant that you would have to create a SaveTable and LoadTable method for each table that you wanted to retrieve from Azure.  The new code you’re about to see is generic and allows you to use a single SaveTable and LoadTable method even if you decide to download 100 tables.

Enough talk already, let’s see some code.  Launch your ContosoCloud solution in Visual Studio and open Database.cs.  I want you to overwrite the existing code with the code shown below:

using System;
using System.Net;
using System.Windows;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.IO.IsolatedStorage;
using System.Runtime.Serialization.Json;

namespace ContosoPhone
{
    sealed class Database
    {
        //Declare Instance
        private static readonly Database instance = new Database();

        //Private Constructor
        private Database() { }

        //The entry point into this Database
        public static Database Instance
        {
            get
            {
                return instance;
            }
        }

        //Serialize ObservableCollection to JSON in Isolated Storage
        public void SaveTable<T>(T tableToSave, string tableName)
        {
            if (tableToSave != null)
            {
                using (IsolatedStorageFile store = IsolatedStorageFile.GetUserStoreForApplication())
                {
                    using (IsolatedStorageFileStream stream = store.CreateFile(tableName + ".txt"))
                    {
                        DataContractJsonSerializer serializer = new DataContractJsonSerializer(typeof(T));
                        serializer.WriteObject(stream, tableToSave);
                    }
                }
            }
            else
            {
                throw new Exception("Table is empty");
            }
        }

        //Deserialize ObservableCollection from JSON in Isolated Storage
        public T LoadTable<T>(T tableToLoad, string tableName)
        {
            using (IsolatedStorageFile store = IsolatedStorageFile.GetUserStoreForApplication())
            {
                if (store.FileExists(tableName + ".txt"))
                {
                    using (IsolatedStorageFileStream stream = store.OpenFile(tableName + ".txt", System.IO.FileMode.Open))
                    {
                        DataContractJsonSerializer serializer = new DataContractJsonSerializer(typeof(T));
                        return (T)serializer.ReadObject(stream);
                    }
                }
                else
                {
                    throw new Exception("Table not found");
                }
            }
        }

        //Delete ObservableCollection from Isolated Storage
        public void DropTable(string tableName)
        {
            using (IsolatedStorageFile store = IsolatedStorageFile.GetUserStoreForApplication())
            {
                if (store.FileExists(tableName + ".txt"))
                {
                    store.DeleteFile(tableName + ".txt");
                }
                else
                {
                    throw new Exception("Table not found");
                }
            }
        }

       
        //Declare Private Table Variables
        private ObservableCollection<Customer> customerTable = null;

        //Customer Table
        public ObservableCollection<Customer> Customers
        {
            get { return customerTable; }
            set { customerTable = value; }
        }
    }
}

Looking from top to bottom, the first change you’ll notice is the new SaveTable method where you pass in the desired ObservableCollection and table name in order to serialize it as JSON using the DataContractJsonSerializer.  The next method down the list is LoadTable where you pass in the same parameters as SaveTable but you get back a de-serialized ObservableCollection.  The last new method in the Database Singleton is DropTable which simply deletes the serialized table from Isolated Storage if you don’t need it anymore.

So how do you call this code?

Bring up MainPage.xaml.cs, and find the click event for Save button.  Delete the existing XmlSerializer code and replace it with the following:

try
{
    Database.Instance.SaveTable<ObservableCollection<Customer>>(Database.Instance.Customers, "Customers");
}
catch (Exception ex)
{
    MessageBox.Show(ex.Message);
}

The code above shows you how to call the SaveTable method in the Singleton with the appropriate syntax to pass in the ObservableCollection type as well as actual ObservableCollection value and name.

Now find the click event for the Load button, delete the existing code and paste in the following:

try
{
    Database.Instance.Customers = Database.Instance.LoadTable<ObservableCollection<Customer>>(Database.Instance.Customers, "Customers");
}
catch (Exception ex)
{
    MessageBox.Show(ex.Message);
}

This code looks pretty much the same as the SaveTable code except that you set Database.Instance.Customers equal to the return value from the method.  For completeness sake, drop another button on MainPage.xaml and call it Drop.  In its click event, paste in the following code:

try
{
    Database.Instance.DropTable("Customers");
}
catch (Exception ex)
{
    MessageBox.Show(ex.Message);
}

For this code, just pass in the name of the table you want to delete from Isolated Storage and it’s gone.

It’s time to hit F5 so you can see how things behave.

When your app comes to life in the emulator, I want you to exercise the system by Getting, Adding, Updating and Deleting Customers.  In between, I want you to tap the Save button, close the app, reload the app and tap the Load button and then View Customers to ensure you’re seeing the list of Customers you expect.  Keep in mind that when you Save, you overwrite the previously saved table.  Likewise, when you Load, you overwrite the current in-memory ObservableCollection.  Additionally, Saving, Loading, and Dropping tables that don’t exist should throw an appropriate error message.

So what’s the big takeaway for these tweaks I’ve made to the in-memory database?

While switching serialization from XML to JSON is a great improvement in size and efficiency, I truly believe that making the SaveTable and LoadTable methods generic and reusable will boost developer productivity.  The new ease with which you can Save and Load 1, 10 or even 1,000 tables makes this more attractive to mobile developers that need to work with local data.

So where do we go from here?

You now have some of the basic elements of a database on Windows Phone 7.  You don’t have ACID support, indexes, stored procedures or triggers but you have a foundation to build on.  So what should be built next? 

To help ensure database consistency, I would add an AutoFlush feature next.  SQL Server Compact flushes its data to disk every 10 seconds and there’s nothing to prevent you from using the SaveTable method to do the same.  A timer set to fire at a user-specified interval that iterates through all the ObservableCollections and saves them will help keep your data safe from battery loss and unforeseen system failures.  The fact that your app can be tombstoned at any moment when a user taps the Back button makes an AutoFlush feature even more important.

Anything else?

At the beginning of this article I mentioned RDA which is a simple form of data synchronization.  It’s simple because it only tracks changes on the client but not the server.  To find out what’s new or changed on the server, RDA requires local tables on the device to be dropped and then re-downloaded from SQL Server.  With the system I’ve built and described throughout this series of articles, we already have this brute force functionality.  So what’s missing is client-side change tracking.  To do this, I would need to add code that fires during INSERTS, UPDATES, and DELETES and then writes the appropriate information to local tracking tables.  To push those changes back to SQL Azure, appropriate code would need to call WCF REST + JSON Services that execute DML code on Windows Azure.

I hope with the improvements I’ve made to the in-memory database in this article, you’ll feel even more empowered to build occasionally-connected Windows Phone 7 solutions for consumers and the enterprise.

Keep coding!

-Rob

So now what?

If you’re like me, you probably want to display the list of Customers in the UI.  You might also want to perform other local operations against this data store.  You could add a new Customer and update or even delete an existing one.

I’m going to apologize in advance for not doing the MVVM thing that everyone seems to be into these days and get right to the point. Drag a button on to MainPage.xaml and call it View Customers.  While you’re at it, drag a listbox below the button and name it listBoxCustomers.  Double-click on this Button and add the following code to the click event:

try
{
    if (Database.Instance.Customers != null)
    {
        listBoxCustomers.DisplayMemberPath = "Name";
        listBoxCustomers.ItemsSource = Database.Instance.Customers;
    }
    else
    {
        MessageBox.Show("The Customer Table is Empty");
    }
}
catch (Exception ex)
{
    MessageBox.Show(ex.Message);
}

In the simple code above, you set the listbox’s ItemsSource equal to the Customer collection in the Database Singleton and set the DisplayMemberPath property equal to the Name property of the Customer objects.

Hit F5 to start debugging this Windows Phone 7 + Azure solution.  As usual, a web page and the emulator will launch.  Tap the Get Customers button to pull the Customer data back from the WCF REST service.  Next, tap on the View Customers button to display the list of Customers from the in-memory database as shown in the picture below:

Now it’s time to add a new Customer so drop a button underneath the listbox and call it Add Customer.  Creating a new Customer object requires setting values for 8 properties.  Instead of having you add 8 textboxes to type in the info, I’ll keep in simple and let you add it in code.  In the click event of the button, paste in the code you see below:

try
{
    if (Database.Instance.Customers != null)
    {
        Customer customer = new Customer();
        customer.CustomerId = 5;
        customer.DistributionCenterId = 1;
        customer.RouteId = 1;
        customer.Name = "ABC Corp";
        customer.StreetAddress = "555 Market Street";
        customer.City = "Seattle";
        customer.StateProvince = "WA";
        customer.PostalCode = "98987";
        Database.Instance.Customers.Add(customer);
    }
}
catch (Exception ex)
{
    MessageBox.Show(ex.Message);
}

After setting all the properties (Columns), you add the Customer object (Row) to the Customers property (Table), in the Singleton Database.  Hit F5, tap the Get Customers button, tap the View Customers button and then tap the Add Customer button.  Through the magic of simple data-binding, you should see the new “ABC Corp” show up in the listbox as shown below:

Now that you’ve added a new Customer, it’s time to update it because the president of the company decided to change the name.  Drag a new button and drop it underneath the Add Customer button.  Call it Update Customer and in it’s click event, paste in the following code:

try
{
    if (Database.Instance.Customers != null)
    {
        foreach (Customer c in Database.Instance.Customers)
        {
            if (c.Equals((Customer)listBoxCustomers.SelectedItem))
            {
                c.Name = "XYZ Inc";
            }
        }
    }
}
catch (Exception ex)
{
    MessageBox.Show(ex.Message);
}

The code above loops through the Customers ObservableCollection until it finds a match for the item that’s been selected in the listbox.  When it finds that match, it updates the Name property to “XYZ Inc” which will automatically update what the user views in the listbox.

Hit F5, tap the Get Customers button, tap the View Customers button and then tap the Add Customer button.  Now tap on “ABC Corp” in the listbox to highlight it.  Clicking the Update Customer button will change it before your eyes.

It turns out that “XYZ Inc” went out of business because the president was an idiot so you need to delete it.  Guess what, you need yet another button beneath the Update Customer button.  Call it Delete Customer and in it’s click event, paste in the following code:

try
{
    if (Database.Instance.Customers != null)
    {
        Database.Instance.Customers.Remove((Customer)listBoxCustomers.SelectedItem);
    }
}
catch (Exception ex)
{
    MessageBox.Show(ex.Message);
}

In the code above, the Customer object that matches the item selected in the listbox is removed from the Customers ObservableCollection.  Pretty simple stuff in this case.

To find out for sure, hit F5, tap the Get Customers button, tap the View Customers button and then tap the Add Customer button.  Now tap on “ABC Corp” in the listbox to highlight it.  Clicking the Update Customer button will change it to “XYZ Inc.”  Highlighting “XYZ Inc” and clicking the Delete Customer button will cause this defunct company to disappear as shown below:

Now suppose you only want to display the Customers from Seattle and not the Eastside.  A little LINQ will do the trick here.  Drag and drop a new button called Seattle next to the Test Uri button and paste the following code in the click event:

if (Database.Instance.Customers != null)
{
    IEnumerable<Customer> customers = from customer in Database.Instance.Customers
                                      where customer.DistributionCenterId == 1
                                      select customer;
    listBoxCustomers.DisplayMemberPath = "Name";
    listBoxCustomers.ItemsSource = customers;
}
else
{
    MessageBox.Show("The Driver Table is Empty");
}

In the code above, I set an IEnumerable<Customer> variable equal to the Customers table where the DistributionCenterId is equal to 1.  Since the DistributionCenter #1 serves the Seattle area, I know the listbox will be filled with just Adventure Works LLC and City Power & Light.  Start debugging and test it for yourself.

The last thing you need to do with local data is store it offline since you can’t always count on the network being there.  Luckily we’ve got Isolated Storage to serialize stuff.  In order to work with Isolated Storage, I need you to add using System.IO.IsolatedStorage; at the top of the class.  Since in this example I’ll demonstrate XML Serialization of the Customers ObservableCollection, I’ll need you to add a reference to System.Xml.Serialization and then add using System.Xml.Serialization; at the top of the class.  With that plumbing in place, let’s write some actual code.

Drag and drop another button called Save next to the Get Customers button and paste the following code in the click event:

if (Database.Instance.Customers != null)
{
    using (IsolatedStorageFile store = IsolatedStorageFile.GetUserStoreForApplication())
    {
        using (IsolatedStorageFileStream stream = store.CreateFile("Customers.xml"))
        {
            XmlSerializer serializer = new XmlSerializer(typeof(ObservableCollection<Customer>));
            serializer.Serialize(stream, Database.Instance.Customers);

        }
    }
}

In the code above, you use the combination of IsolatedStorageFile and IsolatedStorageFileStream to write data to Isolated Storage.  In this case, you’re going to create an XML file to save the Customers ObservableCollection of Customer objects.  Remember back in the 80’s when databases used to save each table as an individual file?  I’m thinking of DBase III+, FoxPro, and Paradox at the moment.  Anyway, this is exactly what happens here using the power of the XmlSerializer.  Feel free to debug and step through the code to ensure that it executes without error.

To complete the picture, you need to be able to retrieve the Customers ObservableCollection from Isolated Storage and work with the data without ever having to call the WCF REST services.  Drag and drop one last button called Load next to the View Customers button and paste the following code in the click event:

using (IsolatedStorageFile store = IsolatedStorageFile.GetUserStoreForApplication())
{
    if (store.FileExists("Customers.xml"))
    {
        using (IsolatedStorageFileStream stream = store.OpenFile("Customers.xml", System.IO.FileMode.Open))
        {
            XmlSerializer serializer = new XmlSerializer(typeof(ObservableCollection<Customer>));
            Database.Instance.Customers = (ObservableCollection<Customer>)serializer.Deserialize(stream);
        }
    }
}

The code above does the reverse of the previous Save code by opening the Customers.xml and re-hydrating those objects back into the Customers table using the XmlSerializer.  This is pretty cool stuff.

To make it all real, hit F5 to start debugging this completed mobile project.  When the app loads, tap the Get Customers button to retrieve the data and then tap View Customers to verify that you can see the data.  Now I want you to tap the Save button to serialize the Customers to an XML file in Isolated Storage.  Lastly, I want you to tap on the Back button to close the application.

Hit F5 again to fire up the application.  This time, I don’t want you to retrieve the data from Azure.  Instead, I want you to tap the Load button to de-serialize the Customer objects from XML.  Now for the moment of truth.  Tap the View Customers button to see if you’re actually working with an offline database.  Hopefully, your app will look like the picture below:

Congratulations!  You’ve made it to the end of this series of articles on building an occasionally-connected Windows Phone 7 application that works with WCF REST + JSON Services in Windows Azure.  Furthermore, you’re now pulling data down from SQL Azure and you’re working with it locally using a simple in-memory database that you built yourself.  I’m hoping that you’ll now feel empowered to go build industrial-strength mobile solutions for your own company or your customers.

Keep coding!

-Rob

Attendees of this session will learn how to save money by steering away from point solutions and moving on to Microsoft’s MEAP stack. Come see what Microsoft’s Next-Gen Mobile Enterprise Application Platform looks like, and learn how it will support the cloud and a broader range of mobile platforms and operating systems, including Windows Phone 7.

-Rob

Launch VS2010 and open the solution you created to build the WCF Service Web Role in Azure last time.  Right-click on the solution and add a Windows Phone Application project.  Change the name to ContosoPhone.

Part of the magic of making all this work is to have both the Azure Development Fabric and the Windows Phone 7 project start when it comes time to debug.  Most developers are accustomed to only having a single startup project so let’s make sure you have everything we need running when you hit F5.  Right-click on the solution and select Properties.  Select Startup Project and then click on the Multiple startup projects radio button.  Set both the AzureRestService and ContosoPhone projects Action value to Start and click OK as shown below:

With the startup configuration complete, the next thing I want you to do is copy Customer.cs from the AzureRestService project to the new ContosoPhone project since you’ll need it to create local Customer objects when you retrieve the data from the WCF call.  In this new Customer class you’ll need to change the Namespace to ContosoPhone and remove using System.Web; at the top of the class.  In order to support the DataContract() and DataMember() attributes, you’ll need to add a reference to System.Runtime.Serialization in order to get it to compile.

Drag a Button on to your MainPage.xaml and call it Test.  Double-click on this Button and add the following WebClient code to the click event:

try
{
    WebClient webClient = new WebClient();
    Uri uri = new Uri("http://127.0.0.1:48632/service1.svc/getdata?number=8");
    webClient.OpenReadCompleted += new OpenReadCompletedEventHandler(OpenReadCompletedTest);
    webClient.OpenReadAsync(uri);
}
catch (Exception ex)
{
    MessageBox.Show(ex.Message);
}

As you can see above, I create a WebClient and a Uri object that points to the local Url presented by the Azure Development Fabric.  The port number my be different on your machine so double-check.  This is the test REST service from the last article used to prove that you’re calls are making it through.  You’ve made this call with your web browser and Windows Phone 7 will call it the same way.  Since all calls are asynchronous and the WebClient above created an event handler, copy the following code into the MainPage class and make sure to add using System.IO; at the top:

void OpenReadCompletedTest(object sender, OpenReadCompletedEventArgs e)
{
    try
    {
        StreamReader reader = new System.IO.StreamReader(e.Result);
        MessageBox.Show(reader.ReadToEnd().ToString());
        reader.Close();
    }
    catch (Exception ex)
    {
        MessageBox.Show(ex.Message);
    }
}

In the event handler above, I use the StreamReader to grab the entire XML return value.  The fact that I’m using WebClient means that callbacks are run on the UI thread thus relieving you of the need to use the dispatcher to update the UI.  This allows you to display the following XML string in the MessageBox:

<string xmlns="http://schemas.microsoft.com/2003/10/Serialization/">You entered: 8</string>

It’s time to try this thing out so hit F5.  If it compiles and everything executes properly, you should see both a web browser pointing to the root of your Azure services as well as your emulator and you can test your services with both.  Click the simple Test Button on your Silverlight MainPage and the MessageBox should pop up after the XML result is returned from Azure displaying the answer:

Click the Back button on the Emulator to close your app and close the web browser window to shut down Azure. 

Now it’s time to do something a little more ambitious like return the JSON-encoded list of Customers from SQL Azure.  Unlike most books and articles on Silverlight and RIA Services that you may have read, I’m not going to return this data and immediately data-bind it to a control.  As a long-time mobile and wireless guy that understands intermittent connectivity and the importance of an offline data store, I’m going to have you put this data in a local database first.  Since SQL Server Compact is nowhere to be found and I don’t want you going off and using a 3rd party or open source embedded database, I’m going to show you how to create a simple one of your own.

Right-click on your ContosoPhone project and select Add | Class.  Name this new class Database. 

You’re going to turn this class into a Singleton to create an in-memory database which will ensure that only instance is available.  For simplicity’s sake, just copy the code below into the new Database class:

using System;
using System.Net;
using System.Windows;
using System.Collections.Generic;
using System.Collections.ObjectModel;

namespace ContosoPhone
{
    sealed class Database
    {
        //Declare Instance
        private static readonly Database instance = new Database();

        //Private Constructor
        private Database() { }

        //The entry point into this Database
        public static Database Instance
        {
            get
            {
                return instance;
            }
        }

        //Declare Private Variables
        private ObservableCollection<Customer> customerTable = null;

        //Customer Table
        public ObservableCollection<Customer> Customers
        {
            get { return customerTable; }
            set { customerTable = value; }
        }
    }
}

As you can see above, I’ve had you create a sealed class with a private, static, readonly constructor as an entry point following a popular .NET method of implementing a Singleton.  The big thing here is the Customers property of type ObservableCollection<Customer>.  Since each Customer object is a Row with properties that are Columns, then it’s easy to think of this ObservableCollection of these Customer objects as a Table.  You now have the beginnings of a simple, in-memory database with which to store all the Customer objects when they arrive from the Cloud.

Start out by dragging another button on to MainPage.xaml and calling it Get Customers.  Double-click on this Button and add the following WebClient code to the click event:

try
{
    WebClient webClient = new WebClient();
    Uri uri = new Uri("http://127.0.0.1:48632/service1.svc/customers");
    webClient.OpenReadCompleted += new OpenReadCompletedEventHandler(OpenReadCompletedCustomers);
    webClient.OpenReadAsync(uri);
}
catch (Exception ex)
{
    MessageBox.Show(ex.Message);
}

You’ll notice that it looks almost identical to the previous WebClient call except this time you’re just adding /customers to the end of service1.svc.  The beauty of the UriTemplate is that Microsoft makes it just that easy to call a REST service from WCF.  Since the above code created an event handler, paste this following code into the MainPage class to handle the callback:

void OpenReadCompletedCustomers(object sender, OpenReadCompletedEventArgs e)
{
    DataContractJsonSerializer ser = null;

    try
    {
        ser = new DataContractJsonSerializer(typeof(ObservableCollection<Customer>));
        Database.Instance.Customers = ser.ReadObject(e.Result) as ObservableCollection<Customer>;
    }
    catch (Exception ex)
    {
        MessageBox.Show(ex.Message);
    }
}

You’ll immediately see a bunch of squiggly lines telling you something is missing.  Add a reference to System.Servicemodel.Web and then add using System.Runtime.Serialization.Json; at the top in order to fix the problems with DataContractJsonSerializer.  Next add using System.Collections.ObjectModel; at the top to fix the ObservableCollection squigglies.  Give your solution a quick Rebuild to ensure that everything compiles.

In analyzing the new code I just had you add, you’ll notice the use of DataContractJsonSerializer which in this case is used to de-serialize JSON-encoded Customer objects that are downloading over the air.  It does this by casting them as ObservableCollection<Customer> which works because I had you copy that class from the AzureRestService project.  In a world of right-clicking to add a Service/Web Reference to consume SOAP web services, you never had to know what was going on under the covers to create an easy to use proxy.  You also never saw Visual Studio create hidden classes that looked just like the ones found on the server side so your client would have something to work with and so that intellisense worked. 

The last important line of code you see is setting the Customers table of your in-memory database equal to the de-serialization of the OpenReadCompletedEventArgs.  I think we’re ready to try out this code.  If all goes well, you’ll have a bunch of downloaded Customer data stored in an in-memory database on Windows Phone 7.

Set a breakpoint at the declaration of the DataContractJsonSerializer, hit F5, and click the Get Customers button!

Success looks like the picture above!  I hovered my mouse over Database.Instance.Customers and revealed that I was now storing Customers with Cities, CustomerId’s, PostalCodes, RouteId’s and StateProvinces for all to see in Visual Studio’s amazing debugger.

You are now consuming wireless-friendly WCF REST + JSON Services from both Windows Server and Windows Azure with data coming from SQL Server/SQL Azure.  Windows Phone 7 is connected to tiny, fast, efficient services designed to thrive in unreliable/slow wireless coverage.  You now know how to serialize/de-serialize .NET objects as JSON and you’ve also created your own in-memory database.

So what’s next?  In my next article, I’ll show you how to query this new in-memory database using LINQ and I’ll show you how to save the data in your local tables to Isolated Storage so you can keep using your apps even when the network has disappeared.

Keep coding and I’ll see everyone at Tech Ed in Berlin!

-Rob

Active Directory

• Since your clients will be passing in their Domain\username + password credentials when they sync, both IIS and SQL Server will make auth requests of the Domain Controller. Ensure that you have at least a primary and backup Domain Controller, that the NTDS.dit disk drives are big enough to handle the creation of a large number of new AD DS objects (mobile users and groups), and that your servers have enough RAM to cache all those objects in memory.

Database Schema

• Ensure your schema is sufficiently de-normalized so that you never have to perform more than a 4-way JOIN across tables. This affects server-side JOIN filters as well as SSCE performance.
• To ensure uniqueness across all nodes participating in the sync infrastructure, use GUIDs for your primary keys so that SQL Server doesn’t have to deal with the overhead of managing Identity ranges. Make sure to mark your GUIDs as ROWGUIDCOL for that table so that Merge won’t try to add an additional Uniqueidentifier column to the table.  Don’t create clustered indexes when using GUIDs as primary keys because they will suffer horrible fragmentation that will rapidly degrade performance.  Just use a normal index.
• Create clustered indexes for your primary keys when using Indentity columns, Datetime, or other natural keys.  Ensure that every column in every table that participates in a WHERE clause is indexed.

Distributor

• If your network connection is fast and reliable like Wi-Fi or Ethernet, your SSCE client has more than 32 MB of free RAM, and SQL Server isn’t experiencing any deadlocks due to contention with ETL operations or too many concurrent Merge Agents, create a new Merge Agent Profile based on the High Volume Server-to-Server Profile so that SQL Server will perform more work per round-trip and speed up your synchronizations.
• If you’re using a 2G/3G Wireless Wide Area Network connection, create a Merge Agent Profile based on the Default Profile so that SQL Server will perform less work and use fewer threads per round-trip during synchronization than the High Volume Server to Server Profile which will help to reduce server locking contention and perform less work per round trip which will make your synchronizations more likely to succeed.
• In order to prevent SQL Server from performing Metadata Cleanup every time a Subscriber synchronizes, set the –MetadataRetentionCleanup parameter to 0.
• As SQL Server has to scale up to handle a higher number of concurrent users in the future, locking contention will increase due to more Merge Agents trying to perform work at the same time.  When this happens, adjust the parameters of the Default Profile so that both  –SrcThreads and –DestThreads are equal to 1.

Publication

• When defining the Articles you’re going to sync, only check the minimum tables and columns needed by the Subscriber to successfully perform its work.
• For Lookup/Reference tables that aren’t modified by the Subscriber, mark those as Download-only to prevent change-tracking metadata from being sent to the Subscriber.
• Despite the fact the column-level tracking sends less data over the air, stick with row-level tracking so SQL Server won’t have to do as much work to track the changes.
• Use the default conflict resolver where the “Server wins” unless you absolutely need a different manner of picking a winner during a conflict.
• Use Static Filters to reduce the amount of server data going out to all Subscribers.
• Make limited use of Parameterized Filters which are designed to reduce and further specify the subset of data going out to a particular Subscriber based on a HOST_NAME() which creates data partitions.  This powerful feature slows performance and reduces scalability with each additional filter, so it must be used sparingly.
• Keep filter queries simple and don’t use IN clauses, sub-selects or any kind of circular logic.
• Strive to always create “well-partitioned” Articles where all changes that are uploaded/downloaded are mapped to only the single partition ID for best performance and scalability.
  • When using Parameterized Filters, always create non-overlapping data partitions where each row from a filtered table only goes to a single Subscriber instead of more than one which will avoid the use of certain Merge metadata tables.
  • Each Article in this scenario can only be pubished to a single Publication
  • A Subscriber cannot insert rows that do not belong to its partition ID.
  • A Subscriber cannot update columns that are involved in filtering.
  • In a join filter hierarchy, a regular article cannot be the parent of a “well-partitioned” article.
  • The join filter in which a well-partitioned article is the child must have the join_unique_key set to a value of 1 which relates to the Unique key check box of the Add Join dialog.  This means there’s a one-to-one or one-to-many relationship with the foreign key.
  • Each “well-partitioned” Article can have only one subset or join filter. The article can have a subset filter and be the parent of a join filter, but cannot have a subset filter and be the child of a join filter.
• Never extend a filter out to more than 4 joined tables.
• Do not filter tables that are primarily lookup/reference tables, small tables, and tables with data that does not change.
• Schedule the Snapshot Agent to run once per day to create an unfiltered schema Snapshot.
• Set your Subscriptions to expire as soon as possible to keep the amount change-tracking metadata SQL Server has to manage to an absolute minimum. Normally, set the value to 2 to accommodate 3-day weekends since 24 hours are automatically added to the time to account for multiple time zones. If server-side change tracking isn’t needed and Subscribers are pulling down a new database every day and aren’t uploading data, then set the expiration value to 1.
• Set Allow parameterized filters equal to True.
• Set Validate Subscribers equal to HOST_NAME().
• Set Precompute partitions equal to True to allow SQL Server to optimize synchronization by computing in advance which data rows belong in which partitions.
• Set Optimize synchronization equal to False if Precompute partitions is equal to True.  Otherwise set it to True to optimize filtered Subscriptions by storing more metadata at the Publisher.
• Set Limit concurrent processes equal to True.
• Set Maximum concurrent processes equal to the number of SQL Server processor cores.  If exceesive locking contention occurs, reduce the number of concurrent processes until the problem is fixed.
• Set Replicate schema changes equal to True.
• Check Automatically define a partition and generate a snapshot if needed when a new Subscriber tries to synchronize. This will reduce Initialization times since SQL Server creates and applies snapshots using the fast BCP utility instead of a series of slower SELECT and INSERT statements.
• Add data partitions based on unique HOST_NAMEs and schedule the Snapshot Agent to create those filtered Snapshots nightly or on the weekend so they’ll be built using the fast BCP utility and waiting for new Subscribers to download in the morning.
• Ensure that SQL Server has 1 processor core and 2 GB of RAM for every 100 concurrent Subscribers utilizing bi-directional sync. Add 1 core and 2 GB of RAM server for every additional 100 concurrent Subscribers you want to add to the system.  Never add more Subscribers and/or IIS servers without also adding new cores and RAM to the Publisher.
• Turn off Hyperthreading in the BIOS of the SQL Server as it has been known to degrade SQL Server performance.
• Do not add your own user-defined triggers to tables on a Published database since Merge places 3 triggers on each table already.
• Add one or more Filegroups to your database to contain multiple, secondary database files spread out across many physical disks.
• Limit use of large object types such as text, ntext, image, varchar(max), nvarchar(max) or varbinary(max) as they require a significant memory allocation and will negatively impact performance.
• Set SQL Servers’s minimum and maximum memory usage to within 2 GB of total system memory so it doesn’t have to allocate more memory on-demand.
• Always use SQL Server 2008 R2 and Windows Server 2008 R2 since they work better together because they take advantage of the next generation networking stack which dramatically increases network throughput. They can also scale up as high as 256 cores.
• Due to how Merge Replication tracks changes with triggers, Merge Agents, and tracking tables, it will create locking contention withDML/ ETL operations.  This contention degrades server performance which negatively impacts sync times with devices.  This contention should be mitgated by performing large INSERT/UPDATE/DELETE DML/ETL operations during a nightly maintenance window when Subscribers aren’t synchronizing.
• Since Published databases result in slower DML/ETL operations, perform changes in bulk by using XML Stored Procedures to boost performance.
• To improve the performance of pre-computed partitions when DML/ETL operations result in lots of data changes, ensure that changes to a Parent table in a join filter are made before corresponding changes in the child tables.  This means that when DML/ETL operations are pushing new data into SQL Server, they must add master data to the parent filter table first, and then add detail data to all the related child tables second, in order for that data to be pre-computed and optimized for sync.
• Create filter partitions based on things that don’t change every day.  Partitions that are added and deleted from SQL Server and Subscribers that move from one partition to another is very disruptive to the performance of Merge Replication.
• Always perform initializations and re-initializations over Wi-Fi or Ethernet when the device is docked because this is the slowest operation where the entire database must be downloaded instead of just deltas.  To determine rough estimates for initialization, multiply the size of the resulting SSCE .sdf file x the bandwidth speed available to the device.  A file copy over the expected network will also yield estimates for mininum sync times.  These times don’t include the work SQL Server and IIS must perform to provide the data or data INSERT times on SSCE.
• If your SQL Server Publisher hits a saturation point with too many concurrent mobile Subscribers, you can scale it out creating a Server/Push Republishing hierarchy. Put the primary SQL Server Publisher at the top of the pyramid and have two or more SQL Servers subscribe to it. These can be unfiltered Subscriptions where all SQL Servers get the same data or the Subscribers can filter their data feeds by region for example. Then have the Subscribing SQL Servers Publish their Subscription for consumption by mobile SSCE clients.
• Create just a single Publication.

Internet Information Services

• Use the x64 version of the SQL Server Compact 3.5 SP2 Server Tools with Windows Server 2008 R2 running inside IIS 7.5.
• Use a single Server Agent in a single Virtual Directory.
• Ensure the IIS Virtual Directory where the Server Agent resides is on a fast solid-state drive that’s separate from the disk where Windows Server is installed to better support file I/O.
• Use a low-end server with 2 processor cores and 2 GB of RAM to support 400 concurrent Subscribers queued at the same time.
• Set the MAX_THREADS_PER_POOL Server Agent registry key equal to 2 to match the IIS processor cores and RAM. Do not set this value to a higher number than the number of cores.
• Set the MAX_PENDING_REQUEST Server Agent registry key equal to 400 which means the Server Agent will queue up to 400 concurrent Subscribers waiting for one of the 2 worker threads to become available to sync with SQL Server.
• Set the IIS Connection Limits property to 400 to prevent an unlimited number of connections reaching the Server Agent.
• Add a new load-balanced IIS server for every additional 400 concurrent Subscribers you want to add to the system.

Subscriber

• Use the appropriate x64, x86 or ARM version of SQL Server Compact 3.5 SP2 to take advantage of the PostSyncCleanup property of the SqlCeReplication object that can reduce the time it takes to perform an initial synchronization. Set the PostSyncCleanup property equal to 3 where neither UpdateStats nor CleanByRetention are performed.
• Increase the Max Buffer Size connection string parameter to 1024 on a phone and 4096 on a PC to boost both replication and SQL query processing performance. If you have more RAM available, set those values even higher until you reach the law of diminishing returns.
• Keep your SSCE database compact and fast by setting the Autoshrink Threshold connection string parameter to 10 so it starts reclaiming empty data pages once the database has become 10% fragmented.
• Replication performance testing must be performed using actual PDAs to observe how available RAM, storage space and CPU speed affect moving data into the device’s memory area and how quickly this data is inserted into the SSCE database tables.  Since the SSCE database doubles in size during replication, the device must have enough storage available or the operation will fail.  Having plenty of available RAM is important so that SSCE can utilize its memory buffer to complete a Merge Replication operation more quickly.  With plenty of available RAM and storage, a fast CPU will make all operations faster.
• The PDA must have at least an extra 32 MB of available free RAM that can be used by the .NET Compact Framework (NETCF) application.  If additional applications are running on the device at the same time, even more RAM is needed.  If a NETCF application has insufficient RAM is will discard its compiled code and run in interpreted mode which will slow the application down.  If the NETCF app is still under memory pressure after discarding compiled code, Windows Mobile will first tell the application to return free memory to the operating system and then will terminate the app if needed.
• Set the CompressionLevel property of the SqlCeReplication object to 0 for fast connections and increment it from 1 to 6 on slower connections like GPRS to increase speed and reduce bandwidth consumption.
• Tune the ConnectionRetryTimeout, ConnectTimeout, ReceiveTimeout and SendTimeout properties of the SqlCeReplication object based on expected bandwidth speeds:

• You can decrease potentially slow SSCE file I/O by adjusting the Flush Interval connection string parameter to write committed transactions to disk less often than the default of every 10 seconds.  Test longer intervals between flushes like 20 or 30 seconds. Keep in mind that these transactions can be lost if the disk or system fails before flushing occurs so be careful.
• When replicating data that has been captured in the field by the device, perform Upload-only syncs to shorten the duration.

Storage

• Use a Fibre Channel SAN with 15k RPM or solid-state disks for best I/O performance.
• Databases should reside on a RAID 10, unshared LUN comprised of at least 6 disks.
• Database logs should reside on a RAID 10, unshared LUN comprised of at least 6 disks.
• Tempdb should reside on a RAID 10, unshared LUN comprised of at least 6 disks.
• The Tempdb log should reside on a RAID 10, unshared LUN comprised of at least 6 disks.
• The Snapshot share should reside on a RAID 10, unshared LUN comprised of at least 6 disks.  This disk array should be large enough to accommodate a growing number of filtered Snapshots. Snapshot folders for Subscribers that no longer use the system must be manually deleted.
• Merge Replication metadata tables should reside on a RAID 10, unshared LUN comprised of at least 6 disks.
• Increase your Host Bus Adapter (HBA) queue depths to 64.
• Your Publication database should be broken up into half the number of files as the SQL Server has processor cores. Each file must be the same size.
• Tempdb should be pre-sized with an auto-growth increment of 10%. It should be broken up into the same number of files as the SQL Server has processor cores. Each file must be the same size.

High Availability

• Load-balance the IIS servers to scale them out. Enable Server Affinity (stickiness) since the Replication Session Control Blocks that transmit data between the Server Agent and SSCE are stateful. Test to ensure that your load-balancer is actually sending equal amounts of Subscriber sync traffic to each IIS server.  Some load-balancers can erroneously send all traffic to a single IIS server if not properly configured.
• Implement Windows Clustering so that SQL Server can failover to a second node.
• Using SQL Server Mirroring so that your Published database will failover to a standby server.
• Make a second SQL Server into an unfiltered Subscriber to your Publisher so that it can take over Merge Replication duties for mobile clients as a Republisher if the primary SQL Server fails. SSCE clients would just have to reinitialize their Subscriptions to begin synchronizing with the new Republisher.

Ongoing Maintenance

• Use the Replication Monitor to have a real-time view of the synchronization performance of all your Subscribers.
• Use the web-based SQL Server Compact Server Agent Statistics and Diagnostics tools to monitor the health and activity of the Server Agent running on IIS.
• Create a SQL Job to execute the sp_MSmakegeneration stored procedure after large DML operations. Regular execution after INSERTING/UPDATING/DELETING data from either DML/ETL operations or after receiving lots of changes from Subscribers will maintain subsequent sync performance. Executing this stored procedure from the server-side is preferable to having it executed as a result of a Subscriber sync which would block all other Subscribers.
• During your nightly maintenance window, rebuild the indexes and update the statistics of the following Merge Replication metadata tables:
  • MSmerge_contents
  • MSmerge_tombstone
  • MSmerge_genhistory
  • MSmerge_current_partition_mappings
  • MSmerge_past_partition_mappings
  • MSmerge_generation_partition_mappings
• If you notice performance degradation during the day due to a large number of Subscribers making large changes to the database, you can updates the statistics (with fullscan) of the Merge Replication metadata tables more frequently throughout the day to force stored proc recompiles to get a better query plan.
  • UPDATE STATISTICS MSmerge_generation_partition_mappings WITH FULLSCAN
  • UPDATE STATISTICS MSmerge_genhistory WITH FULLSCAN
• Rebuild/defrag indexes on your database tables and Merge Replication metadata tables throughout the day to reduce locking contention and maintain performance.
• Use the Missing Indexes feature of SQL Server to tell you which indexes you could add that would give your system a performance boost. Do not add recommended indexes to Merge system tables.
• Use the Database Engine Tuning Advisor to give you comprehensive performance tuning recommendations that cover every aspect of SQL Server.
• Monitor the performance of the following counters:
  • Processor Object: % Processor Time: This counter represents the percentage of processor utilization. A value over 80% is a CPU bottleneck.
  • System Object: Processor Queue Length: This counter represents the number of threads that are delayed in the processor Ready Queue and waiting to be scheduled for execution. A value over 2 is bottleneck and shows that there is more work available than the processor can handle. Remember to divide the value by the number of processor cores on your server.
  • Memory Object: Available Mbytes: This counter represents the amount of physical memory available for allocation to a process or for system use. Values below 10% of total system RAM indicate that you need to add additional RAM to your server.
  • PhysicalDisk Object: % Disk Time: This counter represents the percentage of time that the selected disk is busy responding to read or write requests. A value greater than 50% is an I/O bottleneck.
  • PhysicalDisk Object: Average Disk Queue Length: This counter represents the average number of read/write requests that are queued on a given physical disk. If your disk queue length is greater than 2, you’ve got an I/O bottleneck with too many read/write operations waiting to be performed.
  • PhysicalDisk Object: Average Disk Seconds/Read and Disk Seconds/Write: These counters represent the average time in seconds of a read or write of data to and from a disk. A value of less than 10 ms is what you’re shooting for in terms of best performance. You can get by with subpar values between 10 – 20 ms but anything above that is considered slow. Times above 50 ms represent a very serious I/O bottleneck.
  • PhysicalDisk Object: Average Disk Reads/Second and Disk Writes/Second: These counters represent the rate of read and write operations against a given disk. You need to ensure that these values stay below 85% of a disk’s capacity by adding disks or reducing the load from SQL Server. Disk access times will increase exponentially when you get beyond 85% capacity.
• A limited number of database schema changes can be made and synchronized down to SSCE Subscribers without any code changes which makes it easier to update your system as it evolves over time.
• Use a Merge Replication Test Harness to stress test the entire system.  The ability to simulate hundreds or thousands of concurrent synchronizing Subscribers allows you to monitor performance and the load on the system.  This is helpful in properly configuring and tuning SQL Server, IIS, and the Windows Mobile devices.  It will tell you where you’re having problems and it will let you predict how much server hardware you will need to support growing numbers of Subscribers over time.  It’s also very important to simulate worst-case scenarios that you never expect to happen.

I hope this information sufficiently empowers you to take on the largest MEAP solutions that involve SQL Server Merge Replication and SQL Server Compact.  If you need a deeper dive, go check out my book on Enterprise Data Synchronization http://www.amazon.com/Enterprise-Synchronization-Microsoft-Compact-Replication/dp/0979891213/ref=sr_1_1?ie=UTF8&s=books&qid=1271964573&sr=1-1 over at Amazon.  Now go build a fast and scalable solution for your company or your customers.

Best Regards,

Rob

P.S.  If your solution doesn’t require all the advanced features found in Merge Replication, I highly recommend you use Remote Data Access (RDA).  This is a much simpler sync technology that’s extremely fast, scalable, and easier to manage.