A separate 1099 for Hood Canal Press tells me that my SQL Server Merge Replication books are still killing it!  Our world of disconnected devices require efficient data sync now more than ever.  Despite a variety of sync technologies out there, Merge is still the best!

The new Sync Framework Toolkit found on MSDN builds on the existing Sync Framework 2.1′s ability to create disconnected applications, making it easier to expose data for synchronization to apps running on any client platform.  Where Sync Framework 2.1 required clients to be based on Windows, this free toolkit allows other Microsoft platforms to be used for offline clients such as Silverlight, Windows Phone 7, Windows Mobile, Windows Embedded Handheld, and new Windows Slates.   Additionally, non-Microsoft platforms such as iPhones, iPads, Android phones and tablets, Blackberries and browsers supporting HTML5 are all first-class sync citizens.  The secret is that we no longer require the installation of the Sync Framework runtime on client devices.  When coupled with use of an open protocol like OData for data transport, no platform or programming language is prevented from synchronizing data with our on-premise and cloud databases.  When the data arrives on your device, you can serialize it as JSON, or insert it into SQL Server Compact or SQLite depending on your platform preferences.

The Sync Framework Toolkit provides all the features enabled by theSync Framework 4.0 October 2010 CTP.  We are releasing the toolkit as source code samples on MSDN with the source code utilizing Sync Framework 2.1.  Source code provides the flexibility to customize or extend the capabilities we have provided to suit your specific requirements. The client-side source code in the package is released under the Apache 2.0 license and the server-side source code under the MS-LPL license.  The Sync Framework 2.1 is fully supported by Microsoft and the mobile-enabling source code is yours to use, build upon, and support for the apps you create.

Now some of you might be wondering why you would use a sync technology to move data rather than SOAP or REST web services.  The reason has to do with performance and bandwidth efficiency.  Using SOA, one would retrieve all the data needed to the device in order to see what has changed in SQL Server.  The same goes for uploading data.  Using the Sync Framework Toolkit, only the changes, or deltas, are transmitted over the air.  The boosts performance and reduces bandwidth usage which saves time and money in a world of congested mobile data networks with capped mobile data plans.  You also get a feature called batching, which breaks up the data sent over wireless networks into manageable pieces.  This not only prevents you from blowing out your limited bandwidth, but it also keeps you from using too much RAM memory both on the server and your memory-constrained mobile device.  When combined with conflict resolution and advanced filtering, I’m sold!

I think you’ll find the Sync Framework Toolkit to be an immensely valuable component of your MEAP solutions for the enterprise as well as the ones you build for consumers.

Keep Synching,

Rob

Since Windows Embedded Handheld uses an advanced version of the operating system kernel used by Windows Mobile 6.5.3, many of the techniques and best practices I’ve taugh customers and developers all over the world still apply.  While it still uses the slotted memory model found in Windows CE 5 with 32 processes and 32 MB of memory per process, you’ll find that numerous enhancements and tuning has taken place to give your line of business apps more of what they need.  I’m talking about more memory per process and improved performance.  Therefore, I’d like you to sit back and watch the video of a presentation I delivered at Tech Ed in Los Angeles a couple of years ago so you can better learn what this mobile platform has to offer in the form of better memory management and improved performance:

A recent Gartner report recommends that organizations should stay with Windows Embedded Handheld as the best mobile platform for enterprise line of business needs.  Great devices are available from OEMs like Intermec, Motorola, Psion, and Honeywell just to name a few.  I hope this video helps you with any memory management or performance issues you may need to deal with in your enterprise mobile apps.

Best Regards,
Rob

Enjoy,
Rob

1. Multiple mobile applications
2. Multiple mobile operating systems
3. Multiple backend systems maximizing ROI vs. tactical solutions

It’s already a $1 Billion business and 95% of orgs will choose MEAP over point solutions by 2012.   The picture below represents some of our familiar cloud and on-premise servers on top and a wide spectrum of mobile devices from Microsoft and other manufacturers on the bottom: 

Let’s do a quick rundown of Gartner’s Critical Capability list so you can see how we rise to their challenge:

1. Integrated Development Environment for composing server and client-side logic: Microsoft Visual Studio supports on-premise and cloud server development and targets clients such as Windows, Windows Phone 7, Windows Mobile, the Web, Nokia S60, and the Macintosh. 
2. Application Client Runtime: Various flavors of Microsoft .NET (Silverlight, .NET, Compact Framework) run on Azure, Windows Server, Windows, the Mac, Windows Phone 7, Windows Mobile, and Nokia S60.  Guess what, you can use MonoTouch to take your .NET skills to the iPhone, iPad and iPod Touch.  MonoDroid is in the preview stage and will bring .NET to Android phones and tablets in the future.
3. Enterprise Application Integration Tools: Connecting mobile devices to a variety of backend packages like Dynamics or SAP is critical.  Microsoft supports this integration in the cloud via Windows Azure AppFabric and on-premise though SQL Server Integration Services and dozens of adapters.  Tools like our Business Intelligence Dev Studio make EAI a repeatable, drag and drop exercise.
4. Packaged Mobile Apps: Microsoft delivers the Office suite across Windows, Windows Phone 7, Windows Mobile, the Web and the Mac.  Office will be coming to Nokia in the future and One Note just arrived on iOS.
5. Multichannel Servers: Windows Server + SQL Server on-premise and Windows Azure + SQL Azure in the cloud represents Microsoft’s mobile middleware platforms.  Windows Communication Foundation (WCF) delivers cross-platform SOAP & REST Web Services and cross-platform wire protocols like XML, JSON and OData.
6. Software Distribution: Microsoft System Center Configuration Manager supports pushing software out to Windows and Windows Mobile.  Windows Phone 7 has Marketplace for this function.
7. Security: Data-in-transit is secured by SSL across all platforms.  Data-at-Rest security for apps is facilitated on Windows by BitLocker, Windows Mobile through encryption policies and Windows Phone 7 through AESManaged in Silverlight.  Cross-platform auth is facilitated by Microsoft Windows Identity Foundation so devices can access resources via a Windows Live ID, Facebook, Google, Yahoo, ADFS and others.
8. Hosting: It goes without saying the Microsoft knocks the hosting requirement out of the park with Azure.

So what do I want you to take away from all this?

Microsoft has a great MEAP stack comprised of servers and skillsets you probably already have at your company.  You get maximum reuse on our servers and in our cloud which means you save money when it’s time to build and deploy your second, third and fourth mobile app without new training, new servers, and different technologies each time.  I hope you’re pleasantly surprised to see that our .NET application runtime lives on so many mobile platforms.  Again, this means that your existing .NET skills can be reused on Microsoft devices, the Web, Mac, Nokia and even the iPad.  Who knew?  I’m looking forward to bring Android into the .NET camp as well. 

It’s a brave new world of disparate devices connected to the cloud.  Companies have no choice but to target most all of them when constructing B2C apps to sell products or bring in new customers.  They’ve also found that this is the case in supporting their own employees and business partners with B2E and B2B apps.  No single company has so many different skillsets and competencies to pull this off.

There is one thing that most companies do have though.  A Microsoft infrastructure in their data center or the cloud, Windows on desktops, laptops and tablets, plus teams of .NET developers.  As I’ve just shown you, these .NET developers armed with Visual Studio or MonoTouch can be unleashed to allow you to reach almost every mobile platform.  This dramatically reduces the amount of extra Java and Eclipse skills that you’ll consider bringing in-house or outsourcing in order to target platforms like Android or the Blackberry.  Through the magic of WCF, all these platforms can connect to your critical Microsoft back-end resources and beyond.  You save money on training, use the servers you already have, resuse business logic and get to market faster.  No matter what platform you need to target, Microsoft and its partners want to help you reach your goals.

Looks like you’re already ahead of the game in taking on the Consumerization of IT.

-Rob

Sync technologies like Merge Replication and the Sync Framework track changes on SQL Server using triggers, stored procedures and special tracking tables.  The act of tracking changes made by each SQL Server Compact or Silverlight sync subscriber can cause a lot of locking and blocking on the server.  This diminishes performance and sometimes leads to deadlocks. 

Therefore, don’t listen to your SAN administrator when he says the RAID 5 will do.  RAID 1 or 10 must always be used for all databases, tempdb, and transaction logs.  Furthermore, each of these database objects must be placed on their own dedicated RAID arrays.  No sharing!  Remembers, as a DBA and sync expert, knowledge of SAN configuration must always be part of your skillset.

Keeping synching,

Rob

You have to hand it to Microsoft.   They have certainly fulfilled the old saying, “If at first you don’t succeed, try, try again.”  Microsoft has had a number of previous attempts to build a successful operating system for the mobile market with WinPad, Windows Mobile and Win CE.  These efforts – simply because they were Microsoft – generated some market presence but nowhere near the market share achieved by major players such as RIM (BlackBerry), Apple (iPhone) and Google (Android).

I thought it was poignant when Rob Tiffany, Mobility Architect at Microsoft, told me at CTIA that Microsoft went back to the drawing board to develop a new mobile OS from the ground up.  Steve Ballmer, Microsoft CEO, introduced Windows Phone 7 on Monday, Oct. 11 at a press conference in New York.

The reviews on Windows Phone 7 (WP7) have generally been positive.  I appeared on Brian Sullivan’s show on FoxBusiness to explain why I thought Microsoft would succeed with WP7, especially in the enterprise space.

There are a number of user interface and technical innovations that WP7 brings to the mobile market, including:

• New platform – WP7 is not an enhancement to previous Windows mobile efforts. It’s developed from the ‘ground up’ – no more forcing people to go through the Windows Start menu. It was designed to provide users with easy access to the information they want and need.

• Active tiles – users can decide what’s important to them and allocate tiles to give them the information they need, e.g. a tile for messaging, a tile for social, a tile for news, etc. Take a look at the sample home screen on a sample WP7 phone.  It shows a number of Active Tiles that are user defined to make the initial images on the phone’s start up screen comfortable and personal to the user.

• Panoramas – with panoramas, you swipe left and right to get more information.  This is a new user paradigm much like flip/scroll has become in the iPhone and Android for looking through lists by swiping up and down. This allows you to swipe left and right – a very cool concept. Take a look at the wide panoramas below.  Notice that the phone image at the top can sweep to the right to cover all the information about a topic and the sweep back to the left.  This allows applications to present a lot of information that appears the way the eye looks at the world – in a panoramic fashion.  Vertical scrolling is good for lists where panoramas are good for showing more of one kind of information such as a photo or image or set of items in a group.

• Apps – Microsoft has created solid development tools to make it easy for (consumer and enterprise) developers to build exciting apps, e.g. extending X-Box for gaming, etc. and then publishing them in the Windows Phone Marketplace.

Phones will be produced using WP7 by Samsung, HTC, LG and Dell.  I suspect that Motorola may follow along as well in 2011.  Windows Phones will be distributed through AT&T Mobility and T-Mobile in the US at first and then via Verizon and Sprint  in 2011.  Some Windows Phones will have integrated keyboards and others will be touch screen only.  For example, the Samsung Omnia 7 incorporates a Super AMOLED screen, a 4-inch display, 5-mp camera with HD video and support of Xbox Live gaming and media content.

Microsoft has implemented multiple processes in the first rendition of Windows Phone which allows each app to switch back and forth.  Some developers may need full-scale multi-tasking for background operation which Microsoft will likely support at some future time.  They store the last place the user was in an application and then re-store it back when the app is re-launched to give the feel of being multi-tasking.  But, Microsoft wanted to make sure the first version was solid and, therefore, they deferred true multi-tasking to a later version.

Microsoft has made WP7 work well for both consumers and enterprise.  Consumers get a good user experience right out of the box that they can then personalize with Live Tiles.  Consumers will also get a streaming music service based on Microsoft’s Zune efforts.

I believe that WP7 will be received well in the enterprise for a number of reasons, including:

• Microsoft Office. Right out of the box, WP7 will support opening and editing Word, Excel and PowerPoint files in a mobile edition of MS Office.

• Outlook. Because Outlook is included as well, enterprise users who already are using Exchange/Outlook will get a friendly, familiar user interface for email. 

• OneNote.  This is a note taking application that has seen very little adoption in the desktop but may find a much larger following in WP7 especially when joined with sharing of notes from a meeting with co-workers.

• Security. Microsoft has invested a great deal of effort ‘under the covers’ to incorporate end to end security to make sure that enterprise IT professionals will be comfortable deploying WP7.

• Enterprise Development.  Microsoft has provided the same development tools that many enterprises have used to create mobile applications.

Personally, I would have preferred if Microsoft had made a further separation from Windows by calling the new platform Microsoft Phone (with different version numbers) so that they could then have Windows 7 (for desktop and laptops) and then Phone 7 without the reference to Windows (for phones).

As for the tablet arena, most firms are leveraging the personal user interfaces and environments from the mobile world for tablets.  Apple has done this by using iOS from the iPhone with enhancements in the iPad (rather than using the Mac desktop OS).  A number of tablets (including the Samsung Galaxy TAB) are using Google’s Android mobile OS.  Thus, it seems likely to me that Microsoft will eventually develop a version of Windows Phone that they might dub Windows Tablet to support larger screens, gestures and the application Windows Phone Marketplace in the tablet arena.

I think RIM should be worried with the introduction of Windows Phone. The BlackBerry user interface has not changed much in the past 10 years.  BlackBerry devices are rock solid and work well but don’t provide the ‘sex appeal’ provided in Apple’s iOS or Google’s Android.  Also, Microsoft has great relationships with enterprise IT. They make it easy for enterprises to roll out Windows Phone instead of just BlackBerry phones. It will be interesting to see how RIM responds to Windows Phone over time.

Overall, Microsoft is back in the game with Windows Phone 7.  I look forward to spending some time with a Windows Phone and getting some hands-on experience.  In the end, it’s the users and enterprises that vote with their pocketbook, but it seems highly likely that Microsoft will earn significant market share over the next few years as they evolve Windows Phone.  Kudos to the Microsoft team to give the mobile world another good user experience.

We’ll look back on the mobile market 20-30 years from now and see how important it was to provide a number of different user interfaces and then to see how customers declare what they like the most.  

Written By:

J. Gerry Purdy, Ph.D.
Principal Analyst
Mobile & Wireless
MobileTrax LLC
gerry.purdy@mobiletrax.com
404-406-5309

Active Directory

• Since your clients will be passing in their Domain\username + password credentials when they sync, both IIS and SQL Server will make auth requests of the Domain Controller. Ensure that you have at least a primary and backup Domain Controller, that the NTDS.dit disk drives are big enough to handle the creation of a large number of new AD DS objects (mobile users and groups), and that your servers have enough RAM to cache all those objects in memory.

Database Schema

• Ensure your schema is sufficiently de-normalized so that you never have to perform more than a 4-way JOIN across tables. This affects server-side JOIN filters as well as SSCE performance.
• To ensure uniqueness across all nodes participating in the sync infrastructure, use GUIDs for your primary keys so that SQL Server doesn’t have to deal with the overhead of managing Identity ranges. Make sure to mark your GUIDs as ROWGUIDCOL for that table so that Merge won’t try to add an additional Uniqueidentifier column to the table.  Don’t create clustered indexes when using GUIDs as primary keys because they will suffer horrible fragmentation that will rapidly degrade performance.  Just use a normal index.
• Create clustered indexes for your primary keys when using Indentity columns, Datetime, or other natural keys.  Ensure that every column in every table that participates in a WHERE clause is indexed.

Distributor

• If your network connection is fast and reliable like Wi-Fi or Ethernet, your SSCE client has more than 32 MB of free RAM, and SQL Server isn’t experiencing any deadlocks due to contention with ETL operations or too many concurrent Merge Agents, create a new Merge Agent Profile based on the High Volume Server-to-Server Profile so that SQL Server will perform more work per round-trip and speed up your synchronizations.
• If you’re using a 2G/3G Wireless Wide Area Network connection, create a Merge Agent Profile based on the Default Profile so that SQL Server will perform less work and use fewer threads per round-trip during synchronization than the High Volume Server to Server Profile which will help to reduce server locking contention and perform less work per round trip which will make your synchronizations more likely to succeed.
• In order to prevent SQL Server from performing Metadata Cleanup every time a Subscriber synchronizes, set the –MetadataRetentionCleanup parameter to 0.
• As SQL Server has to scale up to handle a higher number of concurrent users in the future, locking contention will increase due to more Merge Agents trying to perform work at the same time.  When this happens, adjust the parameters of the Default Profile so that both  –SrcThreads and –DestThreads are equal to 1.

Publication

• When defining the Articles you’re going to sync, only check the minimum tables and columns needed by the Subscriber to successfully perform its work.
• For Lookup/Reference tables that aren’t modified by the Subscriber, mark those as Download-only to prevent change-tracking metadata from being sent to the Subscriber.
• Despite the fact the column-level tracking sends less data over the air, stick with row-level tracking so SQL Server won’t have to do as much work to track the changes.
• Use the default conflict resolver where the “Server wins” unless you absolutely need a different manner of picking a winner during a conflict.
• Use Static Filters to reduce the amount of server data going out to all Subscribers.
• Make limited use of Parameterized Filters which are designed to reduce and further specify the subset of data going out to a particular Subscriber based on a HOST_NAME() which creates data partitions.  This powerful feature slows performance and reduces scalability with each additional filter, so it must be used sparingly.
• Keep filter queries simple and don’t use IN clauses, sub-selects or any kind of circular logic.
• Strive to always create “well-partitioned” Articles where all changes that are uploaded/downloaded are mapped to only the single partition ID for best performance and scalability.
  • When using Parameterized Filters, always create non-overlapping data partitions where each row from a filtered table only goes to a single Subscriber instead of more than one which will avoid the use of certain Merge metadata tables.
  • Each Article in this scenario can only be pubished to a single Publication
  • A Subscriber cannot insert rows that do not belong to its partition ID.
  • A Subscriber cannot update columns that are involved in filtering.
  • In a join filter hierarchy, a regular article cannot be the parent of a “well-partitioned” article.
  • The join filter in which a well-partitioned article is the child must have the join_unique_key set to a value of 1 which relates to the Unique key check box of the Add Join dialog.  This means there’s a one-to-one or one-to-many relationship with the foreign key.
  • Each “well-partitioned” Article can have only one subset or join filter. The article can have a subset filter and be the parent of a join filter, but cannot have a subset filter and be the child of a join filter.
• Never extend a filter out to more than 4 joined tables.
• Do not filter tables that are primarily lookup/reference tables, small tables, and tables with data that does not change.
• Schedule the Snapshot Agent to run once per day to create an unfiltered schema Snapshot.
• Set your Subscriptions to expire as soon as possible to keep the amount change-tracking metadata SQL Server has to manage to an absolute minimum. Normally, set the value to 2 to accommodate 3-day weekends since 24 hours are automatically added to the time to account for multiple time zones. If server-side change tracking isn’t needed and Subscribers are pulling down a new database every day and aren’t uploading data, then set the expiration value to 1.
• Set Allow parameterized filters equal to True.
• Set Validate Subscribers equal to HOST_NAME().
• Set Precompute partitions equal to True to allow SQL Server to optimize synchronization by computing in advance which data rows belong in which partitions.
• Set Optimize synchronization equal to False if Precompute partitions is equal to True.  Otherwise set it to True to optimize filtered Subscriptions by storing more metadata at the Publisher.
• Set Limit concurrent processes equal to True.
• Set Maximum concurrent processes equal to the number of SQL Server processor cores.  If exceesive locking contention occurs, reduce the number of concurrent processes until the problem is fixed.
• Set Replicate schema changes equal to True.
• Check Automatically define a partition and generate a snapshot if needed when a new Subscriber tries to synchronize. This will reduce Initialization times since SQL Server creates and applies snapshots using the fast BCP utility instead of a series of slower SELECT and INSERT statements.
• Add data partitions based on unique HOST_NAMEs and schedule the Snapshot Agent to create those filtered Snapshots nightly or on the weekend so they’ll be built using the fast BCP utility and waiting for new Subscribers to download in the morning.
• Ensure that SQL Server has 1 processor core and 2 GB of RAM for every 100 concurrent Subscribers utilizing bi-directional sync. Add 1 core and 2 GB of RAM server for every additional 100 concurrent Subscribers you want to add to the system.  Never add more Subscribers and/or IIS servers without also adding new cores and RAM to the Publisher.
• Turn off Hyperthreading in the BIOS of the SQL Server as it has been known to degrade SQL Server performance.
• Do not add your own user-defined triggers to tables on a Published database since Merge places 3 triggers on each table already.
• Add one or more Filegroups to your database to contain multiple, secondary database files spread out across many physical disks.
• Limit use of large object types such as text, ntext, image, varchar(max), nvarchar(max) or varbinary(max) as they require a significant memory allocation and will negatively impact performance.
• Set SQL Servers’s minimum and maximum memory usage to within 2 GB of total system memory so it doesn’t have to allocate more memory on-demand.
• Always use SQL Server 2008 R2 and Windows Server 2008 R2 since they work better together because they take advantage of the next generation networking stack which dramatically increases network throughput. They can also scale up as high as 256 cores.
• Due to how Merge Replication tracks changes with triggers, Merge Agents, and tracking tables, it will create locking contention withDML/ ETL operations.  This contention degrades server performance which negatively impacts sync times with devices.  This contention should be mitgated by performing large INSERT/UPDATE/DELETE DML/ETL operations during a nightly maintenance window when Subscribers aren’t synchronizing.
• Since Published databases result in slower DML/ETL operations, perform changes in bulk by using XML Stored Procedures to boost performance.
• To improve the performance of pre-computed partitions when DML/ETL operations result in lots of data changes, ensure that changes to a Parent table in a join filter are made before corresponding changes in the child tables.  This means that when DML/ETL operations are pushing new data into SQL Server, they must add master data to the parent filter table first, and then add detail data to all the related child tables second, in order for that data to be pre-computed and optimized for sync.
• Create filter partitions based on things that don’t change every day.  Partitions that are added and deleted from SQL Server and Subscribers that move from one partition to another is very disruptive to the performance of Merge Replication.
• Always perform initializations and re-initializations over Wi-Fi or Ethernet when the device is docked because this is the slowest operation where the entire database must be downloaded instead of just deltas.  To determine rough estimates for initialization, multiply the size of the resulting SSCE .sdf file x the bandwidth speed available to the device.  A file copy over the expected network will also yield estimates for mininum sync times.  These times don’t include the work SQL Server and IIS must perform to provide the data or data INSERT times on SSCE.
• If your SQL Server Publisher hits a saturation point with too many concurrent mobile Subscribers, you can scale it out creating a Server/Push Republishing hierarchy. Put the primary SQL Server Publisher at the top of the pyramid and have two or more SQL Servers subscribe to it. These can be unfiltered Subscriptions where all SQL Servers get the same data or the Subscribers can filter their data feeds by region for example. Then have the Subscribing SQL Servers Publish their Subscription for consumption by mobile SSCE clients.
• Create just a single Publication.

Internet Information Services

• Use the x64 version of the SQL Server Compact 3.5 SP2 Server Tools with Windows Server 2008 R2 running inside IIS 7.5.
• Use a single Server Agent in a single Virtual Directory.
• Ensure the IIS Virtual Directory where the Server Agent resides is on a fast solid-state drive that’s separate from the disk where Windows Server is installed to better support file I/O.
• Use a low-end server with 2 processor cores and 2 GB of RAM to support 400 concurrent Subscribers queued at the same time.
• Set the MAX_THREADS_PER_POOL Server Agent registry key equal to 2 to match the IIS processor cores and RAM. Do not set this value to a higher number than the number of cores.
• Set the MAX_PENDING_REQUEST Server Agent registry key equal to 400 which means the Server Agent will queue up to 400 concurrent Subscribers waiting for one of the 2 worker threads to become available to sync with SQL Server.
• Set the IIS Connection Limits property to 400 to prevent an unlimited number of connections reaching the Server Agent.
• Add a new load-balanced IIS server for every additional 400 concurrent Subscribers you want to add to the system.

Subscriber

• Use the appropriate x64, x86 or ARM version of SQL Server Compact 3.5 SP2 to take advantage of the PostSyncCleanup property of the SqlCeReplication object that can reduce the time it takes to perform an initial synchronization. Set the PostSyncCleanup property equal to 3 where neither UpdateStats nor CleanByRetention are performed.
• Increase the Max Buffer Size connection string parameter to 1024 on a phone and 4096 on a PC to boost both replication and SQL query processing performance. If you have more RAM available, set those values even higher until you reach the law of diminishing returns.
• Keep your SSCE database compact and fast by setting the Autoshrink Threshold connection string parameter to 10 so it starts reclaiming empty data pages once the database has become 10% fragmented.
• Replication performance testing must be performed using actual PDAs to observe how available RAM, storage space and CPU speed affect moving data into the device’s memory area and how quickly this data is inserted into the SSCE database tables.  Since the SSCE database doubles in size during replication, the device must have enough storage available or the operation will fail.  Having plenty of available RAM is important so that SSCE can utilize its memory buffer to complete a Merge Replication operation more quickly.  With plenty of available RAM and storage, a fast CPU will make all operations faster.
• The PDA must have at least an extra 32 MB of available free RAM that can be used by the .NET Compact Framework (NETCF) application.  If additional applications are running on the device at the same time, even more RAM is needed.  If a NETCF application has insufficient RAM is will discard its compiled code and run in interpreted mode which will slow the application down.  If the NETCF app is still under memory pressure after discarding compiled code, Windows Mobile will first tell the application to return free memory to the operating system and then will terminate the app if needed.
• Set the CompressionLevel property of the SqlCeReplication object to 0 for fast connections and increment it from 1 to 6 on slower connections like GPRS to increase speed and reduce bandwidth consumption.
• Tune the ConnectionRetryTimeout, ConnectTimeout, ReceiveTimeout and SendTimeout properties of the SqlCeReplication object based on expected bandwidth speeds:

• You can decrease potentially slow SSCE file I/O by adjusting the Flush Interval connection string parameter to write committed transactions to disk less often than the default of every 10 seconds.  Test longer intervals between flushes like 20 or 30 seconds. Keep in mind that these transactions can be lost if the disk or system fails before flushing occurs so be careful.
• When replicating data that has been captured in the field by the device, perform Upload-only syncs to shorten the duration.

Storage

• Use a Fibre Channel SAN with 15k RPM or solid-state disks for best I/O performance.
• Databases should reside on a RAID 10, unshared LUN comprised of at least 6 disks.
• Database logs should reside on a RAID 10, unshared LUN comprised of at least 6 disks.
• Tempdb should reside on a RAID 10, unshared LUN comprised of at least 6 disks.
• The Tempdb log should reside on a RAID 10, unshared LUN comprised of at least 6 disks.
• The Snapshot share should reside on a RAID 10, unshared LUN comprised of at least 6 disks.  This disk array should be large enough to accommodate a growing number of filtered Snapshots. Snapshot folders for Subscribers that no longer use the system must be manually deleted.
• Merge Replication metadata tables should reside on a RAID 10, unshared LUN comprised of at least 6 disks.
• Increase your Host Bus Adapter (HBA) queue depths to 64.
• Your Publication database should be broken up into half the number of files as the SQL Server has processor cores. Each file must be the same size.
• Tempdb should be pre-sized with an auto-growth increment of 10%. It should be broken up into the same number of files as the SQL Server has processor cores. Each file must be the same size.

High Availability

• Load-balance the IIS servers to scale them out. Enable Server Affinity (stickiness) since the Replication Session Control Blocks that transmit data between the Server Agent and SSCE are stateful. Test to ensure that your load-balancer is actually sending equal amounts of Subscriber sync traffic to each IIS server.  Some load-balancers can erroneously send all traffic to a single IIS server if not properly configured.
• Implement Windows Clustering so that SQL Server can failover to a second node.
• Using SQL Server Mirroring so that your Published database will failover to a standby server.
• Make a second SQL Server into an unfiltered Subscriber to your Publisher so that it can take over Merge Replication duties for mobile clients as a Republisher if the primary SQL Server fails. SSCE clients would just have to reinitialize their Subscriptions to begin synchronizing with the new Republisher.

Ongoing Maintenance

• Use the Replication Monitor to have a real-time view of the synchronization performance of all your Subscribers.
• Use the web-based SQL Server Compact Server Agent Statistics and Diagnostics tools to monitor the health and activity of the Server Agent running on IIS.
• Create a SQL Job to execute the sp_MSmakegeneration stored procedure after large DML operations. Regular execution after INSERTING/UPDATING/DELETING data from either DML/ETL operations or after receiving lots of changes from Subscribers will maintain subsequent sync performance. Executing this stored procedure from the server-side is preferable to having it executed as a result of a Subscriber sync which would block all other Subscribers.
• During your nightly maintenance window, rebuild the indexes and update the statistics of the following Merge Replication metadata tables:
  • MSmerge_contents
  • MSmerge_tombstone
  • MSmerge_genhistory
  • MSmerge_current_partition_mappings
  • MSmerge_past_partition_mappings
  • MSmerge_generation_partition_mappings
• If you notice performance degradation during the day due to a large number of Subscribers making large changes to the database, you can updates the statistics (with fullscan) of the Merge Replication metadata tables more frequently throughout the day to force stored proc recompiles to get a better query plan.
  • UPDATE STATISTICS MSmerge_generation_partition_mappings WITH FULLSCAN
  • UPDATE STATISTICS MSmerge_genhistory WITH FULLSCAN
• Rebuild/defrag indexes on your database tables and Merge Replication metadata tables throughout the day to reduce locking contention and maintain performance.
• Use the Missing Indexes feature of SQL Server to tell you which indexes you could add that would give your system a performance boost. Do not add recommended indexes to Merge system tables.
• Use the Database Engine Tuning Advisor to give you comprehensive performance tuning recommendations that cover every aspect of SQL Server.
• Monitor the performance of the following counters:
  • Processor Object: % Processor Time: This counter represents the percentage of processor utilization. A value over 80% is a CPU bottleneck.
  • System Object: Processor Queue Length: This counter represents the number of threads that are delayed in the processor Ready Queue and waiting to be scheduled for execution. A value over 2 is bottleneck and shows that there is more work available than the processor can handle. Remember to divide the value by the number of processor cores on your server.
  • Memory Object: Available Mbytes: This counter represents the amount of physical memory available for allocation to a process or for system use. Values below 10% of total system RAM indicate that you need to add additional RAM to your server.
  • PhysicalDisk Object: % Disk Time: This counter represents the percentage of time that the selected disk is busy responding to read or write requests. A value greater than 50% is an I/O bottleneck.
  • PhysicalDisk Object: Average Disk Queue Length: This counter represents the average number of read/write requests that are queued on a given physical disk. If your disk queue length is greater than 2, you’ve got an I/O bottleneck with too many read/write operations waiting to be performed.
  • PhysicalDisk Object: Average Disk Seconds/Read and Disk Seconds/Write: These counters represent the average time in seconds of a read or write of data to and from a disk. A value of less than 10 ms is what you’re shooting for in terms of best performance. You can get by with subpar values between 10 – 20 ms but anything above that is considered slow. Times above 50 ms represent a very serious I/O bottleneck.
  • PhysicalDisk Object: Average Disk Reads/Second and Disk Writes/Second: These counters represent the rate of read and write operations against a given disk. You need to ensure that these values stay below 85% of a disk’s capacity by adding disks or reducing the load from SQL Server. Disk access times will increase exponentially when you get beyond 85% capacity.
• A limited number of database schema changes can be made and synchronized down to SSCE Subscribers without any code changes which makes it easier to update your system as it evolves over time.
• Use a Merge Replication Test Harness to stress test the entire system.  The ability to simulate hundreds or thousands of concurrent synchronizing Subscribers allows you to monitor performance and the load on the system.  This is helpful in properly configuring and tuning SQL Server, IIS, and the Windows Mobile devices.  It will tell you where you’re having problems and it will let you predict how much server hardware you will need to support growing numbers of Subscribers over time.  It’s also very important to simulate worst-case scenarios that you never expect to happen.

I hope this information sufficiently empowers you to take on the largest MEAP solutions that involve SQL Server Merge Replication and SQL Server Compact.  If you need a deeper dive, go check out my book on Enterprise Data Synchronization http://www.amazon.com/Enterprise-Synchronization-Microsoft-Compact-Replication/dp/0979891213/ref=sr_1_1?ie=UTF8&s=books&qid=1271964573&sr=1-1 over at Amazon.  Now go build a fast and scalable solution for your company or your customers.

Best Regards,

Rob

P.S.  If your solution doesn’t require all the advanced features found in Merge Replication, I highly recommend you use Remote Data Access (RDA).  This is a much simpler sync technology that’s extremely fast, scalable, and easier to manage.

1. Works the same across laptops, tablets, Netbooks and phones.
2. Gives them reusable mobile middleware that can support multiple simultaneous applications rather than needing something different for each point solution
3. Lowers risk to their projects by reducing the amount of custom code needed to build any given solution.
4. Gives them adapters that plug into the existing enterprise packages they use to run their business.

Regards,
Rob

While you probably knew that My Phone backs up your photos, contacts, text messages, music, documents and IE favorites to the cloud, you might not have known that My Phone can be used to locate and secure a lost phone.  You can ring, lock, erase, and locate your lost phone on a map.  On the CBS Early Show, Natali was able to display a message on her lost Windows phone instructing whoever found it to contact her and return it.  Pretty cool stuff and a great example of Microsoft tying the cloud to the third screen.

Best Regards,

Rob

With the tidal wave of mobile and wireless technologies sweeping across both the consumer and enterprise landscapes, I believe MEAP offerings give us a glimpse of a new standard for designing all future infrastructures.

-Rob

1. When there are 3 or more mobile applications
2. When there are 3 or more targeted operating systems or platforms
3. When they involve the integration of 3 or more back-end systems

Leaders in this space have included Sybase iAnywhere, Antenna, Dexterra, Syclo and Spring Wireless.  Microsoft goes from a large Mobile General Store with myriad solutions to a player in this space with a MEAP solution of our own:  Visual Studio is used to build the mobile logic and UI.  Merge Replication provides occasionally-connected data synchronization between SQL Server Compact on the mobile device and SQL Server in the data center.  SQL Server Business Intelligence Development Studio is used to visually create connections to back-end systems like SAP or databases like Oracle.  Data in transit is secured via SSL or VPN, data at rest is encrypted via device encryption, SQL Server Compact, BitLocker or programmatically through the Crypto API.  Integration packages that communicate with back-end systems are encrypted and digitally signed. 

We already have the best mobile email, calendaring, and contacts product in the business where Exchange Active Sync keeps Outlook and Outlook Mobile always up to date with Exchange Server.  Server-to-device as well as peer-to-peer device notifications are facilitated through WCF Store and Forward on Exchange.  Software and patch distribution along with device settings and policy management is accompished via System Center Configuration Manager.  ISA Server provides both VPN and Reverse Proxy access to roaming applications on the Internet on any platform.

When you put this stack in place and resuse it for multiple mobile applications instead of going with point solutions, ROI savings increase as the need for POCs, Pilots and training are reduced and the need for extra client access licenses is eliminated.  That’s Gartner’s first requirement.  We hit Gartner’s second requirement by uniformly supporting 3 mobile operating systems in the form of Windows, Windows CE, and Windows Mobile.  Last but not least, our SQL Server Integration Services technology combined with dozens of connectors mean we can connect your mobile devices with almost any back-end package or database.

Yes, Microsoft does have a Mobile Enterprise Application Platform that’s already proven to scale to tens of thousands of devices and it will definitely save you time and money.

- Rob

With the world’s largest organizations rolling out tens of thousands of Windows® phones, laptops, tablets and Netbooks to empower their respective mobile workforces, the ability to create mobile line of business solutions that support large numbers of users is absolutely critical. In my fourth book on mobile infrastructure and development, I show you how to take the SQL Server data you use to run your organization and make it available to all of your mobile employees.

Step-by-step, I’ll walk you through the process of building a secure, performant, n-tier, mobile enterprise application platform architecture designed to scale to thousands of users. You’ll also learn how to create occasionally-connected .NET applications designed to thrive in unreliable wireless conditions.

• Learn how to “Mobilize” your organization by making your enterprise data available to employees carrying Windows® phones, laptops, Netbooks and tablets in the field.
• Learn how to build an N-Tier Mobile Sync infrastructure that will scale to thousands of users.
• Learn how to create occasionally-connected .NET applications designed to thrive in unreliable wireless conditions.
• Learn best practices in security, reliability, performance, load-balancing, reverse proxy and hardware configuration.
• Learn how to implement this technology in real world scenarios like supply chain management, retail, sales force automation, healthcare and emergency management.

Keep in mind that the knowledge you gain from this book didn’t come from me dreaming this stuff up in an Ivory Tower.  It came from building some of the worlds largest and most complex data synchronization systems for the world’s largest companies.  In addition to the hands-on experience that went into this book, I’d also like to thank some of my colleagues for their invaluable contributions:

• Liam Cavanagh is a Senior Lead Program Manager for Microsoft’s Sync Framework and Cloud Data Services and he wrote the forward.
• Catherine Wyatt is the Managing Editor for Hood Canal Press who made the publishing of the book possible.
• Darren Shaffer is the CEO of Handheld Logic and he wrote the Chapter on building the Mobile Subscriber. 
• Michael Jimenez is a Mobility Architect at Microsoft and he wrote the Appendix that shows you how to create an ISA Server 2006 Reverse Proxy to publish your sync infrastructure to the Internet.

It’s my sincere hope that this book will encourage you to un-tether your workforce from their desktop computers and boost your organization’s agility by pushing out critical business functions to the point of activity where employees are empowered to make timely decisions and perform tasks that best serve the interests of their customers and their company.

This repudiation of the traditional “connected” software application model increases customer satisfaction, boosts worker efficiency, reduces “missed opportunities” and results in cost savings as “un-wired” employees get their jobs done wherever they happen to be.

Best Regards,

Rob

One Phone for Work and Play

With a Windows phone, people can rely on their phone to balance their lives, from work to home to play. Whether it’s editing a document or sharing several vacation updates through a social networking application, Windows phones help people stay connected to the people and information they care about most. Because people’s phones often match their personality and unique needs, Microsoft now offers the Windows phone Custom Theme Creator. People can create personalized themes for their Windows phone by choosing the color and design that suits their style at http://www.windowsphone.com/theme.

With a Windows phone, people will have familiar work and play experiences right from their Start button, including these:

• A new enhanced Windows Live experience with What’s New feeds and improved Windows Live photo sharing across major social networking sites (such as Twitter, Facebook, MySpace and Flickr)

• A best-in-class e-mail experience and the ability to manage multiple accounts right from their phone with Outlook Mobile and Exchange Server synchronization not to mention Hotmail

• The ability to use PowerPoint and open and edit Word and Excel documents from their phone with Microsoft Office Mobile

• The power to sync files on the phone through Windows Live Media Manager and play media files seamlessly with Windows Media Player

• A redesigned Windows Internet Explorer mobile browser with Adobe Flash Lite that brings the mobile Web browsing experience closer to what people have come to expect from their PC

Powerful Mobile Services

With the launch of these new Windows phones, Microsoft debuts new services that bring added value to people.

My Phone is a free service that helps people manage and back up the invaluable information stored on their phone and provides peace of mind and an easy restore option in the event of a lost or damaged phone. My Phone automatically synchronizes the specific types of a phone’s content the user chooses — from contacts and appointments to texts, photos and more — to a password-protected Web site. People can also publish their photos from the My Phone Web site or their phone directly to Windows Live, Facebook, MySpace and Flickr. As part of the free service, people can go online and map the last known location of their phone from when it was last synchronized. In addition, a set of more advanced “lost phone” features are being offered as a premium package that people can activate as needed. The My Phone Premium package includes the ability to immediately locate the phone’s current location on a map (in the U.S. only); remotely lock a phone and post an “if found” message to its screen; loudly ring the phone even if it is set to vibrate or silent mode; and ultimately, if needed, completely erase the phone to protect personal data from falling into the wrong hands. People using My Phone on their phone running either Windows Mobile 6.0, 6.1 or 6.5 can access the premium package free of charge until Nov. 30, 2009. After that date, seven-day access to the premium package will be available for purchase for $4.99 in the U.S.

Also launching today is Windows Marketplace for Mobile, which offers people an easy way to find and purchase high-quality mobile applications for both work and play, while creating a new opportunity for developers to reach millions of people using Windows phones worldwide. Microsoft is pleased today to introduce 246 quality mobile applications initially in Windows Marketplace for work and play, with more than 753 ISVs worldwide on board to continue building out the catalog. People will have access to not only important line-of-business applications for work, but also popular mobile applications for play such as Facebook, MySpace, Netflix, Twikini, WunderRadio and ZAGAT, as well as leading game titles including Sudoku, “Guitar Hero World Tour” and the “PAC-MAN” series, all of which can be easily purchased and installed directly on a Windows phone. All purchased applications are certified by Microsoft to run on Windows phones and are backed by a simple return policy. Over the next few months, Microsoft will continue to evolve Windows Marketplace for Mobile to bring to market a fresh take on the app store that delivers strong value for developers and a great shopping experience for people.

Choice and Availability: A Phone for Everyone

Windows phones give people the power to choose the phone that best fits their style and budget by offering phones with a full physical QWERTY keyboard, others with just a large touch screen, and some with both touch screen and keyboard.

So what’s new for Developers?

Web developers can now target Internet Explorer Mobile for their applications.  Not only does this browser provide full desktop fidelity, but it also includes the fast IE8 Javascript engine to speed up code execution, DOM manipulation and Ajax calls.  Don’t take my word for it, test it out over at http://asp.net and watch it accurately render all the Ajax controls.

Both Native and Managed code developers get to take advantage of the new Gesture API to add panning and flicking to their apps.  The built-in physics engine allows developers to add smooth, finger-friendly scrolling.

For developers that are more at home with Cascading Style Sheets, HTML, Javascript and Ajax, Windows Mobile 6.5 introduces Widgets as an alternative to C++ or .NET.  These programs take advantage of the new Internet Explorer Mobile to render Rich Internet Applications that run outside the browser but take advantage of the power of the Web.

Any changes under the Hood?

Yes, since the launch of Windows Mobile 6, we’ve been progressively enhancing the Windows CE 5 kernel that powers the Windows Mobile platform.  You should expect better application stability, much more virtual memory available to running apps, and faster execution.

Have fun with your new phones!

- Rob

Besides synchronizing the tables, rows and columns of a complete database between mobile clients and servers, the use of message queuing should be considered for many scenarios due to its high-reliability by ensuring that a critical message arrives at its destination.  Products like MSMQ, MQ Series, Tibco and JMS are used all over the world in the most mission-critical environments to ensure a high level of availability and reliability.  They’re asynchronous by nature and use store and forward mechanisms so that messages get from point A to B to C.  A typical queue message includes the Destination which tells the messge where to go, a Label which describes the message, a Body which contains the message, and a Body Length so the receiver can verify that it received everything.

So how does any of this relate to Windows phones?  A number of years ago, an MSMQ client was made available for download and installation on Windows phones.  Additionally, the .NET Compact Framework 2.0 included classes to work with MSMQ.  Unfortunately, the installation of MSMQ was far from seamless which inhibited its adoption by customers.  More recently, functionality was included in the .NET Compact Framework 3.5 that facilitated store and forward messaging using Exchange 2007 as a transport.  This is a good solution for customers running the newest version of Exchange, have an unlimited data plan for their phones, and don’t mind running line of business applications over their email infrastructure.

So what do we do for customers that have found the Windows Mobile MSMQ client too much of a hassle, don’t have Exchange 2007 or don’t want to use it as a mobile message queue server?  I think the answer has been under our noses all along.  Burned in the ROM of every Windows Mobile 6.x device is SQL Server Compact + a lightweight data sync solution called Remote Data Access (RDA).  For those of you running Windows Mobile 5, XP, Vista or 7, you can easily download these bits to your mobile client.  SQL Server Compact is you local queue, RDA is your transport and SQL Server in the cloud or data center is your message queue server.  So let’s break this down and see how it will work.

A mobile application that captures data in the field would want to drop that info in a local queue.  SQL Server Compact becomes that local queue and the message format is actually a table with the following structure:

This table would be created inside a MessgeQueue database in SQL Server and RDA would pull it down to SQL Server Compact.  In the Pull method you call from .NET on the client, you would add “WHERE 1=0” to the SQL statement.  This filter has the effect of pulling down an empty shell of the table without retrieving any data to the client since that’s all you want.  It also means that when you insert local data into the table and call the Push method, the data will be removed from the client at the completion of a successful sync.

So you’re probably wondering, what makes this so special and message queue-like vs. anything else?  The secret is that unlike other sync technologies, RDA can wrap the upload of data into a transaction.  As the data is being uploaded over wireless, if any of the INSERTs into SQL Server fail for whatever reason, everything gets rolled-back and the original data remains in the local SQL Server Compact queue.  This is the kind of guaranteed commit that you expect from a message queuing system.  It’s an “all or nothing” success or rollback. 

It actually gets better.  A property of both RDA and Merge Replication is called ConnectionRetryTimeout.  This feature is designed to help you with unreliable wireless coverage where you have signal one minute and then lose it the next.  Let’s say you have this timeout value set to 2 minutes and you begin your Push upload of queued data.  Everything is working fine for the first few seconds but then you lose wireless coverage from your mobile operator.  If you regain coverage before the 2 minute time-out, the upload will resume where it left off.  Since both RDA and Merge send and receive data between the SQL Server Compact and IIS in tiny blocks, you never have to worry about running out of memory and you can pick up where you left off in case of a network dropout.

So the big takeaway here is that we do in fact have a Mobile Message Queue solution hidden on our Windows phones.  We have a message format that lets us drop text/xml/whatever data into the body, a label that a server process or SQL trigger can key off of to perform an action, and a transactional upload mechanism that ensures your critical data will cross the wireless chasm and make it to the other side intact.

So what’s next?  Now that you can capture data in a local queue and safely upload it, you might be wondering how queued messages from someone else can be pushed to your device.  Don’t worry, that will be in my next post.  Also, this isn’t just an article on how to solve a big problem in the mobile space, I’m actually building the necessary client and server pieces as well.  We’re all looking for a reliable and unified way to connect mobile devices to corporate assets and this just might be the simple answer we’re looking for.

- Rob