This blog post covers some of the issues that can be encountered when using SQL Server Compact under ASP.NET, in particular in a hosted environment, where the environment can be restricted in unexpected ways.

In order to be able to troubleshoot issues with ASP.NET hosting of web sites using SQL Server Compact 4.0, it can be useful to understand how Medium Trust works.

Medium Trust only works under .NET 4.0 (ASP.NET 4.0), as it depends on a configuration in the global web.config.

The global web.config is located at C:\Windows\Microsoft.NET\Framework\v4.0.30319\Config or C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Config, and contains the following setting, that was configured when .NET 4.0 was installed:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <location allowOverride="true">
    <system.web>
      <fullTrustAssemblies>
        <add
            assemblyName="System.Data.SqlServerCe"
            version="4.0.0.0"
            publicKey="0024000004800000940000000602000000240000525341310004000001000100272736ad6e5f9586bac2d531eabc3acc666c2f8ec879fa94f8f7b0327d2ff2ed523448f83c3d5c5dd2dfc7bc99c5286b2c125117bf5cbe242b9d41750732b2bdffe649c6efb8e5526d526fdd130095ecdb7bf210809c6cdad8824faa9ac0310ac3cba2aa0523567b2dfa7fe250b30facbd62d4ec99b94ac47c7d3b28f1f6e4c8"
                />
      </fullTrustAssemblies>
      <partialTrustVisibleAssemblies />
    </system.web>
  </location>

So this was added during installation of .NET 4.0, long before SQL Server Compact 4.0 was released in January 2011. The fullTrustAssemblies element is new in .NET 4.0: http://weblogs.asp.net/asptest/archive/2010/04/23/what-is-new-in-asp-net-4-0-code-access-security.aspx

If this section has been removed from the global web.config file, SQL Server Compact 4.0 will not run under medium trust.

It is possible to configure similar functionality under .NET 3.5 SP1, as described here: http://msdn.microsoft.com/en-us/library/ms174612(v=SQL.110).aspx

Other issues you may encounter when running ASP.NET and SQL Server Compact 4.0 can be:

- The process identity running your web application must have write access to the App_Data folder or the folder where your database file resides:

http://social.msdn.microsoft.com/Forums/en-US/sqlce/thread/6c1c8798-85d7-4d8f-9908-e9f7dc1c17e4

- SQL Server Compact database files must be located on a local drive, not shared folders:

http://stackoverflow.com/questions/7791345/sqlce-4-ef4-1-internal-error-cannot-open-the-shared-memory-region/7804049#7804049

- If SQL Server Compact is not installed locally, a number of files must be included in the bin folder of your web site:

http://stackoverflow.com/questions/3223359/cant-get-sql-server-compact-3-5-4-to-work-with-asp-net-mvc-2/3223450#3223450

http://msdn.microsoft.com/en-us/library/gg286946.aspx

- If only .NET 4.0 is present on the server, you may be missing the Visual C++ 2008 SP1 Redistributable Package (which is installed only with .NET 3.5 SP1)

I will update this blog post if I notice more “common issues”.

After more than 66.000 downloads, version 2.5 of my SQL Server Compact Toolbox extension for Visual Studio 2010 is now available for download. This blog post is a visual guide to the new features included in this release, many suggested by users of the tool via the CodePlex issue tracker

Properties of selected Database and Table displayed in Properties window

When you navigate the list of SQL Server Compact databases and database objects, the Toolbox now displays properties for a Database and a Table.

The Database properties are the following:

And the table properties are these:

DGML file (database graph) now also contains object descriptions

The database graph (DGML) file has been enhanced to display descriptions of Database, Table and Column objects, based on the documentation feature introduced in version 2.3:

Entity Data Model dialog now allows adding configuration for Private Desktop Deployment to app.config

When using Entity Framework with Private Deployment of SQL Server Compact, some entries in app.config are required (for Desktop application), as described here and here. This is required, as Entity Framework build in the DbProvider interfaces, which requires some configuration entries.

These settings can now be added when creating an EDM as shown below:

Ability to add 3.5 Connections from Toolbox

It is now possible to add 3.5 Database Connections to the Server Explorer and the Toolbox directly from the Toolbox, rather than having to go to Server Explorer, add the connection, and then Refresh the Toolbox. You can now do this without leaving the Toolbox, and the Toolbox will refresh “automagically”.

Improved VS UI Guidelines compliance

The Toolbars, SQL Editor Font, Dialogs (frame, background and buttons) have been overhauled to comply better with my recent discovery of “Visual Studio UI Guidelines”, available for download here. In addition, the Toolbox now follows the chosen Visual Studio Theme (to some extent, anyway).

This is the “new look” for the Toolbox Explorer and SQL Editor:

Other minor improvements and fixes

- Explorer Treeview: ArgumentException when getting connections
- WinPhone DataContext: Split files failed with empty database
- SQL editor: Check if .sqlplan is supported
- SQL editor: Save button was not working
- SQL editor: Results pane not always cleared
- SQL editor: Results as text improved formatting
- SQL editor: Text scrollbar was overlaid by splitter bar

As usual, the full source code for these new features is available on CodePlex for you to re-use or maybe improve!

A major update to SQL Server Compact 3.5 SP2 has just been released, version 3.5.8088.0, disguised as a “Cumulative Update Package”. Microsoft knowledgebase article 2628887 describes the update.

The update contains updated Server Tools, and updated desktop and device runtimes, all updated to support Merge Replication with the next version of SQL Server, SQL Server 2012.

For a complete list of Cumulative Updates released for SQL Server Compact 3.5 SP2, see my blog post here.

It is nice to see that the 3.5 SP2 product, with it’s full range of device support and synchronization technologies is kept alive and kicking.

NOTE: I blogged about this update earlier, but it was pulled back. Now it is finally available, and all downloads can be requested. (I have downloaded all the ENU ones, anyway)

This is an attempt at a comprehensive list of SQL Server Compact 3rd party tools, both commercial and non-commercial. Please let me know if I have missed anything, and I will add it.

Just to summarize, the current Microsoft supplied tools are the following:

Visual Studio 2010 (3.5 SP2)
Visual Studio 2010 SP1 + 4.0 Tools (3.5 SP2 + 4.0)
WebMatrix (4.0)

New or updated entries are noted with (new) or (updated)
”Last updated” indicates when the tool was last updated by it’s author.

Commercial tools

Name: Data Port Console
Link: http://primeworks-mobile.com/Products/DataPortConsole.html
SQL CE Versions: 2.0, 3.0, 3.5, 3.5, 4.0
Last updated: Current
Platforms: Windows CE, Win32
”Manage all your SQL Compact Edition databases from one central location with the new Data Port Console.”

Name: DesktopSqlCe
Link: http://primeworks-mobile.com/Products/DesktopSqlCe.html
SQL CE Versions: 2.0, 3.0, 3.5
Last updated: 2010
Platforms: Windows CE, Win32
”DesktopSqlCe is a desktop .NET Framework component to remotely access SQL Server Compact Edition,  SQL Mobile and SQL CE 2.0 databases on mobile or embedded devices.”

Name: Data Port Wizard
Link: http://primeworks-mobile.com/Products/DataPortWizard.html
SQL CE Versions: 2.0, 3.0, 3.5, 4.0
Last updated: Current
Platforms: Windows CE, Win32
”Quickly and easily transfer databases from the desktop to the Mobile Device and back. Data Port Wizard will make full copies of your Microsoft Access or Microsoft SQL Server databases into SQL Server Compact Edition 3.5 and 3.0, and SQL CE 2.0 databases - and back.”

Name: Data Port Command
Link: http://primeworks-mobile.com/Products/DataPortCommand.html
SQL CE Versions: 2.0, 3.0, 3.5
Last updated: 2010
Platforms: Windows CE, Win32
”Quickly and easily transfer databases from the desktop to the Mobile Device and back. Data Port Command will make full copies of your Microsoft Access or Microsoft SQL Server databases into SQL Server Compact Edition databases - and back.”

Name: SQLCE Desktop Manager v2.00
Link: http://www.b3team.com/sqlcedesktopmanager.html
SQL CE Versions: 2.0, 3.0, 3.5
Last updated: 2007
Platforms: Windows CE, Win32
“SQLCE Desktop Manager v2.00 is available! This application will help you manage your SQL Server Compact Edition, Mobile Edition, SQL Server CE and SQLite 3 databases on PC or Windows CE/Mobile devices directly from desktop computer. With it you can create databases, design tables, edit data and run SQL queries.”

Name: SQLCE Database Viewer
Link: http://www.pocketpctoolkit.com/index.php
SQL CE Versions: 2.0, 3.0, 3.5
Last updated: 2007
Platforms: Windows CE, Win32
”The easiest way to view and edit your device's SQLCE databases, directly on your Desktop ! View and edit the tables, columns and data in your device's SQLCE 2.0 and SQL Server Mobile, and SQLCE Compact Edition v3.5 tables. Export their data structures (and data) into any ODBC data source. View and edit your device's registry entries & processes. Built-in file explorer - the easiest way to view, open and execute the files on your device.”

Name: SDF Viewer 
Link: http://www.flyhoward.com/SDF_Viewer.aspx
SQL CE Versions: 3.0, 3.5, 4.0
Last updated: Current
Platforms: Win32
“The simple way to create, view, edit or export the data in your Windows Mobile or Pocket PC device's SQLCE database, from the convenience of your Desktop. Run SQL commands on your SQLCE databases, and view the results instantly.”

Name: RemoteSqlCe
Link: http://www.gui-innovations.com/html/remotesqlce.html
SQL CE Versions: 2.0, 3.0
Last updated: 2007
Platforms: Windows CE, Win32
”SqlServerCE is a great database for Pocket PC’s, but designing, maintaining and querying databases on the PPC, especially whilst developing, can be wearing. We developed this tool to help us when developing for SqlServerCE. With RemoteSQLCe you can completely manage your SqlServerCE database from the desktop. RemoteSQLCE works in the cradle, and over ethernet/bluetooth.” 

Non-commercial (free) tools

My CodePlex Tools

Name: SQL Server Compact Toolbox (updated)
Link: http://sqlcetoolbox.codeplex.com/
SQL CE Versions: 3.5, 4.0
Last updated: Current
Platforms: Win32, Win64 
Added to list: 29th June 2010
”Add-in for Visual Studio 2010 (for 3.5 and 4.0). Standalone app (for 4.0)
Adds several features to help your SQL Server Compact development efforts:
- Scripting of tables, including data.
- Script of entire schema, optionally with data, both of SQL Server Compact and SQL Server 2005 or later databases, including SQL Azure.
- Import to SQL Compact 3.5 from a server database or CSV file.
- Basic, free form query execution.
- Generation of DGML files for visualizing table relationships
- Database Diff Script
- Detect database file version
- Generate EDMX in any project type
- Generate Windows Phone DataContext
- Fix broken connection definitions”

Name: SqlCeScripting.dll (updated)
Link: http://exportsqlce.codeplex.com/
SQL CE Versions: 3.5, 4.0
Last updated: Current
Platforms: Win32, Win64 
Added to list: 29th June 2010
”Scripting library for inclusion in your own application, enumerates SQL Compact objects and allows scripting of these.”

Name: SqlCeCmd 
Link: http://sqlcecmd.codeplex.com/
SQL CE Versions: 3.5, 4.0
Last updated: 2010
Platforms: Win32, Win64
”Administer SQL Compact databases directly from the command line, without having Visual Studio or SQL Server Management Studio installed. Only requires NetFx 2.0 and SQL Compact installed. Command line options similar to sqlcmd. Runs scripts generated by ExportSqlCe.”

Name: ExportSqlCe 
Link: http://exportsqlce.codeplex.com/
SQL CE Versions: 3.0, 3.5, 4.0
Last updated: Current
Platforms: Win32, Win64 
”This command line utility helps you better manage your SQL Compact development efforts. If allows you to script schema and data from any SQL Compact database file to a .sql file, which can be used in any context. The utility is developed in C# with Visual Studio 2008. You can use the resulting script for documentation, programmatic scripting of SQL Compact database files, or migration of schema and data to SQL Server (Express) 2005 or later.

Name: Export2SqlCe
Link: http://exportsqlce.codeplex.com/
SQL Server Versions: 2005, 2008 
Last updated: Current
Platforms: Win32
Added to list: 4th March 2010
”A command line utility to script an entire SQL Server 2005 or later database with or without data to a SQL Compact T-SQL script.”

Name: ExportSqlCe SSMS 2008 Addin
Link: http://exportsqlce.codeplex.com/
SQL CE Versions: 3.5
Last updated: 2011
Platforms: Win32
”Script database/table schema/data and also DML  (SELECT, INSERT, UPDATE, DELETE) directly from the Object Explorer in SQL Server 2008 Management Studio (Express). In addition, Edit table data, Import data from as CSV file and Rename the table.”

Name: SqlCeBulkCopy
Link: http://sqlcebulkcopy.codeplex.com/
SQL CE Versions: 3.5, 4.0
Last updated: Current 
Platforms: .NET library, .NET CF library 
Added to list: 4th March 2010 
”.NET Library for loading data fast (doing bulk inserts) into a SQL Server Compact database file. Attempts to mimic the SQLClient SqlBulkCopy API. Includes support for BULK INSERTing any IEnumerable”

Query and database management tools

Name: CompactView (new)
Link: http://sourceforge.net/p/compactview/home/Home/
SQL CE Versions: 3.1, 3.5, 4.0
Last updated: Current
Platforms: Win32, Win64
Added to list: 14th November 2011
CompactView is a viewer for Microsoft® SQL Server® Compact Edition (SQLCE) database files (*.sdf). CompactView can open database files of versions 3.1, 3.5 and 4.0

Name: SQL Compact Query Analyzer
Link: http://sqlcequery.codeplex.com/
SQL CE Versions: 3.1, 3.5, 4.0
Last updated: Current
Platforms: Win32, Win64
Added to list: 3rd July 2011 
Features:
- Execute SQL Queries against a SQL Server Compact Edition database
- Table Data Editor to easily edit the contents of the database
- Supports SQLCE 3.0, 3.1, 3.5 and 4.0
- Execute multiple SQL queries (delimited by a semi colon ;)
- Display query result as XML
- Shrink and Compact Databases
- Generate Schema and Data Scripts
- Display database and schema information

Name: LINQPad
Link: http://www.linqpad.net/
SQL CE Versions: 3.5, 4.0
Last updated: Current
Platforms: Win32, Win64
Added to list: 14th December 2010
Linqpad allows you to query various data sources via LINQ, including SQL Server Compact 3.5 and 4.0. See this blog post.

Name: Database .NET
Link: http://fishcodelib.com/Database.htm
SQL CE Versions: 3.1, 3.5, 4.0
Last updated: Current
Platforms: Win32, Win64
Added to list: 20th September 2010
“Database .NET is a simple and intuitive database management tool for multiple databases. Making Local and Remote Database Administration Easier, With it you can Create databases, Design tables, Edit data, Export data, Generate scripts and run SQL queries.”

Name: Compact Query
Link: http://www.yohz.com/
SQL CE Versions: 3.0, 3.5
Last updated: 2010
Platforms: Win32
Added to list: 4th March 2010 
”Compact Query is our FREE native code IDE for SQL Server Compact. Have you ever wanted to deploy a lightweight IDE for SQL Server Compact, to allow users to troubleshoot data integrity issues? Then Query Compact is the tool for you. It's magnitudes smaller than Management Studio, is a native code application (so it does not require the .Net framework to be installed)”

Name: SQL CE Browser
Link: http://ericwillis.com/notes/2009/12/sql-ce-browser-v-110/
SQL CE Versions: 3.5
Last updated: 2010
Platforms: Win32, Win64 (?)
Added to list: 4th March 2010 
”Open regular and encrypted/passworded databases for viewing or editing. Write and work with any number of queries at once via tabs. SQL syntax highlighting and SQL syntax verification. Explore columns, keys, and indexes“

Name: Mini SQL Query
Link: http://minisqlquery.codeplex.com/ 
SQL CE Versions: 3.5
Last updated: Current
Platforms: Win32, Win64 (?)
Added to list: 4th March 2010 
”Mini SQL Query is a minimalist SQL query tool for multiple databases (MSSQL, MSSQL CE, SQLite, OLEDB, MS Access/Excel files etc). The tool uses a generic schema engine taking advantage of the ADO.NET providers. Mini SQL Query is also easily extended with plugins.“

Name: SQL Server CE Query Tool
Link: http://cequery.codeplex.com
SQL CE Versions: 3.5, 4.0
Last updated: Current 
Platforms: Win32, Win64 (?)
”CEQuery is written in C# with the help of Visual Studio 2008 platform. Some of the features of this tool are described below:
User is able to create CE database. If the schema is provided, the tool is able to create the tables and columns. User is able to overview an existing CE database with its table and column structure in tree format. User can design a query by dragging and dropping tables and columns from the database tree. User can manipulate [Insert/Update/Delete] records in a selected table. User is able to generate scripts from a single table or of whole databases. [Both for SQL CE or SQL Server]. User is able to open multiple SQL Server CE files simultaneously. User is able to convert a SQL Server database to a SDF file with schema and data. User is able to change password of a SQL CE DB. The tool is now capable of handling the database with no password.“

Name: SeasonStar Database Management(SSDM)
Link: http://ssdm.codeplex.com/
SQL CE Versions: 3.5?, 4.0 
Last updated: Current
Platforms: Win32
”With support for all major database formats, top-class export functions and advanced feaures like Linq Data Context availabe, SSDM is probably your best choice for DB management. It's even free software!”

Name: SQL CE Database Editor
Link: http://sqlcedatabaseeditor.codeplex.com/
SQL CE Versions: 3.5
Last updated: 2009 
Platforms: Win32
“Small lightweight application to browse and edit Microsoft SQL CE server databases. Does not require SQL CE server to be installed and can run custom queries.”

Name: DataBoy
Link:http://databoy.codeplex.com/
SQL CE Versions: 3.0, 3.5
Last updated: 2008
Platforms: Win32
“Small footprint, portable and simple database query tool. Support SQL CE 3.1, SQL CE 3.5 and SQL Server.”

Name: SqlCeExplorer
Link:http://sqlceexplorer.codeplex.com/
SQL CE Versions: 3.5, 4.0
Last updated: Current 
Platforms: Win32
“A Simple GUI for SQL CE Server, using Microsoft .Net windows forms. This utility will help developers to create, and manage SQL CE database, without Visual Studio or SQL Server Management studio.”

Other tools and libraries

Name: SQLCE SCRIPTER (new)
Link: http://sqlmgmt.com/sql/SQLCE-Scripter
SQL CE Versions: 3.5, 4.0
Last updated: Current
Platforms: Win32, Win64
Added to list: 14th November, 2011
“Create SQL schema scripts for tables, indexes and referential relations. Create data scripts so that CE data can be moved to SQL Server or other data store. User can select specific tables only. User can select Script Schema, Script Data or both.”

Name: SQLCE Code Generator
Link: http://sqlcecodegen.codeplex.com/
SQL CE Versions: 3.5, 4.0
Last updated: Current 
Platforms: Win32, Win64
Added to list: 27th March, 2011
“Contains a stand alone GUI application and a Visual Studio Custom Tool for automatically generating a .NET data access layer code with Unit tests for objects in a SQL Server Compact Edition database. Also generates a Windows Phone DataContext”

Name: OpenNETCF.ORM Framework
Link: http://orm.codeplex.com/
SQL CE Versions: 3.5
Last updated: Current 
Platforms: WinCE, Win32, Win64
Added to list: 27th March, 2011
“The OpenNETCF.ORM library is a lightweight object-relational mapping framework designed for use with the .NET Compact Framework.The framework is extensible to allow any type of backing store. An initial implementation using SQL Server Compact is provided.”

Name: SQL Metal Open Source User Interface
Link: http://sqlmetalosui.codeplex.com/
SQL CE Versions: 3.5
Last updated: 2008 
Platforms: Win32
“This project provides an user interface for SqlMetal.exe tool.
SQLMetalOSUI allows users to generate code and mapping files for their SQL Server Compact Edition databases. SQL Server Compact Edition 3.5 is fully compatible with LINQ to SQL but the designer does not. This project will be focused to reach this initial purpose.”

Name: MDB2SDF
Link: http://www.aroooy.com/mdb2sdf/
SQL CE Versions: 3.0
Last updated: 2006
Platforms: N/A
”A web page that allows you to upload an Microsoft Access MDB file and converts it to a SQL Compact 3.0 file”

It may sometimes by useful to be able to see the SQL statements generated by an ORM, in order to understand better what is going on, and why things perform the way they do. With LINQ to SQL on desktop, you can assign any TextWriter object to the DataContext Log property, and this will allow you to log to a file or the console.

On Windows Phone, this gets a little bit complicated, as files must be stored in Isolated Storage, making them hard to get at, and there is no Console. This MSDN blog includes a TextWriter implementation, that writes to the console of a attached debugger, for example the Visual Studio Debug output Window. This class (which I have renamed to DebugWriter) will be part of the DataContext generated by the next SQL Server Compact Toolbox release. So now you can add the following while debugging your LINQ to SQL statements:

db.Log = new DebugWriter();

We can now try some various LINQ queries against our database to find out how they have been implemented in SQL (I am using the excellent Chinook sample database, available for download here):

db.Album.Where(al => al.AlbumId == 10).SingleOrDefault();

And the output in the Debug output window:

db.Album.FirstOrDefault(); 

Becomes:

SELECT TOP (1) [t0].[AlbumId], [t0].[Title], [t0].[ArtistId]
FROM [Album] AS [t0]

db.Album.Where(al => al.Title.StartsWith("For those%")).FirstOrDefault();

Becomes:

SELECT TOP (1) [t0].[AlbumId], [t0].[Title], [t0].[ArtistId]
FROM [Album] AS [t0]
WHERE [t0].[Title] LIKE @p0

db.Album.Skip(50).Take(25).ToList();

Becomes:

SELECT [t0].[AlbumId], [t0].[Title], [t0].[ArtistId]
FROM [Album] AS [t0]
ORDER BY [t0].[AlbumId] OFFSET @p0 ROWS FETCH NEXT @p1 ROWS ONLY

Happy SQL debugging!

You could imagine a scenario, where one of the tables in your local database need to be periodically updated. For Windows Phone Local Database, there is no support for Sync Framework or similar technologies, so you may have to simply add a bunch of rows to your table, based on data retrieved from a web service. This blog post explores how SQL Server Compact on the device does INSERTs, and how batch INSERTs can be tweaked for optimum performance.

Let’s start by using the DebugWriter class from my previous post, and examine what INSERT statements look like. For this test, I have included a copy of the Chinook database with the phone application as an embedded resource, so the test table already contains 275 rows. Then use this code to view INSERT statements.

private void button1_Click(object sender, RoutedEventArgs e)
{
    CleanDatabase();
    using (ChinookContext db = new ChinookContext(ChinookContext.ConnectionString))
    {
        db.LogDebug = true;

        var artist = new Artist();
        artist.Name = Guid.NewGuid().ToString();

        db.Artist.InsertOnSubmit(artist);
        db.SubmitChanges();
    }
}

private static void CleanDatabase()
{
    using (var store = IsolatedStorageFile.GetUserStoreForApplication())
    {
        store.Remove();
    }

    using (ChinookContext db = new ChinookContext(ChinookContext.ConnectionString))
    {
        db.CreateIfNotExists();
    }
}

This results – surprisingly – in the following output:

-- CURSOR BASED INSERT [Artist]
-- [Name] <= [2807fb28-49d7-4a25-8350-e4e40245934a]

-- AUTOSYNC [ArtistId] <= [276]

This indicates that in order to perform INSERTs to the Local Database, the query processor is bypassed (no “INSERT” SQL statement is shown). Instead, the LINQ to SQL implementation on the device uses the SqlCeResultSet and SqlCeUpdateableRecord classes (or equivalent), that allows you to access the database tables without the overhead of the query processor, similar to what I use in the SqlCeBulkCopy library for performing fast INSERTs in SQL Server Compact databases on the desktop. So for INSERTs, the device implementation is already optimized in that respect.

Let’s then look at the speed of “many” INSERTs, in this case 10000. The following code implements this test:

CleanDatabase();

using (ChinookContext db = new ChinookContext(ChinookContext.ConnectionString))
{
    List<Artist> artistList = new List<Artist>();

    for (int i = 0; i < 10000; i++)
    {
        var artist = new Artist();
        artist.Name = Guid.NewGuid().ToString();

        artistList.Add(artist);
    }

    db.Artist.InsertAllOnSubmit(artistList);
    System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch();
    sw.Start();
    db.SubmitChanges();
    sw.Stop();
    System.Diagnostics.Debug.WriteLine(sw.ElapsedMilliseconds);            
}

On my emulator, this takes about 2500 ms for each run. So 2.5 seconds to insert 10.000 rows – no too bad. Let’s see if there are any knobs available in the connection string parameters that will allow us to tweak this. The connection string parameters available for Windows Phone Local Database are documented here.

Inspired by this article by Joao Paulo Figueira, we could try to increase the Max Buffer Size from it’s default value of 384 K to 1024 K.

This can be done in the connection string like this:

using (ChinookContext db = new ChinookContext(ChinookContext.ConnectionString + ";max buffer size=1024"))

And lo and behold, the same process of inserting 10000 rows now takes 1.6 seconds on my emulator, a 36% improvement! (Your mileage may vary, of course).

Note: Increasing above 1024 buffer size had no positive effect.

Hope you found this useful.

Speaking of Data on Windows Phone, I will be reviewing the Windows Phone 7.5 Data Cookbook soon, you can download a sample chapter on Isolated Storage from here.

Josué Yeray Julián, (blog in Spanish here), has done some interesting tests on INSERT performance, both on the emulator (like I started doing in my previous blog post) and on two real devices, an HTC Mazaa and Nokia Lumia 800. (Sadly, I do not own a real device – yet!)

In summary,  the following parameters can improve performance for doing INSERTs of 500.000 rows:

1: Set “Max Database Size” on your connection string  to an expected size in advance (default is 32 MB, max is 512 MB on Windows Phone)

2: Increase “Max Buffer Size” to a value higher than the default of 384 KB. On the tested real devices the max of 4096 worked best.

3: Run the INSERT process in a background thread.

4: Remove any Version columns (if only INSERTSs matter, not UPDATEs)

5: Use InsertAllOnSubmit instead of InsertOnSubmit

These tweaks reduced the total processing time on the Nokia Lumia 800 from 29:20 to 16:04!

But - keep in mind that you can use Database first development with the SQL Server Compact Toolbox add-in and then include the database with your application package. You can also use the SqlCeBlukCopy API to insert 1.000.000 rows (twice as many as tested) in 6 seconds on the desktop!

You can read the Microsoft Translator versions of his 2 blog posts here and here.

My SSMS and Visual Studio add-ins for SQL Server Compact includes the option to import CSV files. But this GUI based feature does not work so well with larger CSV files, as the generated INSERT script can become very large. So I have now made available a command line utility that performs the same functionality, but works better with larger CSV files (and allows you to automate the process).

Please give the utility a try, and provide feedback here.

After more than 80.000 downloads, version 2.6 of my SQL Server Compact Toolbox extension for Visual Studio 2010 is now available for download. This blog post is a visual guide to the new features included in this release, many suggested by users of the tool via the CodePlex issue tracker

Edit Table Data

This feature, which allows you to edit the table data in a grid, and allows column sorting, image import/export and quick search, has been enhanced with the following new features:

The grid now only loads the first 200 rows, to better handle large tables. The number of rows initially loaded can be configured via the Options dialog:

In addition, you can use any SELECT statement as the basis for the grid, by entering this statement using the SQL button at the bottom of the grid:

Pressing this button will give you access to enter any SQL SELECT statement:

Windows Phone DataContext

The Windows Phone DataContext feature allows you generate a DataContext for use in Windows Phone projects, based on an existing database, saving you a large coding effort. This feature has been enhanced as follows:

You can now generate a DataContext in VB code, in addition to C# code:

The generated DataContext contains to new useful properties:

using (PostCodesContext db = new PostCodesContext(PostCodesContext.ConnectionStringReadOnly))
{
    if (System.Diagnostics.Debugger.IsAttached)
    {
        db.LogDebug = true;
    }
    _allItems = db.PostCode.ToList();
}

ConnectionStringReadOnly allows you to include a database for reference data, and using the appdata:// syntax on the connection string, this database can reside in your programs folder, no need to copy to Isolated Storage.

LogDebug allows you to log any SQL statement generated to the Visual Studio debug output window, as described in my blog post here.

Explorer Tree

The Tables node now has a Refresh menu item, allowing you to refresh only objects in a single database:

In addition, when running ALTER, DROP and CREATE scripts, a refresh will be attempted.

Object Descriptions

You can now edit all the descriptions for a table in a single window, thanks to new contributor giddy:

Other minor improvements

All Dialogs cleaned up for VS UI compliance ( Thanks to new contributor giddy )


Stopped handling unhandled exceptions, had undesired side effects


Warning when Primary Keys were missing when creating DataContext and EDMX


Using a "nicer" propertygrid

See this overview of the work items in this release.

As you may know, you can generate a Windows Phone DataContext based on an existing database for use with Windows Phone Mango “Local Database”, as described here and here. But this feature is only available if you own a paid Visual Studio edition (Pro or higher), as the Express editions do not support add-ins.

But users of Visual Studio 2010 Express for Windows Phone can now also take advantage of all the productivity benefits of the “Database First” approach to Local Database development on Windows Phone.

Provided you have a SQL Server Compact Database ready, designed according to the guidelines here, and possibly created by using a tool, you can follow this simple procedure to generate and add the DataContext to your Windows Phone solution.

Anyone using the Express tools will notice that SQLMetal (which is used behind the scenes to create the DataContext file) is not included. You can get it from the Microsoft SDK bundle http://www.microsoft.com/download/en/details.aspx?id=8279. You only need to choose the following options to get the required files.

Then go to the download page for my ExportSqlCe tool, and download the latest version of the Exportsqlce.exe command line utility.

You can now generate a DataContext file from the command line, as documented here.

In this example, I will generate a DataContext based on the Chinook sample database.

exportsqlce wpdc "Data Source=C:\projects\Chinook\Chinook.sdf" C:\temp\Chinook.cs

Notice that the format of the database must be SQL Server Compact 3.5.

In Visual Studio, open your Windows Mango project, and from Solution Explorer, select Add, Existing Item, and browse to the file the was just created.

Remember to add a reference to System.Data.Linq to your project.

You can now start using the generated DataContext in your project!

using (ChinookContext db = new ChinookContext(ChinookContext.ConnectionString))
{
    db.LogDebug = true;

    var artist = db.Artists.FirstOrDefault();
    artist.Name = Guid.NewGuid().ToString();

    db.SubmitChanges();
}

Happy coding…

There are several options for initializing the database for Window Phone, depending on your approach. By “initializing” I mean preparing the database for first use, just after the application has ben installed.

Whether you create the DataContext by hand or use RAD “Database First” with the SQL Server Compact Toolbox or the ExportSqlCe command line utility is not relevant.

You can read more about Database First here: http://erikej.blogspot.com/2012/01/generating-linq-to-sql-datacontext-with.html and more about Code First here: http://msdn.microsoft.com/en-us/library/hh202876(v=VS.92).aspx

I will assume you have used one of my tools to create the DataContext classes.

Option 1: No database file included

In this case, there is no initial data in the local database, it will be populated either by getting data from the web, or by manual entry by the user. The database must be writable, and must therefore reside in Isolated Storage. The connection string look like this: "Data Source=isostore:/Chinook.sdf"

To create the database based on the definition in the DataContext classes, you can use the following code, for example for each database call, or during app start:

using (ChinookContext db = new ChinookContext(ChinookContext.ConnectionString))
{
    if (!db.DatabaseExists())
        db.CreateDatabase();
}
OR:

using (ChinookContext db = new ChinookContext(ChinookContext.ConnectionString))
{
    db.CreateIfNotExists();
}

Option 2: Database file included, read and write

With this option, you include a database (.sdf file) in your project as en embedded resource:

You database must reside in Isolated Storage to be writable, and it must be copied to Isolated Storage as required. The connection string looks like this: "Data Source=isostore:/Chinook.sdf"

To copy the database to Isolated Storage, you can use the following code, for example for each database call, or during app start:

using (ChinookContext db = new ChinookContext(ChinookContext.ConnectionString))
{
    db.CreateIfNotExists();
}

Option 3: Database file included, read only

With this option, you use the database for reference data only, and it can reside in the program files folder as a read-only resource. You include the database (.sdf file) as Content in your project:

You do not need to copy the database anywhere, it is installed with the other content in your application. The connection string looks like this: "Data Source=appdata:/Chinook.sdf;File Mode=Read Only;"

To connect to the database, use the following code:

using (ChinookContext db = new ChinookContext(ChinookContext.ConnectionStringReadOnly))
{
    db.LogDebug = true;
    // More data access here...
}

The “Windows Phone 7.5 Data Cookbook” is a collection of more that 30 so-called recipes  for working with with data in Windows Phone 7.5 applications. It covers all data related tasks from UI data binding and MVVM to using WCF Data Services.

A recipe is a step-by-step walkthrough of a sample mini application, that covers the subject at hand. All the samples are available for download from the publishers web site for registered users.

The book covers the following topics:
- Data binding, including element binding, DataContext, data templates, converters.
- Isolated storage
- Using XML files
- Consuming OData, and performing CRUD against OData sources
- Using on-device databases, including Perst, SQLite and SQL Server Compact with LINQ to SQL
- Consuming REST feeds, and creating a REST service
- WCF Data Services
- MVVM

The hands-on approach is useful for any developer new to the Windows Phone platform, and can also be used a samples for seasoned developers in need of a quick solution to a problem.

You can also download a sample chapter of the book, Chapter 2 – Isolated Storage to get a feel for the format.

It seems that the recent Cumulative Update 4 to SQL Server 2008 R2 contains a couple of essential hotfixes, if you are using Merge Replication with SQL Server Compact 3.5 against a SQL Server 2008 R2 instance.

The 2 hotfixes are:

FIX: Poor performance when you synchronize the data from a SQL Server 2008 R2 publisher to a SQL Server compact 3.5 subscriber 2616718

FIX: Merge replication changes are missing on a SQL Server Compact subscriber when they subscribe to a partition in SQL Server 2008 R2 2644396

As always, test before applying this Update in production.

According to the Local Database best practices, it is recommended to use a rowversion column in your local database table, if you are performing many updates. This blog post will examine the performance and changes to UPDATE handling that are caused by the presence of a rowversion column.

But first, how to add a rovversion column: If you are using code first, this simple column declaration will suffice:
[Column(IsVersion=true)]
private Binary _version;

If you are using database first, simply add a rowversion column like so:
ALTER TABLE [Artist]
ADD [Version] rowversion NOT NULL

Let us run the following test code which updates 100 rows with and without a rowversion column. I am using a prepopulated Chinook database, added as en embedded resource to the project. See this blog post for details on how to do this.

//New database from embedded resource
            using (ChinookContext db = new ChinookContext(ChinookContext.ConnectionString))
            {
                if (db.DatabaseExists())
                    db.DeleteDatabase();

                db.CreateIfNotExists();
            }

            using (ChinookContext db = new ChinookContext(ChinookContext.ConnectionString))
            {

                try
                {
                    db.LogDebug = true;

                    var arists = db.Artist.Take(100);

                    foreach (var artist in arists)
                    {
                        artist.Name = Guid.NewGuid().ToString();
                    }

                    System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch();
                    sw.Start();

                    db.SubmitChanges();

                    sw.Stop();
                    System.Diagnostics.Debug.WriteLine(sw.ElapsedMilliseconds);
                }
                catch (Exception ex)
                {
                    System.Diagnostics.Debug.WriteLine(ex.ToString());
                }
            }

Now lets us see what the SQL statements executed look like without a rowversion column on the table:




UPDATE [Artist]


SET [Name] = @p2


WHERE ([ArtistId] = @p0) AND ([Name] = @p1)

As you can see, a standard UPDATE statement is executed, including a WHERE clause referring to all columns in the table, in order to trap concurrency exceptions. The time to do 100 updates is about 320 ms.

Now let us add the rowversion column to the Artist table, recreate the DataContext so it contains the Version column, and run the same code.

Now the SQL statements look like so, indicating that the query processor is being bypassed:

-- CURSOR BASED INDEX UPDATE [Artist].[PK_Artist] (

--     ArtistId: [99]


--    )


-- EQUALITY CHECK [Version], [System.Byte[]] = [System.Byte[]]


-- [Name] <= [5ad7e0d2-925e-44f7-bcd7-c3bca32de745]


-- AUTOSYNC [Version] <= [System.Byte[]]

The time to do 100 updates is now about 110 ms, a significant improvement from 320 ms. So if you expect to do UPDATEs on your local Windows Phone database, add a rowversion column to your tables!

In this post, I will try to give an overview of the free tools available for developers to move databases from SQL Server to SQL Server Compact and vice versa. I will also show how you can do this with the SQL Server Compact Toolbox (add-in and standalone editions).

Moving databases from SQL Server Compact to SQL Server

This can be useful for situations where you already have developed an application that depends on SQL Server Compact, and would like the increased power of SQL Server or would like to use some feature, that is not available on SQL Server Compact. I have an informal comparison of the two products here. Microsoft offers a GUI based tool and a command line tool to do this: WebMatrix and MsDeploy. You can also use the ExportSqlCe command line tool or the SQL Server Compact Toolbox to do this. To use the ExportSqlCE (or ExportSqlCE40) command line, use a command similar to:

ExportSQLCE.exe "Data Source=D:\Northwind.sdf;" Northwind.sql

The resulting script file (Northwind.sql) can the be run against a SQL Server database, using for example the SQL Server sqlcmd command line utility:

sqlcmd -S mySQLServer –d NorthWindSQL -i C:\Northwind.sql

To use the SQL Server Compact Toolbox:

Connect the Toolbox to the database file that you want to move to SQL Server:

Right click the database connection, and select to script Schema and Data:

Optionally, select which tables to script and click OK:

Enter the filename for the script, default extension is .sqlce:

Click OK to the confirmation message:

You can now open the generated script in Management Studio and execute it against a SQL Server database, or run it with sqlcmd as described above.

Moving databases from SQL Server to SQL Server Compact

Microsoft offers no tools for doing this “downsizing” of a SQL Server database to SQL Server Compact, and of course not all objects in a SQL Server database CAN be downsized, as only tables exists in a SQL Server Compact database, so no stored procedures, views, triggers, users, schema and so on. I have blogged about how this can be done from the command line, and you can also do this with the SQL Server Compact Toolbox (of course):

From the root node, select Script Server Data and Schema:

Follow a procedure like the one above, but connecting to a SQL Server database instead.

The export process will convert the SQL Server data types to a matching SQL Server Compact data type, for example varchar(50) becomes nvarchar(50) and so on. Any unsupported data types will be ignored, this includes for example computed columns and sql_variant. The new date types in SQL Server 2008+, like date, time, datetime2 will be converted to nvarchar based data types, as only datetime is supported in SQL Server Compact. A full list of the SQL Server Compact data types is available here.

(Nice short name!) SQL Server Compact 4.0 SP1 CTP1 is now available for download: http://www.microsoft.com/download/en/details.aspx?id=29037 – is is also installed with Visual Studio 11 Ultimate beta. Installing this .MSI will replace your current SQL Server Compact 4.0 binaries in Program Files, and change the version to 4.0.8854.1.

No blog post or KB article yet, I will update this post when they become available. I the meantime, I have skimmed though the code in the ADO.NET provider, which reveals the following changes (all is just guesses, and it is beta software, of course):

New connection string keyword: “File Access Retry Timeout” – will reattempt connection open from 0 – 30 seconds, with a default value of 0.

Improved local C++ runtime verification.

SqlCeConnection.GetSchema bug fixes

SqlCeParameter.Add(String parameterName, Object value) has been deprecated.  Use AddWithValue(String parameterName, Object value) instead.

According to this thread, this bug has been fixed: Getting exception "The data area passed to a system call is too small." in SQL Server Compact 4

SQL Server Compact on Windows Phone supports encryption of the database file. By specifying a password on the ConnectionString during encryption and subsequent openings, the contents of the database file will be encrypted.
Beware that the encryption will degrade database performance, due to the extra processing required to decrypt and encrypt data.
Also keep in mind when using encryption, that unless you protect your connection string (the password), anyone that knows it can open the database file.

Some Phone developers (see this forum post, for example) have encountered issues when moving databases from device to desktop and vice versa, when encryption is involved. In this blog post I will try to explore what can and cannot be done in this respect.

From the root context menu in the SQL Server Compact Toolbox, select “Add 3.5 connection” and select Create:

Specify a password of “zyx” and click OK. Now we have a connection to a desktop encrypted database file.

Open the SQL Editor, and create a basic table:

CREATE Table Test
(
Id int PRIMARY KEY,
Data nvarchar(10)
);

Now create a Windows Phone 7.1 Application, and add the database file from above as Content:

Right click the database connection in the SQL Server Compact Toolbox, and select “Add Windows Phone DataContext…” and click OK, the WP7EncryptContext.cs file has now been added to the project.

Add a reference to System.Data.Linq, and compile the project.

Add a button called “Connect”, and add a click event handler, with this code:

            using (WP7EncryptContext db = new WP7EncryptContext(WP7EncryptContext.ConnectionStringReadOnly + ";Password=zyx"))
            {
                var list = db.Test.ToList();
            }

Now run the app in the emulator by pressing F5 (debug):

Obviously the desktop encryption is not compatible with the device database engine!

Now remove the database file from the project, and create the database file in isolated storage – change the click event handler code:

using (WP7EncryptContext db = new WP7EncryptContext(WP7EncryptContext.ConnectionString + ";Password=zyx"))
{
    db.CreateIfNotExists();
    var list = db.Test.ToList();
    var test = new Test();
    test.Id = 2;
    test.Data = "ErikEJ";
    db.Test.InsertOnSubmit(test);
    db.SubmitChanges();

}

Now the database will be created in Isolated Storage on the device. Run with F5 again, and no errors will occur. And we have “pre-populated” the database with a single row. Let’s try to move this database file to the desktop, I use Windows Phone Power Tools from CodePlex to do this:

When you add a connection to the file form the device, and click “Test Connection”, you will get this error message:

We already know that some version 4.0 features have been made available in the 3.5 Windows Phone database engine, like support for OFFSET/FETCH to allow paging. Has the enhanced version 4.0 encryption also been move to the device?

In the Toolbox, select “Add 4.0 Connection” and try to connect to the database from the device:

(Remember to specify the password!)

Reply Yes! And now we can browse and inspect both schema and data in the file – keep in mind that this file cannot be moved back to the device.

Now, GET the file again from the device via the Power Tools, and add the new copy to the project again, also as Content (as above, but the database file comes “pre-populated” from the device.)

Again, change the button click event handler as follows:

using (WP7EncryptContext db = new WP7EncryptContext(WP7EncryptContext.ConnectionStringReadOnly + ";Password=zyx"))
{
    var list = db.Test.ToList();
    if (list.Count > 0)
        MessageBox.Show("It works " + list[0].Data);

}

Conclusions:

You CANNOT:

- Use a desktop encrypted database on the device
- Open a device encrypted database on the desktop

You CAN:

- Prepopulate an encrypted database (but only on a device)
- Convert a device encrypted database to 4.0 format on the desktop, and browse it.

Credits: Thanks to Marc-Anthon Flohr for prompting me to write this blog post. His overview is nice, although his conclusions are slightly different.

My SQL Server Compact Toolbox add-in for Visual Studio 2010 and 11 beta has now had more than 100.000 downloads. (Actually, more than 105.000, but who’s counting.)

Thanks to all the users of this tool, and your continous feedback. I always think that the most recent release will be the last, but I am already perparing for the next relase, which will add features like:

- Provision a 3.5 database for Sync Framework
- Ability to Migrate to a SQL Server/Express/LocalDB database in a single click (like what you can do with WebMatrix, but from within Visual Studio)
- Support for SQL Server Compact 3.5 connections with Visual Studio 11 beta (to support Windows Phone users and others)
- Option to add a ConnectionStringBuilder class to Windows Phone projects
- When exporting table data, the tables are now ordered topologically
- The Server based DGML files now respect schemas
- Server date and datetime2 columns are now converted to datetime

And in connection with the MVP Summit I got the opportunity to appear on the Visual Studio Toolbox show on Channel 9, where I demo some of the Toolbox features. Maybe you will discover features you did not know about during the demo.

In the previous instalments in this series of Windows Phone Local Database tips, I have been exploring SELECT, INSERT, UPDATE, CREATE DATABASE and Encryption. Now the time has come to have a closer look at DELETE, and investigate how they can be optimized (if possible). I will look at deleting multiple rows in a single call to SubmitChanges.

To test this, I will be using the code below, and as usual use the Chinook sample database.

//New database from embedded resource
using (ChinookContext db = new ChinookContext(ChinookContext.ConnectionString))
{
    if (db.DatabaseExists())
        db.DeleteDatabase();

    db.CreateIfNotExists();
}

using (ChinookContext db = new ChinookContext(ChinookContext.ConnectionString))
{

    try
    {
        //db.LogDebug = true;

        var list = db.InvoiceLine.Take(100);

        db.InvoiceLine.DeleteAllOnSubmit(list);

        System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch();
        sw.Start();

        db.SubmitChanges();

        sw.Stop();
        System.Diagnostics.Debug.WriteLine(sw.ElapsedMilliseconds);
    }
    catch (Exception ex)
    {
        System.Diagnostics.Debug.WriteLine(ex.ToString());
    }
}

Running this code using the InvoiceLines table from the Chinook sample database will result in 100 calls to a DELETE statement like this:

DELETE FROM [InvoiceLine] WHERE ([InvoiceLineId] = @p0) AND ([InvoiceId] = @p1) AND ([TrackId] = @p2) AND ([UnitPrice] = @p3) AND ([Quantity] = @p4)
-- @p0: Input Int32 (Size = 4; Prec = 0; Scale = 0) [100]
-- @p1: Input Int32 (Size = 4; Prec = 0; Scale = 0) [19]
-- @p2: Input Int32 (Size = 4; Prec = 0; Scale = 0) [581]
-- @p3: Input Decimal (Size = 19; Prec = 10; Scale = 2) [0.99]
-- @p4: Input Int32 (Size = 4; Prec = 0; Scale = 0) [1]

Notice that all columns are included in the WHERE statement, to check for concurrency conflicts (you can argue if this is really required when deleting). The 100 deletes runs in 320 ms on the emulator.

Now let us try to add a rowversion column to the InvoiceLines table, like we did for the UPDATE testing, and see if this has any effect.

You can also use the new feature in the SQL Server Compact Toolbox, which will allow you to add rowversion columns to all your tables, for the benefit of UPDATE and maybe also DELETE statements.

Let us try to run the same test code again.

Now the SQL statement looks like this, indicating that the query processor is bypassed, unlike in the statements above.

-- CURSOR BASED INDEX DELETE [InvoiceLine].[PK_InvoiceLine] (
--     InvoiceLineId: [1]
--    )
-- EQUALITY CHECK [versioncolumn], [System.Byte[]] = [System.Byte[]]

The time to perform the 100 DELETE statements is now down from 320 ms to under 70 ms, a significant improvement.

BUG ALERT!

If you attempt to take advantage of rowversion columns in an existing database, make sure that the table in question does not have more indexes that cover the primary key columns. If you have that, your SubmitChanges statement will “bomb out” your app.

For example the Album table in Chinook has a PK_Album index and PFK_Album index, that both are indexes on the AlbumId column. DROP the PFK_Album index in this case (you can use the new Index context menu option in the Toolbox to do that).

Hope you found this useful.