Living in the Tech Avalanche Generation

Recently I spent a fair bit of time poking around the Entity Framework in Reflector (the reason for which will be apparent soon) and whilst I was poking around ObjectQuery, ObjectQuery<T>, ObjectContext and a host of other objects in the framework’s API, I was immediately reminded of the significance of the effort  that had gone into providing this tool. I have voiced my opinions in the past when it came to things I felt were lacking in the design aspects of the framework, particularly as they related to the style of working that the current version imposed on me. However, that discussion is based on a ’style’ of working that I wish to apply and not the general purpose application for which the framework was intended. There is nothing stopping me from using NHibernate or any other ORM if I choose to and it’s true that I went to the extent of developing my own lite framework to enable me in pursuing this preferred ’style’ of work with the Entity Framework. The ’style’ to which I refer is DDD and perhaps it’s true that often I don’t practice it with a purist’s heart when it comes to the Entity Framework, however it should be noted that users of this style are most likely in a minority. I don’t recall an outcry when ADO.Net 2.0 was released? Sure, ADO.Net never purported to being an ORM and that’s ok I get the distinction but who said an ORM should be built to keep the practitioners of DDD happy anyway?

As I said, I have pointed out in the past some areas I felt were lacking in the design of the Entity Framework however every story requires to be told in balance, with the good and bad and it should be stated equally when there is praise to be made. I find the Entity Framework an excellent Data Access Technology in general and I hope (and indeed believe) that those like me, who wish to develop in styles that were perhaps not foremost in the EF teams minds initially, will be gradually accommodated in future versions. For example, transparent lazy loading will be available in .NET 4.0 and will be managed by a binary switch in the ObjectContext. Until such time there are other options.

What started all this?

So I got a bit off topic, lets focus on why I was digging around the EF in Reflector. Recently, Ayende’s NHibernate Profiler tool had me wishing for something similar’ish in the Entity Framework space. So to understand better how that might be achieved I began looking deeper into the Entity Framework ‘goo’ to see how feasible, an albeit much more lite weight approach than Ayende’s, might be possible. One initial limitation i chose to impose at the start was that the  profiler will not be built (by me) using Microsoft’s profiling API, unless someone can point out to me a completely managed option.

Motivation

It is always a concern that developers who choose to enlist the services of LINQ To Entities or LINQ To SQL might not vigilantly monitor the SQL that those implementations are producing. Developers should never rest on their laurels and become satisfied that their ORM ensures that everything is working ok, as long as the data returned is as expected then there are no problems, we should never assume that our performance is optimal because the ORM will make it so. Furthermore if your using LINQ To Entities, it’s not impossible that what looks to make sense in your LINQ C#/VB code is not making sense as SQL. I recently described a way of working with LinqPad that will allow you to write queries in both the Entity Framework and LINQ To SQL however it will only produce the TSQL in it’s viewer when using LINQ To SQL. Some other tools / methods on offer to monitor provider specific generated SQL are the following:

• LINQ To SQL
  • DebuggerWriter by Chris Vandermotten
  • LINQ To SQL Debug Visualizer.
• Entity Framework
  • Julie Lerman’s Logging Class

And so, to help myself in this very endeavour I set out to build an Entity Framework Profiler to try to address the dislocated nature of having to switch between viewing output from .ToTraceString(), SQL Profiler and LINQPAD.

How does it work?

Using WCF the Profiler listens for messages at the localhost (loopback) address. The messages handled by it’s service, contain meta data and the SQL query text sent to the database by the Entity Framework. So how does the Entity Framework know how to send messages? ASPECT oriented code weaving is what creates this illusion for us. I had previously used PostSharp to help develop a framework that unifies the both Lazy and Eager loading, provides Specifications for dynamic querying and includes a generic repository. After having that positive experience with PostSharp It was clear that it would again be useful in the design for the Profiler. The profiler relies on Entity Framework consumers to use a custom PostSharp attribute to weave code into the ObjectQuery<T> implementations in their Entity Models (.edmx files). Here listed are the more pertinent parts of the code found in the EntityFrameworkProfilingAttribute class:

private EntityInterceptMessage BuildMessageFromMetaData
    (MethodExecutionEventArgs eventArgs,
     ref TimeSpan time_taken, ObjectQuery q, ObjectContext ctx)
{
    var query =
        from meta in ctx.MetadataWorkspace.
            GetEntityContainer(ctx.DefaultContainerName,
            DataSpace.CSpace).BaseEntitySets
        let fields = from f in meta.ElementType.Members.ToList()
                     select new
                     {
                         FieldName = f.Name,
                         FieldType = f.TypeUsage.EdmType.Name
                     }
        where meta.BuiltInTypeKind == BuiltInTypeKind.EntitySet &&
              meta.Name == eventArgs.Method.Name.Replace(“get_”, “”)
        select new
        {
            EntitySetName = meta.Name,
            EntityName = meta.ElementType.Name,
            Members = fields
        };

    EntityInterceptMessage entity_message =
        new EntityInterceptMessage()
        {
            SqlQueryText = q.ToTraceString(),
            ProcessID = AppDomain.CurrentDomain.Id.ToString(),
            ThreadID = Thread.CurrentThread.ManagedThreadId.ToString(),
            EntityModelName = ctx.DefaultContainerName,
            Succeeded = eventArgs.Exception == null ? true : false,
            ExecutionTime = string.Format(“{0} Ms {1} Ticks”,
                time_taken.TotalMilliseconds,
                time_taken.Ticks),
            MetaData = new EntityInterceptMessage.EntityInterceptMeta()
            {
                ContextName = ctx.DefaultContainerName,
                EntitySetName = query.First().EntitySetName,
                EntityTypeName = query.First().EntityName,
                EntityTypeFields = (from f in query.First().Members
                        select new
                               EntityInterceptMessage.EntityFieldNameTypePair()
                        {
                            FieldName = f.FieldName,
                            FieldTypeName = f.FieldType
                        }).ToList()
            }
        };
    return entity_message;
}

The Service Code:

private void CreateAndSendMessageToProfiler
        (EntityInterceptMessage entity_message)
{
    EndpointAddress address =
        new EndpointAddress
            (“net.tcp://localhost:9002/EntityProfilerSvc”);

    var binding =
        new NetTcpBinding { SendTimeout = new TimeSpan(0, 10, 0) };

    ChannelFactory<IProfilerService> factory =
        new ChannelFactory<IProfilerService>(binding, address);

    var channel = factory.CreateChannel();

    XmlReader reader = MessageSerializer.SerializeMessage(entity_message);

    //Create the Message with the serialized data 
    //embedded into its body
    var msg = chans.Message.CreateMessage(
        binding.MessageVersion,
        “urn:IProfilerService-Send”, reader);

    channel.Send(msg);

    factory.Close();
}

If you dig around in Reflector you can see the code woven into by PostSharp.

The biggest problem I have with this approach is that Entity SQL is not catered for. I would have preferred to use a true Profiling API like the Microsoft one, however the requirement in using unmanaged code left me to bypass that option for now. I very well might re-work the entire application using that API when I get more time to look more deeply into it, but for now it was more important to me to get an alternative going quickly.

What information does the Profiler provide?

The single most important data to report on was always going to be the SQL Query text created by an ObjectQuery<T>. Along with this I have packaged up the execution time for the query (from the managed codes perspective not the database), the fields involved in the query and the process and thread ID’s that execute each query.

What’s Next?

The next thing to add is definitely going to be persistence to disk, making it possible to review query data from previous sessions. Once that’s in place I am going to build an IronRuby console into the application so you can write REPL scripts against your Entity Framework Entity Models and profile the results right there in the profiler – instant Entity Framework gratification! And finally to create a Visual Studio template for the EntityFrameworkProfilingAttribute class so it’s easy to add it to your own libraries that include Entity Models. The code box for the SQL Text is licensed under the code project open license and usage of the Profiler requires accepting this license. As I have said all along, suggestions are welcome. To see how this all works, you can get the code here and just run the tests and you will see the messages appear in the Profiler.

Personally I have a pretty rapid fire kind of scanning action that allows me to try to find information that matches my requirement quickly. Sometimes I need to slow down when the topic is too deep, for example Databinding in WPF. The examples out on the web for Databinding WPF are not that great yet (IMO) and too many only touch the simplest kind of Databinding scenarios, focussing largely on how the data is structured and from a Models point of view (no pun intended). Models can get pretty deep and nested, yet for the most part the binding examples focus on the really trivial data scenarios where the depth is not considerable and furthermore often binding to static resources rather than data that came from the Database courtesy of an ORM or perhaps Astoria or even plain old ADO.NET. So enough complaining cause that wont change anything and like Roy said ‘what are you going to do about it’ ; time to get on with putting some examples together myself! Of course they will end up here when they are done.

In my last post I spoke about a Profiler for the Entity Framework that I have been working on as a proof of concept and ultimately as a tool for personal use and by extension, anyone else who can find a use for it.

Previously I posted a single image as a preview of the User Interface. At that point in time the UI was particularly empty and didn’t really outline or describe the data that the Profiler would provide the user. In an effort to build some anticipation, I thought it might be useful to give one last look at the application before I post the code in the next few days at which point I expect it will be useful and easily setup to run for an end user.

Meta View (without expanded data)

This image above shows some data relating to an Entity Model that has executed queries. The ‘Meta’ tab offers some context regarding the underlying ObjectContext EntitySet and it’s type, it also contains the SQL executed and the field meta data, as is apparent when the expanders are collapsed as per the next image below. The meta data is derived directly out of the MetaDataWorkspace in the Entity Framework itself. Here is a snippet of the code:

var query =
    from meta in ctx.MetadataWorkspace.
        GetEntityContainer(ctx.DefaultContainerName,
        DataSpace.CSpace).BaseEntitySets
    let fields = from f in meta.ElementType.Members.ToList()
                 select new
                 {
                     FieldName = f.Name,
                     FieldType = f.TypeUsage.EdmType.Name
                 }
    where meta.BuiltInTypeKind == BuiltInTypeKind.EntitySet &&
          meta.Name == eventArgs.Method.Name.Replace(“get_”, “”)
    select new
    {
        EntitySetName = meta.Name,
        EntityName = meta.ElementType.Name,
        Members = fields
    };

Meta View (expanded)

The next image displays a view that contains a dedicated view of the SQL that has been executed. The SQL Text window is provided by a free (slightly re-worked) third party user control (more on that later).

SQL View

Here is the code run against the Entity Model that was profiled, producing the data displayed in the images exhibited in this post.

public class EFProfilerIntegrationTests
{
    public void message_sent_with_sqltext1()
    {
        NorthwindEntities ctx =
            new NorthwindEntities();
        var custs = from c in ctx.Customers
                    select c;
    }

    public void message_sent_with_sqltext2()
    {
        NorthwindEntities ctx =
            new NorthwindEntities();
        var custs = from c in ctx.Orders
                    select c;
    }

    public void message_sent_with_sqltext3()
    {
        NorthwindEntities ctx =
            new NorthwindEntities();
        var custs = from c in ctx.OrderLines
                    select c;
    }
}

The menu currently has no functionality but I plan to use it for saving and reading back data to and from disk allowing users to execute a playback of previous sessions profiled. I expect to get a working copy of the code posted sometime over the next few days or so. Remember I am keen to hear functionality requests.