Living in the Tech Avalanche Generation

Recently I spent a fair bit of time poking around the Entity Framework in Reflector (the reason for which will be apparent soon) and whilst I was poking around ObjectQuery, ObjectQuery<T>, ObjectContext and a host of other objects in the framework’s API, I was immediately reminded of the significance of the effort  that had gone into providing this tool. I have voiced my opinions in the past when it came to things I felt were lacking in the design aspects of the framework, particularly as they related to the style of working that the current version imposed on me. However, that discussion is based on a ’style’ of working that I wish to apply and not the general purpose application for which the framework was intended. There is nothing stopping me from using NHibernate or any other ORM if I choose to and it’s true that I went to the extent of developing my own lite framework to enable me in pursuing this preferred ’style’ of work with the Entity Framework. The ’style’ to which I refer is DDD and perhaps it’s true that often I don’t practice it with a purist’s heart when it comes to the Entity Framework, however it should be noted that users of this style are most likely in a minority. I don’t recall an outcry when ADO.Net 2.0 was released? Sure, ADO.Net never purported to being an ORM and that’s ok I get the distinction but who said an ORM should be built to keep the practitioners of DDD happy anyway?

As I said, I have pointed out in the past some areas I felt were lacking in the design of the Entity Framework however every story requires to be told in balance, with the good and bad and it should be stated equally when there is praise to be made. I find the Entity Framework an excellent Data Access Technology in general and I hope (and indeed believe) that those like me, who wish to develop in styles that were perhaps not foremost in the EF teams minds initially, will be gradually accommodated in future versions. For example, transparent lazy loading will be available in .NET 4.0 and will be managed by a binary switch in the ObjectContext. Until such time there are other options.

What started all this?

So I got a bit off topic, lets focus on why I was digging around the EF in Reflector. Recently, Ayende’s NHibernate Profiler tool had me wishing for something similar’ish in the Entity Framework space. So to understand better how that might be achieved I began looking deeper into the Entity Framework ‘goo’ to see how feasible, an albeit much more lite weight approach than Ayende’s, might be possible. One initial limitation i chose to impose at the start was that the  profiler will not be built (by me) using Microsoft’s profiling API, unless someone can point out to me a completely managed option.

Motivation

It is always a concern that developers who choose to enlist the services of LINQ To Entities or LINQ To SQL might not vigilantly monitor the SQL that those implementations are producing. Developers should never rest on their laurels and become satisfied that their ORM ensures that everything is working ok, as long as the data returned is as expected then there are no problems, we should never assume that our performance is optimal because the ORM will make it so. Furthermore if your using LINQ To Entities, it’s not impossible that what looks to make sense in your LINQ C#/VB code is not making sense as SQL. I recently described a way of working with LinqPad that will allow you to write queries in both the Entity Framework and LINQ To SQL however it will only produce the TSQL in it’s viewer when using LINQ To SQL. Some other tools / methods on offer to monitor provider specific generated SQL are the following:

• LINQ To SQL
  • DebuggerWriter by Chris Vandermotten
  • LINQ To SQL Debug Visualizer.
• Entity Framework
  • Julie Lerman’s Logging Class

And so, to help myself in this very endeavour I set out to build an Entity Framework Profiler to try to address the dislocated nature of having to switch between viewing output from .ToTraceString(), SQL Profiler and LINQPAD.

How does it work?

Using WCF the Profiler listens for messages at the localhost (loopback) address. The messages handled by it’s service, contain meta data and the SQL query text sent to the database by the Entity Framework. So how does the Entity Framework know how to send messages? ASPECT oriented code weaving is what creates this illusion for us. I had previously used PostSharp to help develop a framework that unifies the both Lazy and Eager loading, provides Specifications for dynamic querying and includes a generic repository. After having that positive experience with PostSharp It was clear that it would again be useful in the design for the Profiler. The profiler relies on Entity Framework consumers to use a custom PostSharp attribute to weave code into the ObjectQuery<T> implementations in their Entity Models (.edmx files). Here listed are the more pertinent parts of the code found in the EntityFrameworkProfilingAttribute class:

private EntityInterceptMessage BuildMessageFromMetaData
    (MethodExecutionEventArgs eventArgs,
     ref TimeSpan time_taken, ObjectQuery q, ObjectContext ctx)
{
    var query =
        from meta in ctx.MetadataWorkspace.
            GetEntityContainer(ctx.DefaultContainerName,
            DataSpace.CSpace).BaseEntitySets
        let fields = from f in meta.ElementType.Members.ToList()
                     select new
                     {
                         FieldName = f.Name,
                         FieldType = f.TypeUsage.EdmType.Name
                     }
        where meta.BuiltInTypeKind == BuiltInTypeKind.EntitySet &&
              meta.Name == eventArgs.Method.Name.Replace(“get_”, “”)
        select new
        {
            EntitySetName = meta.Name,
            EntityName = meta.ElementType.Name,
            Members = fields
        };

    EntityInterceptMessage entity_message =
        new EntityInterceptMessage()
        {
            SqlQueryText = q.ToTraceString(),
            ProcessID = AppDomain.CurrentDomain.Id.ToString(),
            ThreadID = Thread.CurrentThread.ManagedThreadId.ToString(),
            EntityModelName = ctx.DefaultContainerName,
            Succeeded = eventArgs.Exception == null ? true : false,
            ExecutionTime = string.Format(“{0} Ms {1} Ticks”,
                time_taken.TotalMilliseconds,
                time_taken.Ticks),
            MetaData = new EntityInterceptMessage.EntityInterceptMeta()
            {
                ContextName = ctx.DefaultContainerName,
                EntitySetName = query.First().EntitySetName,
                EntityTypeName = query.First().EntityName,
                EntityTypeFields = (from f in query.First().Members
                        select new
                               EntityInterceptMessage.EntityFieldNameTypePair()
                        {
                            FieldName = f.FieldName,
                            FieldTypeName = f.FieldType
                        }).ToList()
            }
        };
    return entity_message;
}

The Service Code:

private void CreateAndSendMessageToProfiler
        (EntityInterceptMessage entity_message)
{
    EndpointAddress address =
        new EndpointAddress
            (“net.tcp://localhost:9002/EntityProfilerSvc”);

    var binding =
        new NetTcpBinding { SendTimeout = new TimeSpan(0, 10, 0) };

    ChannelFactory<IProfilerService> factory =
        new ChannelFactory<IProfilerService>(binding, address);

    var channel = factory.CreateChannel();

    XmlReader reader = MessageSerializer.SerializeMessage(entity_message);

    //Create the Message with the serialized data 
    //embedded into its body
    var msg = chans.Message.CreateMessage(
        binding.MessageVersion,
        “urn:IProfilerService-Send”, reader);

    channel.Send(msg);

    factory.Close();
}

If you dig around in Reflector you can see the code woven into by PostSharp.

The biggest problem I have with this approach is that Entity SQL is not catered for. I would have preferred to use a true Profiling API like the Microsoft one, however the requirement in using unmanaged code left me to bypass that option for now. I very well might re-work the entire application using that API when I get more time to look more deeply into it, but for now it was more important to me to get an alternative going quickly.

What information does the Profiler provide?

The single most important data to report on was always going to be the SQL Query text created by an ObjectQuery<T>. Along with this I have packaged up the execution time for the query (from the managed codes perspective not the database), the fields involved in the query and the process and thread ID’s that execute each query.

What’s Next?

The next thing to add is definitely going to be persistence to disk, making it possible to review query data from previous sessions. Once that’s in place I am going to build an IronRuby console into the application so you can write REPL scripts against your Entity Framework Entity Models and profile the results right there in the profiler – instant Entity Framework gratification! And finally to create a Visual Studio template for the EntityFrameworkProfilingAttribute class so it’s easy to add it to your own libraries that include Entity Models. The code box for the SQL Text is licensed under the code project open license and usage of the Profiler requires accepting this license. As I have said all along, suggestions are welcome. To see how this all works, you can get the code here and just run the tests and you will see the messages appear in the Profiler.

It has just recently come to my attention that Udi will soon be running his advanced distributed systems design course in Austin Texas with Jeffrey Palermo’s crew at Headspring. I was fortunate enough to take this very course earlier this year in Australia and I can truly say that my thinking, planning, designing and output have all been enhanced in the most profound ways since doing so, it was one of those great eye opening experiences that can move you to another plane of thinking.

Judging from the course outline on Udi’s site, it would appear that some new material has found it’s way onto the syllabus (I’m jealous), which covers Ultra High scaling possibilities by leveraging REST and after hearing Udi’s podcast on that topic I think I have a feel for what that promises and can only say that I wish I could make the trip to the US to be there.

Whilst I gained enormously from the SOA and DDD content, I also took away some wonderful stuff from the ‘Smart Client’ sections of the training. My team have been using our own home grown MVP framework for some time now and Udi certainly expanded our thinking there, particularly with regard to multi threading issues - greatly useful stuff.

When I had the privilege of sitting through this training I did so with my team and cant say enough about how it invigorated, improved and turbo charged the mind-set and output of all who attended. Udi is a very skilful presenter and a wonderful teacher and significantly has amassed some great wisdom that anyone who is serious about building Service Oriented systems should go out of their way to experience and imbibe. If your in Austin Texas or anywhere in the United States for that matter, I thoroughly recommend you take this course, it is worth far more than the advertised price. Prepare to have some of your beliefs challenged and come with a spirit of learning and you will enjoy the rewards - I know I have and so have many of my colleagues. I know that I personally often find enormous value in listening and learning from people of the quality of Udi and Randy, so imagine spending a week in the company of one such teacher!!!

Have you ever needed to use inheritance with LINQ To SQL? The designer experience isn’t exactly what your OO professor probably had in mind when it comes to modelling inheritance and to honest the whole process feels wrong. Let me explain.

I don’t like the whole ‘partial’ class approach to inheritance to begin with. I want the freedom to inherit without the need for using Partial classes and better still I don’t want my business objects to be tied to the LINQ To SQL platform. If we consider the following scenario where my LINQ To SQL designer experience looks like this:

Figure 1.0

What figure 1.0 demonstrates is the we have an employee entity, from which three other entities all inherit. SalaryEmployee, CommissionEmployee and ContractEmployee all have employee as their base class. In the case of SalaryEmployee, it distinctly encapsulates a property for the middle initial via the property ‘minit’. Sure the designer looks great but from the perspective of the code, the designer has generated things don’t look so nice to me, as demonstrated in the following code listing.

Code Listing 1.0

[Table(Name="dbo.employee")]
[InheritanceMapping(Code="1", Type=typeof(SalaryEmployee)]
[InheritanceMapping(Code="2", Type=typeof(CommissionEmployee))]
[InheritanceMapping(Code="3", Type=typeof(ContractEmployee))]
public partial class employee : INotifyPropertyChanging,
                                INotifyPropertyChanged
{
    //…….
}

public partial class SalaryEmployee : employee
{
    private System.Nullable<char> _minit;

    #region Extensibility Method Definitions

    partial void OnLoaded();
    partial void OnValidate(System.Data.Linq.ChangeAction action);
    partial void OnCreated();
    partial void OnminitChanging(System.Nullable<char> value);
    partial void OnminitChanged();

    #endregion

    public SalaryEmployee()
    {
        OnCreated();
    }

    [Column(Storage="_minit", DbType="Char(1)")]
    public System.Nullable<char> minit
    {
        get
        {
            return this._minit;
        }
        set
        {
            if ((this._minit != value))
            {
                this.OnminitChanging(value);
                this.SendPropertyChanging();
                this._minit = value;
                this.SendPropertyChanged("minit");
                this.OnminitChanged();
            }
        }
    }
}

public partial class CommissionEmployee: employee
{
    //…….
}

public partial class ContractEmployee: employee
{
    //…….
}

As demonstrated above, the designer generates code that comes decorated with quite a bit of LINQ To SQL infrastructure attributes or noise as I like to call it, along with some partial methods that developers can implement in their own partial classes designed to extend those generated by the designer. I have included the entire code generated for the SalaryEmployee entity for the sake of demonstration and to show that further LINQ To SQL infrastructure code has been used to decorate the entity. Now it’s very important to point out here, that this code should NOT be touched in the IDE and it’s linked inextricably to the designer, so if I just start to add polymorphic behaviours or new members to the entities via the generated code, then the moment I refresh or add anything via the designer, it will blow away any changes made by directly to the code. Microsoft have designed the entity classes in a such a way that by using partial classes we can implement a subsequent file to that provided by the code generator and implement all of our new behaviours and members using our own hand rolled partial class. I could do something like this:

Code Listing 2.0

public interface IEmployee
{
    bool QualifiesAsPrefered();
}

public partial class employee : IEmployee
{
    #region IEmployee Members

    public virtual bool QualifiesAsPrefered()
    {
        return false;
    }

    #endregion
}

I could create this code above in a new code file and thus extend my designer generated entities by using an interface to describe any polymorphic intent against my partial class. This does begin to become unwieldy in my opinion and I consider the use of the partial class a bit of code smell to be blunt and its really only required because of the need to maintain synchronicity between the code generated entity and the designer.

Now this is not to say that developers should not use the designer and take the approach just demonstrated, in fact I should point out that given the right circumstances, such an approach is completely justified. Lets say you work for a small or medium sized project driven business and there is a need to write a data driven application that contains less than significant business logic, has low support requirements, a small user audience and is required in the shortest amount of time possible. Sounds familiar right! Well those are typical times when you might want to do just that. However, what about if I have a team of 6 - 8 developers on a project spanning eight months and the application is driven heavily by data? This is the kind of circumstance that this approach just wont feel comfortable and will cause you headaches because it’s not particularly reusable or portable across a large team, it also leads to business entities getting tied to a Data Access strategy and will have the developer team dragging around all that infrastructure code with them. What is this going to mean for sharing entities within services or what are my serialisation concerns and should I consider breaking my partial classes into separate projects and…….you can see where this is leading, more confusion and pain for all. So what’s the solution?

By using a POCO (Plain Old CLR Object) approach (see DDD) along with LINQ To SQL mapping, we can choose to use our tried and trusted OO skills to describe our classes (entities), cutting down on the number of artifacts it takes to achieve that and in so doing simplify our lives by making the code less ambiguous. So how different would that be? Lets start with the mapping file used to map entities to tables in the database.

Code Listing 3.0

<Database Name="Northwind"
    xmlns="http://schemas.microsoft.com/linqtosql/mapping/2007">
  <Table Name="Customers" Member="Customer">
    <Type Name="Org.TechAvalanche.Orm.Tests.Domain.Customer">
      <Column Name="CustomerID" Member="CustomerID"
        DbType="NChar(5)
            NOT NULL" IsPrimaryKey="true"
        CanBeNull="false" UpdateCheck="Never" />
      <Column Name="CompanyName" Member="CompanyName"
        DbType="NVarChar(40)
            NOT NULL"
        CanBeNull="false" UpdateCheck="Never" />
      <Column Name="ContactName" Member="ContactName"
        DbType="NVarChar(30)"
        CanBeNull="true" UpdateCheck="Never" />
      <Column Name="ContactTitle" Member="ContactTitle"
        DbType="NVarChar(30)" CanBeNull="true"
        UpdateCheck="Never" />
      <Column Name="Address" Member="Address"
        DbType="NVarChar(60)" CanBeNull="true"
        UpdateCheck="Never" />
      <Column Name="City" Member="City"
        DbType="NVarChar(15)" CanBeNull="true"
        UpdateCheck="Never" />
      <Column Name="Region" Member="Region"
        DbType="NVarChar(15)" CanBeNull="true"
        UpdateCheck="Never" />
      <Column Name="PostalCode" Member="PostalCode"
        DbType="NVarChar(10)" CanBeNull="true"
        UpdateCheck="Never" />
      <Column Name="Country" Member="Country"
        DbType="NVarChar(15)" CanBeNull="true"
        UpdateCheck="Never" />
      <Column Name="Phone" Member="Phone"
        DbType="NVarChar(24)" CanBeNull="true"
        UpdateCheck="Never" />
      <Column Name="Fax" Member="Fax"
        DbType="NVarChar(24)" CanBeNull="true"
        UpdateCheck="Never" />
      <Column Name="CustType" Member="CustomerType"
        DbType="Int" CanBeNull="false" IsDiscriminator="true"
        UpdateCheck="Never" />
      <Association Name="Customer_Order"
        Member="Orders" OtherKey="CustomerID" />

      <!– These Two Elements used for inheritance –>
      <Type
        Name="Org.TechAvalanche.Orm.Tests.Domain.InternalCustomer"
        InheritanceCode="1" IsInheritanceDefault="true"/>
      <Type
        Name="Org.TechAvalanche.Orm.Tests.Domain.ExternalCustomer"
        InheritanceCode="2" IsInheritanceDefault="false"/>
    </Type>
  </Table>
  <Table Name="Orders" Member="Order">
    <Type Name="Org.TechAvalanche.Orm.Tests.Domain.Order">
      <Column Name="OrderID" Member="OrderID"
        DbType="Int NOT NULL
            IDENTITY" IsPrimaryKey="true"
        IsDbGenerated="true" CanBeNull="false"
        UpdateCheck="Never" />
      <Column Name="CustomerID" Member="CustomerID"
        DbType="NChar(5)" CanBeNull="true"
        UpdateCheck="Never" />
      <Column Name="EmployeeID" Member="EmployeeID"
        DbType="Int" CanBeNull="true"
        UpdateCheck="Never" />
      <Column Name="OrderDate" Member="OrderDate"
        DbType="DateTime" CanBeNull="true"
        UpdateCheck="Never" />
      <Column Name="RequiredDate" Member="RequiredDate"
        DbType="DateTime" CanBeNull="true"
        UpdateCheck="Never" />
      <Column Name="ShippedDate" Member="ShippedDate"
        DbType="DateTime" CanBeNull="true"
        UpdateCheck="Never" />
      <Column Name="ShipVia" Member="ShipVia"
        DbType="Int" CanBeNull="true"
        UpdateCheck="Never" />
      <Column Name="Freight" Member="Freight"
        DbType="Money" CanBeNull="true"
        UpdateCheck="Never" />
      <Column Name="ShipName" Member="ShipName"
        DbType="NVarChar(40)" CanBeNull="true"
        UpdateCheck="Never" />
      <Column Name="ShipAddress" Member="ShipAddress"
        DbType="NVarChar(60)" CanBeNull="true"
        UpdateCheck="Never" />
      <Column Name="ShipCity" Member="ShipCity"
        DbType="NVarChar(15)" CanBeNull="true"
        UpdateCheck="Never" />
      <Column Name="ShipRegion" Member="ShipRegion"
        DbType="NVarChar(15)" CanBeNull="true"
        UpdateCheck="Never" />
      <Column Name="ShipPostalCode" Member="ShipPostalCode"
        DbType="NVarChar(10)" CanBeNull="true"
        UpdateCheck="Never" />
      <Column Name="ShipCountry" Member="ShipCountry"
        DbType="NVarChar(15)" CanBeNull="true"
        UpdateCheck="Never" />
      <Association Name="Order_OrderLine" Member="OrderLines"
        OtherKey="OrderID" />
      <Association Name="Customer_Order" Member="Customer"
        ThisKey="CustomerID" IsForeignKey="true" />
    </Type>
  </Table>
  <Table Name="[Order Details]" Member="OrderLine">
    <Type Name="Org.TechAvalanche.Orm.Tests.Domain.OrderLine">
      <Column Name="OrderID" Member="OrderID"
        DbType="Int NOT NULL" IsPrimaryKey="true"
        CanBeNull="false" UpdateCheck="Never" />
      <Column Name="ProductID" Member="ProductID"
        DbType="Int NOT NULL" IsPrimaryKey="true"
        CanBeNull="false" UpdateCheck="Never" />
      <Column Name="UnitPrice" Member="UnitPrice"
        DbType="Money NOT
            NULL" CanBeNull="false"
        UpdateCheck="Never" />
      <Column Name="Quantity" Member="Quantity"
        DbType="SmallInt
            NOT NULL" CanBeNull="false"
        UpdateCheck="Never" />
      <Column Name="Discount" Member="Discount"
        DbType="Real NOT
            NULL" CanBeNull="false"
        UpdateCheck="Never" />
      <Association Name="Order_OrderLine" Member="Order"
        ThisKey="OrderID" IsForeignKey="true" />
    </Type>
  </Table>
</Database>

The two entries above that sit directly beneath the comment highlighted in large font demonstrate how to set up mapping XML for regular DotNet CLR objects that are NOT generated by the designer and account for inheritance. The entity code to go with this mapping follows below.

Code Listing 4.0

interface ICustomerData
{
    string Address { get; set; }
    string City { get; set; }
    string CompanyName { get; set; }
    string ContactName { get; set; }
    string ContactTitle { get; set; }
    string Country { get; set; }
    string CustomerID { get; set; }
    string Fax { get; set; }
    IList<Order> Orders { get; set; }
    string Phone { get; set; }
    string PostalCode { get; set; }
    string Region { get; set; }
    int CustomerType { get; set; }
}

public interface ICustomerLoyaltyDiscount
{
    void LoyaltyDiscount();
}

interface ICustomer : ICustomerData, ICustomerLoyaltyDiscount { }

public class Customer : ICustomer
{
    public void LoyaltyDiscount()
    {
        foreach (Order o in Orders)
        {
            o.Freight = 0M;
        }
    }

}

public class InternalCustomer : Customer
{
    //………etc
} 

public class ExternalCustomer : Customer
{
    //………etc
}

If you really must have a designer to view and work with your entities then it’s no sweat to create a class diagram in VS.Net and enjoy a similar design experience as you would with the LINQ To SQL designer however enjoy the benefits of going POCO at the same time.

Figure 2.0

The consumer code would look something like this:

public void SimpleRepositoryFetchingStrategyTest()
{
    Specification<Customer> simonsCoSpec =
        new Specification<Customer>
        (c => c.Country == "Germany");

    DataContext ctx = GetContext();

    Repository<Customer> customerRepository =
        new Repository<Customer>(ctx);

    CustomerLoyaltyDiscountFetchingStrategy strategy =
        new CustomerLoyaltyDiscountFetchingStrategy();

    var custs = customerRepository.All(simonsCoSpec, strategy);

    foreach (var cust in custs)
    {
        cust.LoyaltyDiscount();
        Console.WriteLine("The Name of the Customer " +
            "from Germany is {0}", cust.CompanyName);
        foreach (var order in cust.Orders)
        {
            Console.WriteLine("\tOrder Number {0}",
                order.OrderID);
            foreach (var orderLine in order.OrderLines)
            {
                Console.WriteLine("\t\tProduct ID {0} Amount {1}",
                        orderLine.ProductID,
                        (orderLine.Quantity * orderLine.UnitPrice));
            }
        }
    }
}

private static DataContext GetContext()
{
    //get a stream reader from the mapping file

    StreamReader sr =
        new StreamReader(@"D:\Mapping\DomainMapping.xml");

    //set the mapping source up
    XmlMappingSource mapping =
        XmlMappingSource.FromStream(sr.BaseStream);

    DataContext db = null;

    //new up a persistance repository
    db = new DataContext
        ("server=localhost;initial catalog=Northwind;" +
        "user id=sa;password=mrbig", mapping);

    return db;
}

I will shortly be posting a framework that makes working in a POCO and DDD style much simpler. The Framework includes implemented Specification Patterns, Fetching Strategies and Extensible Repositories for your LINQ To SQL pleasure.