Living in the Tech Avalanche Generation

The content of this series of posts is based around a spiking effort / proof of concept that aims to modularize (I am avoiding the word framework) a set of libraries for the express purpose of addressing business intent for Domain models persisted with the Entity Framework. In part 1.0 we laid the groundwork and now we dig deeper.

I’m going to start this post at the end and then rewind all the way to beginning and step through how we got here. Here is some code that will return a list of products and in so doing, will dynamically determine the correct Entity type to return (a Product), will eagerly fetch their reference supplier and will apply a ‘where’ predicate to the LINQ To Entities query. All this will happen out of clear view as a result of being explicit and specifying a role / interface to the ORM when we ask it do something.

public void Some_Method_In_My_Service_Layer
{
    var products = Repository.Get<IRunOutDiscountForProducts>();
}

Now let’s briefly forget we saw this code and go back to the start.

Lets start by imagining some simple data access code that’s enormously compact and equally deceptive in it’s apparent simplicity. For a scenario, let’s propose that we want to get all the products in our system that are eligible for applying a run out discount. The business logic in achieving this function, specifies that a Products Unit Price Price  be greater 20.50 dollars and is purchased from suppliers in a given enumerable list of regions. For the sake of the example we will also say that this would require a Product entity and Supplier Entity in the following relationship.

So given our the scenario we would be required to map the Entities appropriately, choose an appropriate method of loading (lazy or eager) and apply a suitable predicate to the SQL query. Let’s now imagine that by specifying a role (in this case the interface IRunOutDiscountForProducts), that we have provided the infrastructure code enough information to query the database for products and do so in a way that meets the proposed scenario requirements enumerated above.

var conParams = new Dictionary<string, string>();
conParams .Add(_conParamName, _connectionStringOnly);

var repository = FetchSpec.Configure
                    .With(new StubIoc()
                        { ScannedAssemblyPath = _scannedAssemblyPath })
                    .AndBuildSession<IRunOutDiscountForProducts>(conParams)
                    .ForRepository();

var products = repository.Get<IRunOutDiscountForProducts>();

The line of code that interests us most at this point is the last one. By applying the role of IRunOutDiscountForProducts to the GET<T> function, we are able to indicate to the infrastructure that our intent is to retrieve a list of Products for the express reason to apply a ‘run out’ discount to them. The next line of code you might expect to see would logically be something like:

products = repo.Get<IRunOutDiscountForProducts>();
foreach (var product in products)
{
    product.DiscountProductForRunOut();
}

repo.Save();

What figure 1.0 shows is that we would like our Repository to be smart enough to know exactly what and how to fetch based on the role (the generic parameterized interface).

Figure 1.0

I promise we will get to the end soon enough but bear with me for a minute or two. One thing about being explicit with roles, it makes understanding code all the more simple. The less ambiguity the less likely someone misunderstands intent. In the realm of software development the humble interface (not the UI kind) has carried the torch ably in helping us declare our intent more explicitly, make our code better understood and free of concrete dependencies / coupling. With reverence and honour for my past (COM programming veteran status), I apologise in advance for the ubiquitous IDog example but it makes a point.

public class Poodle : IDog
{
    //….etc
}

Explicitly a Poodle must now conform polymorphically et al to the contract specified by IDog – real 101 stuff no doubt. What about a marker interface? True enough, a very useful construct, bereft of polymorphic intent and largely a utility to express membership to a conceptual entity group of some kind, often put to use in frameworks to support dynamism of some description. Let’s use an NServiceBus IMessage as an example:

public interface IMessage {}

public class OrderCancelledMessage : IMessage
{
    //….etc
}

Without going too deep into this at a code level, suffice to say that the NServiceBus infrastructure will behave in certain ways when it comes across types that implement the IMessage interface, in other words the framework is directing us to ‘mark’ a given class (or interface) as being recognizable as a ‘Message’, making it identifiable to the frameworks internal workings.

The virtues of programming by interface are now well documented and for .NET developers the advent and popularity of generics has encouraged our creativity in placing stable abstractions  around things to make code more explicit. For example, a while back I wrote in the abstract on the use of interfaces to define roles; roles describe the polymorphic nature of a business domain command and arguably event. This series of posts is an extension of those made in the past, the focus however now moves to a proof of concept for the up coming Entity Framework 4.0.

What is a Role?

Roles identify or describe commands / behaviours in the domain, an level of intent that can be be nicely defined by an interface.

public interface IRunOutDiscountForProducts : IProductData
{
    void DiscountProductForRunOut();
}

We can use these roles (interfaces) to indicate to our infrastructure (an ORM in this case) what our intent is and have the infrastructure behave appropriately based on configuration and / or convention.

Since we started with an example that focused around products, lets continue with that subject for a moment. It may be that my system needs to fetch Products for a variety of reasons and not always with the same intent. We shouldn’t be asking for a generic list of Products, considering that we are interested in them with a very specific context in mind and its the context determines the behaviour and describes the a role.

Once our system becomes explicit in this way through the use of roles in our domain, we can do some interesting things. We can instruct the ORM how to load the entity graph by implementing Fetching Strategies, dynamically resolve the most appropriate Entity Mapping to use and which predicate to apply to the query for the purpose of business logic filtering. Not only does the the role provide a stable abstraction, it insulates other parts of the architecture, perhaps a service layer or controller, from changes, allowing the actors to vary independently.

The Roles Quadrant:

Whilst working on this proof of concept in my attempts at cajoling the Entity Framework to work in this fashion, it became evident that there were four main pieces of the puzzle, Fetching Strategies, Specifications, Mapping, and Repositories. One thing that I haven’t mentioned yet, is the effect on concurrency, scale and throughput that this approach can have. Through the use of roles we can plug into the architecture yet again and explicitly pick an isolation level for the operation at hand.

using (var scope = new TransactionScope(TransactionScopeOption.Required,
    new TransactionOptions()
    {
        IsolationLevel =
            SessionBuilder<IRunOutDiscountForProducts>.ScopeIsolation()
    }))
{
    //…..do some stuff here like load data 
    //…..change data by calling into your domain model
    //…..and finally save the changes

    scope.Complete();
}

Notice that in constructing the Transaction scope we use the SessionBuilder and ask it to supply an Isolation Level for the role specified. Of course their is some IOC magic involved here but essentially the only thing required to make work is the following interface. Here is an example implementation.

public interface IProvideIsolationLevelFor<TRole>
{
    IsolationLevel GetScopeIsolationForRole { get;}
}

///Implementation

public class RunoutDiscountIsolation :
    IProvideIsolationLevelFor<IRunOutDiscountForProducts>
{
    public IsolationLevel GetScopeIsolationForRole
    {
        get { return IsolationLevel.ReadCommitted; }
    }
}

Loaded Up

_container.Register(AllTypes
    .Of(typeof(IFetchingStrategy<>))
    .FromAssembly(assembly)
    .Configure(c => c.LifeStyle.Singleton).WithService.FirstInterface());

_container.Register(AllTypes
    .Of(typeof(ISpecification<>))
    .FromAssembly(assembly)
    .Configure(c => c.LifeStyle.Singleton).WithService.FirstInterface());

_container.Register(AllTypes
    .Of(typeof(IMappingRole<>))
    .FromAssembly(assembly)
    .Configure(c => c.LifeStyle.Singleton).WithService.FirstInterface());

_container.Register(AllTypes
    .Of(typeof(IProvideIsolationLevelFor<>))
    .FromAssembly(assembly)
    .Configure(c => c.LifeStyle.Singleton).WithService.FirstInterface());

_container.Register(AllTypes
    .Of(typeof(IEntity))
    .FromAssembly(assembly));

Quadrant Position 1.0 – Fetching Strategies

Fetching strategies encapsulate our expression of requirement with respect to Eager and Lazy Loading. This feature is useful, when one considers that the Entity Framework (and most ORM’s for that matter) do not provide these capabilities with a consistent API experience, nor allow for an out of box experience friendly to a configurable or convention based approach to specifying loading characteristics for a given expression of intent. Wouldn’t it be nice if we could ask the ORM to fetch an Entity using a role, one specific to a command that exists within a given business capability? This functionality does exist with NHibernate but not currently with either version of Entity Framework, but the good news is that it is possible. There are a couple of ways of doing this and I have blogged about it more than once. At this point of the discussion it would be useful to show some code and how to configure a Fetching Strategy. First the interface:

public interface IFetchingStrategy<TRole> : IQueryConfigurable
{
    IEnumerable<IEagerFetchingIntention> Intentions { get; }
    IEnumerable<string> Includes { get; }
    void AddIntentions(IEagerFetchingIntention[] intentions);
    bool ShouldLazyLoad { get; }
    bool HasInstructions { get; }
}

Next let’s look at an implementation (we will skip the base class for now).

public class CustomerPreferedFetchingStratey :
             FetchingStrategy<IMakeCustomerPrefered>
{
    public CustomerPreferedFetchingStratey()
        : base(false) {

            var orders_intention =
                EagerFetchingIntention
                .CreateInstance<IMakeCustomerPrefered,
                                ICollection<IOrder>>(c => c.Orders);
            var orderlines_intention =
                EagerFetchingIntention
                .CreateInstance<IOrder,
                                ICollection<IOrderLine>>(o => o.OrderLines);

            this.AddIntentions(new IEagerFetchingIntention[]
            {
                orders_intention,
                orders_intention.And(orderlines_intention)
            });
    }
}

Quadrant Position 2.0 – Specification

The Specification pattern evolved around the problem of testing that the state condition of an object is met according to a given specification. This pattern has taken on some new potential with the help of LINQ. Using LINQ Expressions it’s possible to massage this pattern into something of a predicate store and when we think about developing with roles it becomes potentially more powerful again. Here is a quick look at a Specification and some usage:

var spec = new Specification<IRunOutDiscountForProducts>
              (p => p.UnitPrice < 20M && p.Supplier.Region == “London”);

Pretty simple, the constructor takes the Expression to initialise the boolean test for the specified state. The classical purpose of the pattern is to provide a single function, commonly named “IsSatisfied” to carry out the test against a given object to see if it’s state indeed meets that that has been specified.

var productToTest = new Product(){Supplier = new Supplier()};
var satisfied = spec.IsSatisfiedBy(productToTest);

This will code will obviously fail the specification test and IsSatisfiedBy() will return false. Whilst very useful in and of itself, the LINQ Expression enabled version / implementation of this pattern has some features that can be harnessed to apply a level of dynamism to the resolution of query predicates. Let’s have a closer look:

public interface ISpecification<T> : IQueryConfigurable
{
    Expression<Func<T, bool>> EvalPredicate { get; }
    Func<T, bool> EvalFunc { get; }
    bool IsSatisfiedBy(T entity);
}

The interface details the contract to provide both the Expression and Func that will provide the computation required to discern whether a specification has been satisfied. Further to that, these new properties give us opportunities to plug predicates or lambdas into the machinery as well. Looking a little deeper under the covers of the proof of concept code we will find this in operation:

return _ctx
       .CreateObjectSet<TRole>()
       .Where(predicate.EvalPredicate)
       .ToList();

Through operator overloading, Specifications can also be combined for OR and AND conditions:

Specification<Customer> german_customer_spec =
    new Specification<Customer>(c => c.Country == “Germany”);

Specification<Customer> us_customer_spec =
    new Specification<Customer>(c => c.Country == “Usa”);

var comb_country_spec = (german_customer_spec | us_customer_spec);

In the case where query predicates help to form a semi static business rule at the query level through filtering, it becomes a reasonably trivial task to deploy a new version or entirely new Specification for any given role, simply by deploying the library.

Quadrant Position 3.0 – Mapping

Now to the latest part of the puzzle to be rolled into the Proof of concept, mapping. Entity Framework 4.0 (sometime after initial release) will be augmented by code only mapping (hallelujah). When one looks at the (current CTP) API for this functionality it becomes immediately evident that this too is something that can be very easily made deterministic by adding a layer of indirection to accommodate this role based approach. The out of the box mapping in EF 4.0 makes working in this manner quite simple given the possibilities provided by the EntityConfiguration<T> class. Here is a simple example:

public class CustomerMapping : EntityConfiguration<ICustomer>
{
    public CustomerMapping()
    {
        Property(c => c.CustomerId).IsIdentity();
        Property(c => c.FirstName).HasMaxLength(50).IsRequired();
        Property(c => c.LastName).HasMaxLength(50).IsRequired();
        Relationship(c => c.Orders).IsOptional();
        Relationship(c => c.Orders).FromProperty(o => o.Customer);
    }
}

What’s required is to take this out of the box solution and add a layer of indirection requiring only a few new actors, an IMapping, an IMappingRole<T>  and a Mapping<T>.

public interface IMapping
{
    Type EntityType { get; }
    string ObjectSetName { get; }
    StructuralTypeConfiguration Configuration { get; }
}

public interface IMappingRole<TRole>
{
    IMapping[] Mappings { get; }
}

public abstract class Mapping<TEntity> :
                      EntityConfiguration<TEntity>, IMapping
{
    private string _objectSetName;

    public Type EntityType
    {
        get { return typeof(TEntity); }
    }

    public string ObjectSetName
    {
        get { return _objectSetName; }
    }

    public Mapping(string objectSetName)
    {
        _objectSetName = objectSetName;
    }

    public StructuralTypeConfiguration Configuration
    {
        get { return (StructuralTypeConfiguration)this; }
    }
}

The aggregation of IMapping’s in the Mapping Role will accommodate mapping an aggregate root with reference and collection entities. Here’s a more complete look that includes some implementing Mapping classes.

Below is an example Mapping class, notice that we still need to use the actual Entity Type and not an interface. This is due to EntityConfiguration<T> lacking in support for interfaces (as of today, perhaps that will change or already has with RC 1).

public class ProductMapping : Mapping<Product>
{
    public ProductMapping(string objectSetName) : base(objectSetName)
    {
        Property(p => p.ProductID).IsIdentity();
        Property(p => p.ProductName).HasMaxLength(50).IsRequired();
        Property(p => p.UnitPrice).Precision = 19;
        Property(p => p.UnitPrice).Scale = 4;
        Relationship(p => p.Supplier).IsOptional();
        Relationship(p => p.Supplier).IsOptional().FromProperty(s => s.Products);
    }
}

Quadrant Position 4.0 – The Repository

The final square on the grid is the Repository. The repository offers nothing more than five overrides of the one method, Get<T>. The five versions of the method are to cover the variable requirements that might crop up when consumers wish to provide Fetching Strategies, Specifications and Entities (concrete ones) inline and directly to the infrastructure. Fetching Strategies are probably the least likely candidate for this requirement but it may come in useful, Specifications on the other hand will from time to time no doubt be populated dynamically from user input, served as part of a request or message and subsequently used as a filter parameter. Let’s have a brief look at the Interface:

public interface IRepository : IUnitOfWork
{
    IList<TRole> Get<TRole>();
    IList<TRole> Get<TRole>(IQueryConfigurable[] configurables);
    IList<TRole> Get<TRole>(ISpecification<TRole> predicateWhere)
                            where TRole : class;
    IList<TRole> Get<TRole>(IFetchingStrategy<TRole> strategy)
                            where TRole : class;
    IList<TRole> Get<TRole>(ISpecification<TRole> predicateWhere,
                            IFetchingStrategy<TRole> strategy)
                            where TRole : class;
    IoC IocContainer { set; }
    ObjectContext Session { set; }
}

The first two versions of Get<T> will resolve as much as they can from the container, a Fetching Strategy, a Specification, the Mapping and will use the closest adjacent Entity type, all based on the identity of the generic parameter – that is, the ROLE. The last three versions of the Get<T> function is provided to work with the Entity Framework in a more familiar mode where and EDMX file can be specified, the concrete Entity Type itself is provided as the role parameter, we can also push through Fetching Strategies and Specifications ‘in-line’ in these scenarios. Probably not my preferred method but it’s available.

Configuration

I decided at the death to implement a fluent configuration to prime the infrastructure and create a Repository. The fluent interface is designed with the view to keeping things compact and easy to setup.

Resolve it all

public void resolves_everything_from_container()
{
    var compParams = new Dictionary<string, string>();
    compParams.Add(_conParamName, _connectionStringOnly);

    var repo = FetchSpec.Configure
                        .With(new StubIoc()
                            { ScannedAssemblyPath = _scannedAssemblyPath })
                        .AndBuildSession<IRunOutDiscountForProducts>(compParams)
                        .ForRepository();

    var products = repo.Get<IRunOutDiscountForProducts>();
}

In this example above, we are using a stub for IoC, and hard wiring the scanned assembly path to find all the implementing Fetching Strategies, Specifications, Mappings etc. The default behaviour in the spiking code is to enumerate the assemblies with the executing processes deployment.

Provide it all

For those that like the EDMX way of doing things and want to provide all the Fetching and Speciation instructions in line, this is the way to go about it:

public class CustomerBuyingStatusRepository : Repository
{
    public CustomerBuyingStatusRepository()
    {
        //BEWARE THIS STRING WONT COMPILE ITS MESSED UP
        //FOR THE PURPOSE OF VIEWING IN THE BROWSER.
        const string constring = “metadata=res://*/Northwind.csdl|res://*/” +
                                 “Northwind.ssdl|res://*/Northwind.msl;” +
                                 “provider=System.Data.SqlClient;provider “ +
                                 “etc, etc, etc”;

        this.IocContainer = new IoC();
        this.Session = new NorthwindEntities(constring);
    }
}

public void using_concrete_repository_entity_and_inline_spec_and_fetching()
{
    var customersStartWithA =
        new Specification<Customer>(c => c.CompanyName.StartsWith(“A”));

    var fetch = new FetchingStrategy<Customer>(true);

    fetch.AddIntentions(new IEagerFetchingIntention[]
    {
        EagerFetchingIntention.CreateInstance<Customer,
                                              ICollection<IOrder>>(c => c.Orders)
    });

    var repo = new CustomerBuyingStatusRepository();

    var custs = repo.Get<Customer>(customersStartWithA, fetch);

    foreach (var cust in custs)
    {
        Console.WriteLine(cust.CompanyName);
        cust.MakePrefered();
    }
}

I could have easily left out the ability to support this in-line method, given that the purpose of the spike was to proof the role based approach, having it’s greatest value proposition in being resolved dynamically against the Role. Nonetheless it is possible to mix and match or exclusively do either if you absolutely must. No prizes for guessing which mode I am more partial to, however I will reserve final judgement until it gets a good hit out in the real world, remember this is all the result of a brute force spiking effort.

What’s Left?

In part 3.0 we will look at concurrency and throughput through the lens of a service layer. I will also make the code download available in part 3.0.

This is the final part of the series on using IronPython together with the Entity Framework.

Part 1.0

In this first instalment we setup the discussion on what we were trying achieve, how to use the __clrtype__ class in IronPython 2.6 to create PODO’s (Plain Old Dlr Objects) ala POCO, for integration with the Entity Framework. Our aim is to use the Entity Framework using the Beta 2 of Entity Framework 4.0.

Part 2.0

We dig in deep and explore the internals of what’s required to create a PODO (POCO for the DLR). We also build an IronPython ObjectContext and determine how to manage mapping between the Entities and Database. We achieve a successful test run and prove that we can in simple cases get IronPython and Entity Framework to worth together in this fashion. Finally in part 2.0 we take steps to verify lazy loading for an entity graph and we strike our first hurdle, leaving us to look forward to part 3 to see if the problems encountered might be overcome.

Part 3.0

We start to examine the issues with scalar / native types and see how proxies fit into the puzzle. We check out eager loading and whilst we do have some success we still face a problem in rounding out the whole story. Finally we prepare to journey into debugging a crash dump file for part 4.

Part 4.0

Using Visual Studio 2010 we successfully load up our crash dump file in the hope of jumping over the final hurdle. We discover that the specialized version of IronPython 2.6 does not have publicly available debug symbols but we can manage to see where the active thread was when the fatal crash occurred.

The State of Play

The current set of problems unfortunately leaves us short in terms of using the Entity Framework from IronPython, however I did previously demonstrate (in two parts here and here) how this can be achieved by way of introducing an interop layer with C# addressing the Entity Framework.

Part 1.0

Part 2.0

Part 3.0

Part 4.0

It’s my hope that this post will make the previous posts easier to find given my recent problems with a blog scraping site. Finding the original content has been difficult in light of my successfully having the scraping site in question remove my content.

We left off with IPY.exe crashing and leaving us a dump file. The code that forced the crash:

orders = ctx.Orders.Where(“it.CustomerID > ‘C’“)
for order in Orders:
    print Order.CustomerID

I speculated that something other than member assignment (made by the Entity Framework) to string types in the underlying CLR type of the Python entity was failing and you might remember some proof of concept code was demonstrated to show a variety of ways in specifying types to _clrfields and property decorators.

The Dump File

So decided to re-watch this timely video presentation by Tess Ferrandez and hosted by Scott Hanselman, to see if I could get myself up to speed quick enough to debug the crash dump in VS 2010.

Not that it’s relevant to this topic per`se but did find something odd in locating the dump file and thought it’s worth mentioning here: when the system notifies you of a crash that produces a dump file (on Vista 64 in this case), it went through a process of:

1. Prompting with the message that IPY.exe had indeed crashed
2. Where to find the dump file

After noting the path of the dump file and clearing these two dialogs, I went to retrieve the dump file only to find it was not at all where the dialog said it would be. I ran the process again and watch the target folder for the dump file and it appears that when the second dialog is cleared the dump file will be deleted – so make a copy before that happens!

Anyway back to the problem at hand. Once having secured a copy of the crash dump file I loaded into VS.NET 2010 (very slow with downloading the debug symbols for the first time). Eventually this is what I was presented with:

The information of most note here is the exception information:

“The thread tried to read from or write to a virtual address for which it does not have the appropriate access”

Obviously it’s important to know what was happening on the active thread when debugging for a system that crashed. When prompted to break by visual studio when all the debug symbols have been loaded I received the following message:

An unhandled exception of type ‘System.ExecutionEngineException’ occurred in Unknown Module.

Not very useful so far huh! When I open the parallel call stacks I can see that eight threads were active at the time of the crash and that execution stopped in NT.DLL and in particular on the call listed below.

ntdll.dll!_ZwWaitForSingleObject@12()  + 0×15 bytes

I’m certainly not familiar with debugging dump files but here is what I garnered from the video with Tess. As we look through Call Stack (above) in Visual Studio, we can see that DLR, IronPython and Entity Framework 4.0 functions were called just prior to the failure however it would seem that debug symbols for all three are not currently available on the Symbol Servers from Microsoft. I haven’t yet been able to find them nor been able to locate the source for the special IronPython 2.6 Beta Version for .NET framework 4.0. So we appear to be at impasse, I will continue to see if I can get any deeper into the why and wherefore of the crash and hope to find that one day Plain Old DLR Objects (PODO’s) and Entity Framework can play together in harmony.

Conclusion

It this point in time I would certainly say that using Entity Framework with IronPython will require you build a managed wrapper class(es) that roll up what you want from the entity framework. I showed this to some extent previously with IronRuby in these two posts. Perhaps with a bit of collaborative work the DLR and ADO.NET teams can do a mind meld and move that bit close to making the DLR languages have a more compelling story in this endeavour, however I suppose the question should be asked, is this kind of interop that they have in mind for the dynamic languages?