Living in the Tech Avalanche Generation

This is the final part of the series on using IronPython together with the Entity Framework.

Part 1.0

In this first instalment we setup the discussion on what we were trying achieve, how to use the __clrtype__ class in IronPython 2.6 to create PODO’s (Plain Old Dlr Objects) ala POCO, for integration with the Entity Framework. Our aim is to use the Entity Framework using the Beta 2 of Entity Framework 4.0.

Part 2.0

We dig in deep and explore the internals of what’s required to create a PODO (POCO for the DLR). We also build an IronPython ObjectContext and determine how to manage mapping between the Entities and Database. We achieve a successful test run and prove that we can in simple cases get IronPython and Entity Framework to worth together in this fashion. Finally in part 2.0 we take steps to verify lazy loading for an entity graph and we strike our first hurdle, leaving us to look forward to part 3 to see if the problems encountered might be overcome.

Part 3.0

We start to examine the issues with scalar / native types and see how proxies fit into the puzzle. We check out eager loading and whilst we do have some success we still face a problem in rounding out the whole story. Finally we prepare to journey into debugging a crash dump file for part 4.

Part 4.0

Using Visual Studio 2010 we successfully load up our crash dump file in the hope of jumping over the final hurdle. We discover that the specialized version of IronPython 2.6 does not have publicly available debug symbols but we can manage to see where the active thread was when the fatal crash occurred.

The State of Play

The current set of problems unfortunately leaves us short in terms of using the Entity Framework from IronPython, however I did previously demonstrate (in two parts here and here) how this can be achieved by way of introducing an interop layer with C# addressing the Entity Framework.

Part 1.0

Part 2.0

Part 3.0

Part 4.0

It’s my hope that this post will make the previous posts easier to find given my recent problems with a blog scraping site. Finding the original content has been difficult in light of my successfully having the scraping site in question remove my content.

We left off with IPY.exe crashing and leaving us a dump file. The code that forced the crash:

orders = ctx.Orders.Where(“it.CustomerID > ‘C’“)
for order in Orders:
    print Order.CustomerID

I speculated that something other than member assignment (made by the Entity Framework) to string types in the underlying CLR type of the Python entity was failing and you might remember some proof of concept code was demonstrated to show a variety of ways in specifying types to _clrfields and property decorators.

The Dump File

So decided to re-watch this timely video presentation by Tess Ferrandez and hosted by Scott Hanselman, to see if I could get myself up to speed quick enough to debug the crash dump in VS 2010.

Not that it’s relevant to this topic per`se but did find something odd in locating the dump file and thought it’s worth mentioning here: when the system notifies you of a crash that produces a dump file (on Vista 64 in this case), it went through a process of:

1. Prompting with the message that IPY.exe had indeed crashed
2. Where to find the dump file

After noting the path of the dump file and clearing these two dialogs, I went to retrieve the dump file only to find it was not at all where the dialog said it would be. I ran the process again and watch the target folder for the dump file and it appears that when the second dialog is cleared the dump file will be deleted – so make a copy before that happens!

Anyway back to the problem at hand. Once having secured a copy of the crash dump file I loaded into VS.NET 2010 (very slow with downloading the debug symbols for the first time). Eventually this is what I was presented with:

The information of most note here is the exception information:

“The thread tried to read from or write to a virtual address for which it does not have the appropriate access”

Obviously it’s important to know what was happening on the active thread when debugging for a system that crashed. When prompted to break by visual studio when all the debug symbols have been loaded I received the following message:

An unhandled exception of type ‘System.ExecutionEngineException’ occurred in Unknown Module.

Not very useful so far huh! When I open the parallel call stacks I can see that eight threads were active at the time of the crash and that execution stopped in NT.DLL and in particular on the call listed below.

ntdll.dll!_ZwWaitForSingleObject@12()  + 0×15 bytes

I’m certainly not familiar with debugging dump files but here is what I garnered from the video with Tess. As we look through Call Stack (above) in Visual Studio, we can see that DLR, IronPython and Entity Framework 4.0 functions were called just prior to the failure however it would seem that debug symbols for all three are not currently available on the Symbol Servers from Microsoft. I haven’t yet been able to find them nor been able to locate the source for the special IronPython 2.6 Beta Version for .NET framework 4.0. So we appear to be at impasse, I will continue to see if I can get any deeper into the why and wherefore of the crash and hope to find that one day Plain Old DLR Objects (PODO’s) and Entity Framework can play together in harmony.

Conclusion

It this point in time I would certainly say that using Entity Framework with IronPython will require you build a managed wrapper class(es) that roll up what you want from the entity framework. I showed this to some extent previously with IronRuby in these two posts. Perhaps with a bit of collaborative work the DLR and ADO.NET teams can do a mind meld and move that bit close to making the DLR languages have a more compelling story in this endeavour, however I suppose the question should be asked, is this kind of interop that they have in mind for the dynamic languages?

Before we dive into proxies when using POCO in the Entity Framework lets have a quick recap on where we left off in part 2.0.

• Entity Framework will load PODO’s (plain old DLR Objects) fine as long as:
  • No primitives other than strings are used for fields and properties
• Eager Loading works when ObjectQuery[T].Include() is used.
  • TSQL profiling confirms this.
• Lazy Loading does not appear to work at all
• Type names of materialized entities are not what are expected for runtime generated proxies.

Let’s talk about proxies a little. I wont delve too deep only enough that’s relevant to the problem at hand. Entity Framework creates proxies at runtime for our POCO entities, essentially so the framework can know the entities a vice versa, they maintain the glue so to speak. The glue is there to enable lazy loading and change tracking specifically. If you want to know more about proxies I suggest reading this post from the ADO.Net team blog. Figure 1.0 depicts debug data of a materialized proxy Entity in a C# library. Notice the unusual name of the proxy Entity in the watch window in figure 1.0 – the Entity Framework generates proxy types in the System.Data.Entity.DynamicProxies namespace.

Figure 1.0

At this point, we know we are not seeing proxy type names come back when we ask our PODO for it’s full name and we are not getting any love from Lazy Loading at all. Our materialized Entities are indeed the type as engineered via __clrtype__ and we saw this when querying for the namespace of the Entity from IronPython in part 2.0.

In a nutshell, the Entity Framework is querying the database and materializing our PODO’s, however given the circumstances its reasonable to argue that no proxying is taking place and hence no lazy loading. Let’s put that theory to the test just a little more shall we! When proxies are employed in the Entity Framework there are two basic usages; proxies that support either Lazy Loading only or Change Tracking with Lazy Loading, the latter option requires all mapped properties be declared virtual and be public or protected. In Python class properties, functions and methods are considered to be as good as virtual, so we will just assume for the moment that this clears the way for the Entity Framework to work with our PODO’s as any other normal compliant class that is proxyable (of course there is no such word).

Since we are getting no proxy generation with our PODO’s then we would equally expect change tracking to fail would we not? Note in figure 1.0 that a (C#) proxy when examined has a reference to the Entity Tracker and other Entity Framework plumbing required to manage it, which we would expect to be missing if our theory about what’s happening to our PODO’s is correct. Let’s now put the whole proxy and change tracking question to rest.

def IsProxy(theType):
    ctxType = ObjectContext.GetObjectType(theType.GetType())
    notNull = theType != None
    isCtxObj = ctxType != theType.GetType();
    print “\tThis entity is a %(podo)s“ \
                %{‘podo‘:theType.GetType()}
    print “\tIt’s type in the ObjectContext is %(ctxType)s“ \
                %{‘ctxType‘:ctxType}
    return notNull and isCtxObj

def IsChangeTracking(podo):
    state = ctx.ObjectStateManager.GetObjectStateEntry(podo).State
    print “\tIt’s state is: %(state)s“ % {‘state‘:state}
    return state == System.Data.EntityState.Modified

ctx.ProxyCreationEnabled  = True

parents = ctx.ParentRecords\
             .Where(“it.ParentID != ‘bob’“)\
             .Include(“ChildRecords“)

for parent in parents:
    print parent.ParentID
    parent.ParentName = “change tracking works nonetheless“
    isProxy = IsProxy(parent).ToString()
    isChange = IsChangeTracking(parent).ToString()
    print “\tIt is %(proxy)s that this PODO is a proxy“ \
                %{‘proxy‘:isProxy}
    print “\tIt is %(change)s that this PODO supports “ \
                “proxy style change tracking“ %{‘change‘:isChange}
    if parent.ChildRecords != None:
        for child in parent.ChildRecords:
            print “\t\tChildID is : “ + child.ChildID

ctx.SaveChanges()

We have added two methods, one to check if an Entity is a proxy and another to verify if change tracking is working via our proxy. Running this code produces the following TSQL, subsequent changes to the database and debug window output in NetBeans.

exec sp_executesql N‘update [dbo].[ParentRecords]
set [ParentName] = @0
where ([ParentID] = @1)
‘,N‘@0 nvarchar(50),@1 char(5)’,@0=N‘change tracking’,@1=‘FGHIJ’

An interesting set of results. As suspected we have no proxies with Lazy Loading or Change tracking ability but we do have change tracking nonetheless when we get a diff from the ObjectContext. The call to .SaveChanges() on the ObjectContext is still persisting the change to the database regardless of the absence of the proxy creation.

To summarise, we now have established that loading data into PODO’s works despite producing no proxy. We can eager load only and we still have the rather large problem with natives that are not strings; this still leaves us with the little problem of this ugly message.

One thing that I meant to mention in one of the earlier posts is that clrtype.py does not support the use of iterables and IronPython classes in the @Accepts and @Returns decorators out of the box. Thanks to help given by Lukas Cenovsky on the IronPython mailing list who made the required changes to the validate_clr_types method in clrtype.py to get over this hurdle.

So now it’s time I thought to jump into VS 2010 and see what we can discover by checking out the dump file created by the IPY.exe crash. And that is where part 4.0 of the series will start. Until then.