Living in the Tech Avalanche Generation

I want to create a Domain Specific Language for a product that specifies a published MS Access Schema as it’s generalized medium for all ETL processing between external systems and the product suits flagship product (a document management system). To do this I need to be able to express concepts through the DSL that will perform CRUD operations over this ETL schema and therefore MS Access Databases. I also want to employ an ORM to achieve the data access yet sadly to do this I am still stuck with plain old ADO.Net considering that LINQ TO SQL and the Entity Framework don’t support Access and I have experienced the pain of trying to achieve this with ActiveRecord so I am left asking myself this question:

“Will Microsoft or anyone else for that matter, provide us with an ORM product that works nicely with Microsoft Access and doesn’t hit my wallet”

I am not a big Access fan per se, in fact if you ask those that know me well they would probably get a good laugh out of the fact that I am working with it, however it surely hasn’t escaped the mass of business’ developers who have to deal with it on a daily basis, that they have to this point been ignored with regards to support from the new ORM products. Let’s hope that someone helps us out with an Entity Framework provider soon, LINQ To SQL I am pretty sure would be a complete no go zone in light of recent announcements.

BizTalk provides us with a function known as Recoverable Interchange Processing and if you aren’t familiar with the concept of an ‘Interchange’, it’s really about the granularity of messages. For example lets say I have messages that contain distinct messages within them, i.e. a collection of contained messages, this is known as an interchange. A simple singular message is also technically a complete interchange in its own right. In both scenarios each message will carry the same interchange ID but where an envelope contains multiple messages each message will receive it’s own message ID. Multi message interchanges are generally split up by a custom pipeline component but it is possible to use configuration to split these messages by employing the XmlDisassembler to break apart the messages. For more discussion on this please check Richards post. I should point out at this point that the rest of this post documents what I believe may be a bug in BTS 2006 and assumes BizTalk knowledge with regard to schema definitions specifically envelope schema’s that contain multiple messages.

Let’s say I have an Envelope Schema and wish to separate out the good messages contained within the envelope from the bad and have them processed, whilst the bad messages get suspended and eventually dealt with. Ok so lets use recoverable interchange processing. So here’s what I noticed recently given the following messages:

<?xml version=”1.0″ encoding=”utf-8″ ?>
<ns0:People xmlns:ns0=”http://RecoverableInterchange.People”>
    <ns1:Person xmlns:ns1=”http://RecoverableInterchange.Person”>
        <firstName>simon</firstName>
        <lastName>segal</lastName>
    </ns1:Person>
    <ns1:Person xmlns:ns1=”http://RecoverableInterchange.Person”>
        <firstName>mark</firstName>
        <lastName>harris</lastName>
    </ns1:Person>
    <ns1:Person xmlns:ns1=”http://RecoverableInterchange.Person”>
        <firstName>steve</firstName>
        <lastName>cassidy</lastName>
    </ns1:Person>
</ns0:People>

which conforms to the following message schema’s

<?xml version=”1.0″ encoding=”utf-16″?>
<xs:schema xmlns:ns0=”http://RecoverableInterchange.Person”
           xmlns:b=”http://schemas.microsoft.com/BizTalk/2003″
           xmlns=”http://RecoverableInterchange.People”
           targetNamespace=”http://RecoverableInterchange.People”
           xmlns:xs=”http://www.w3.org/2001/XMLSchema”>
  <xs:import schemaLocation=”.\person.xsd”
             namespace=”http://RecoverableInterchange.Person” />
  <xs:annotation>
    <xs:appinfo>
      <b:schemaInfo is_envelope=”yes”
                    xmlns:b=”http://schemas.microsoft.com/BizTalk/2003″ />
      <b:references>
        <b:reference
            targetNamespace=”http://RecoverableInterchange.Person” />
      </b:references>
    </xs:appinfo>
  </xs:annotation>
  <xs:element name=”People”>
    <xs:annotation>
      <xs:appinfo>
        <b:recordInfo
            body_xpath=”/*[local-name()='People' and
                namespace-uri()='http://RecoverableInterchange.People']“ />
      </xs:appinfo>
    </xs:annotation>
    <xs:complexType>
      <xs:sequence>
        <xs:element minOccurs=”1″
                    maxOccurs=”unbounded”
                    ref=”ns0:Person” />
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>

Note the underline section above: this allows BizTalk to identify this message type to the receive port pipeline’s XmlDisassembler.

The Envelope schema imports the following schema

<?xml version=”1.0″ encoding=”utf-16″?>
<xs:schema xmlns:b=”http://schemas.microsoft.com/BizTalk/2003″
           xmlns=”http://RecoverableInterchange.Person”
           targetNamespace=”http://RecoverableInterchange.Person”
           xmlns:xs=”http://www.w3.org/2001/XMLSchema”>
  <xs:element name=”Person” type=”Person” />
  <xs:complexType name=”Person”>
    <xs:sequence>
      <xs:element name=”firstName” type=”xs:string” />
      <xs:element name=”lastName” type=”xs:string” />
    </xs:sequence>
  </xs:complexType>
</xs:schema>

Now given these schemas and the example message instance, we have a perfectly good envelope schema with a multi-part message instance that should present no issues given I have the following subscriptions:

The RipSendPortOk port has the following filter:

BTS.ReceivePortName == RipReceivePort

Therefore, given the following message instance, we would expect two valid messages and one invalid message (see the bold green node) to produce an error when the PersonBad node is encountered. The problem here is that the message does not conform to the Person schema imported into our envelope message schema.

<?xml version=”1.0″ encoding=”utf-8″ ?>
<ns0:People xmlns:ns0=”http://RecoverableInterchange.People”>
    <ns1:Person xmlns:ns1=”http://RecoverableInterchange.Person”>
        <firstName>simon</firstName>
        <lastName>segal</lastName>
    </ns1:Person>
    <ns1:PersonBad xmlns:ns1=”http://RecoverableInterchange.Person”>
        <firstName>mark</firstName>
        <lastName>harris</lastName>
    </ns1:PersonBad>
    <ns1:Person xmlns:ns1=”http://RecoverableInterchange.Person”>
        <firstName>steve</firstName>
        <lastName>cassidy</lastName>
    </ns1:Person>
</ns0:People>

Ok, so no surprises here, recoverable interchange is working exactly as expected.

Next I try the following message, that exhibits an altogether different problem, this time it represents malformed XML and is non-schema conforming. Note the closing People node <//ns:0People>, that contains two forward slashes and not one.

<?xml version=”1.0″ encoding=”utf-8″ ?>
<ns0:People xmlns:ns0=”http://RecoverableInterchange.People”>
    <ns1:Person xmlns:ns1=”http://RecoverableInterchange.Person”>
        <firstName>simon</firstName>
        <lastName>malformed</lastName>
    </ns1:Person>
    <ns1:Person xmlns:ns1=”http://RecoverableInterchange.Person”>
        <firstName>mark</firstName>
        <lastName>malformed</lastName>
    </ns1:Person>
    <ns1:Person xmlns:ns1=”http://RecoverableInterchange.Person”>
        <firstName>steve</firstName>
        <lastName>malformed</lastName>
    </ns1:Person>
<//ns0:People>

This results in a suspended / resumable message, which is the expected behaviour.

Our next and final message is as follows:

<?xml version=”1.0″ encoding=”utf-8″ ?>
<?xml version=”1.0″ encoding=”utf-8″ ?>
<ns0:People xmlns:ns0=”http://RecoverableInterchange.People”>
    <ns1:Person xmlns:ns1=”http://RecoverableInterchange.Person”>
        <firstName>simon</firstName>
        <lastName>malformed</lastName>
    </ns1:Person>
    <ns1:Person xmlns:ns1=”http://RecoverableInterchange.Person”>
        <firstName>mark</firstName>
        <lastName>malformed</lastName>
    </ns1:Person>
    <ns1:Person xmlns:ns1=”http://RecoverableInterchange.Person”>
        <firstName>steve</firstName>
        <lastName>malformed</lastName>
    </ns1:Person>
</ns0:People>

What’s wrong with this you ask? Look closely; yep it’s got duplicate XML declarations at the very beginning. Now if your like me then your thinking that this interchange is going to fail as a whole and therefore all three contained messages will be suspended. Guess again! When this message comes into port, it is indeed pick out as being BAD and does in fact get suspended. The Event Viewer reports the following two errors which clearly show that the XML disassembler in the pipeline cannot make sense of the message and match it to anything expected.

Looking at this you might expect that the malformed XML wont clear the gate keeper and the entire envelope will be suspended (that would have been my guess). Even though the messages contained within are correct per se, the entire original message is not valid XML, so nothing doing in gaining any of the Recoverable Interchange benefits cause the whole message is considered rubbish in this particular case (after all it’s invalid XML so fair enough). Next thing I tried was to resume this suspended message which should duly be expected to fail and become promptly re-suspended but you guessed it, no chocolates. Looking further into the issue I discovered that my message has actually fallen into a BizTalk black hole altogether and I have no evidence of my message at all except for the following two errors reported in the event viewer.

First error message indicates that the error occurred again but suffered a routing failure and then the second message tells us that an error occurred whilst trying to re-suspend the message. Unfortunately what none of this tells us that BizTalk has also gone and deleted the message and just when I wanted to recover failed messages! Upon further investigation it turns out that I had incorrectly set both subscribing send ports with exactly the same filter predicates.

What I should have done was setup one send port subscription on the receive ports name and another send port subscribing to errors on the same receive port. What I actually had done mistakenly was to set both send ports subscription filters identically and it would seem that when combined with enabling message routing for failed messages, will produce this error. Beware the combination of malformed XML, enabled routing for failed messages and RIP or your message might not Rest In Peace! Finally I want to mention that the example envelope and messages in the the RIP scenario scenario presented here is based on the example provided by Richard Blewett on his blog.

I have the overwhelming feeling that Microsoft will be putting more and more effort into REST, ATOM and a variety of other technologies that do not tow the SOAP / WSDL / Web Service & WS* party lines. This is extremely good for those who felt that all the technology on that stack was becoming overly complex and burdensome and to boot was not adopted by the larger internet players from whom all of us corporate behind the firewall types can learn a lot. Growing up over the past 3 or 4 years it has become more and more pronounced to me that there were these two competing worlds of technology, the one for the internet scale thinkers and doer’s and the other corporate small scale network folks, the later becoming ever so more curious about the black magic being practiced by the likes of Google and Amazon and want to come to terms with why it seemed so different and also saw benefit in borrowing their ideas and applying them to the problems in their environments.

We should take into account that Microsoft provide some highly usable and reliable tools for the corporate world, for example programming against an object (albeit a proxy) generated off a WSDL document, provided the kind of familiar comfort we feel with an old pair of socks.

Enter REST, ATOM and some of their friends and I cant help but notice the change in pitch of the new breed of Microsoft technology development teams. A lot of excitement seems to be being generated around REST and remarks like “keep using SOAP and WSDL if it works for you and your comfortable with it”, only seem to support that notion that the focus is shifting. Throw in the fact that the recruiting at Redmond also seems to be somewhat focused on a blend of great academic minds and community thinkers and celebrities, the noticeable change in culture at Microsoft is palpable. This all very good, very good indeed.