Living in the Tech Avalanche Generation

BizTalk provides us with a function known as Recoverable Interchange Processing and if you aren’t familiar with the concept of an ‘Interchange’, it’s really about the granularity of messages. For example lets say I have messages that contain distinct messages within them, i.e. a collection of contained messages, this is known as an interchange. A simple singular message is also technically a complete interchange in its own right. In both scenarios each message will carry the same interchange ID but where an envelope contains multiple messages each message will receive it’s own message ID. Multi message interchanges are generally split up by a custom pipeline component but it is possible to use configuration to split these messages by employing the XmlDisassembler to break apart the messages. For more discussion on this please check Richards post. I should point out at this point that the rest of this post documents what I believe may be a bug in BTS 2006 and assumes BizTalk knowledge with regard to schema definitions specifically envelope schema’s that contain multiple messages.

Let’s say I have an Envelope Schema and wish to separate out the good messages contained within the envelope from the bad and have them processed, whilst the bad messages get suspended and eventually dealt with. Ok so lets use recoverable interchange processing. So here’s what I noticed recently given the following messages:

<?xml version=”1.0″ encoding=”utf-8″ ?>
<ns0:People xmlns:ns0=”http://RecoverableInterchange.People”>
    <ns1:Person xmlns:ns1=”http://RecoverableInterchange.Person”>
        <firstName>simon</firstName>
        <lastName>segal</lastName>
    </ns1:Person>
    <ns1:Person xmlns:ns1=”http://RecoverableInterchange.Person”>
        <firstName>mark</firstName>
        <lastName>harris</lastName>
    </ns1:Person>
    <ns1:Person xmlns:ns1=”http://RecoverableInterchange.Person”>
        <firstName>steve</firstName>
        <lastName>cassidy</lastName>
    </ns1:Person>
</ns0:People>

which conforms to the following message schema’s

<?xml version=”1.0″ encoding=”utf-16″?>
<xs:schema xmlns:ns0=”http://RecoverableInterchange.Person”
           xmlns:b=”http://schemas.microsoft.com/BizTalk/2003″
           xmlns=”http://RecoverableInterchange.People”
           targetNamespace=”http://RecoverableInterchange.People”
           xmlns:xs=”http://www.w3.org/2001/XMLSchema”>
  <xs:import schemaLocation=”.\person.xsd”
             namespace=”http://RecoverableInterchange.Person” />
  <xs:annotation>
    <xs:appinfo>
      <b:schemaInfo is_envelope=”yes”
                    xmlns:b=”http://schemas.microsoft.com/BizTalk/2003″ />
      <b:references>
        <b:reference
            targetNamespace=”http://RecoverableInterchange.Person” />
      </b:references>
    </xs:appinfo>
  </xs:annotation>
  <xs:element name=”People”>
    <xs:annotation>
      <xs:appinfo>
        <b:recordInfo
            body_xpath=”/*[local-name()='People' and
                namespace-uri()='http://RecoverableInterchange.People']“ />
      </xs:appinfo>
    </xs:annotation>
    <xs:complexType>
      <xs:sequence>
        <xs:element minOccurs=”1″
                    maxOccurs=”unbounded”
                    ref=”ns0:Person” />
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>

Note the underline section above: this allows BizTalk to identify this message type to the receive port pipeline’s XmlDisassembler.

The Envelope schema imports the following schema

<?xml version=”1.0″ encoding=”utf-16″?>
<xs:schema xmlns:b=”http://schemas.microsoft.com/BizTalk/2003″
           xmlns=”http://RecoverableInterchange.Person”
           targetNamespace=”http://RecoverableInterchange.Person”
           xmlns:xs=”http://www.w3.org/2001/XMLSchema”>
  <xs:element name=”Person” type=”Person” />
  <xs:complexType name=”Person”>
    <xs:sequence>
      <xs:element name=”firstName” type=”xs:string” />
      <xs:element name=”lastName” type=”xs:string” />
    </xs:sequence>
  </xs:complexType>
</xs:schema>

Now given these schemas and the example message instance, we have a perfectly good envelope schema with a multi-part message instance that should present no issues given I have the following subscriptions:

The RipSendPortOk port has the following filter:

BTS.ReceivePortName == RipReceivePort

Therefore, given the following message instance, we would expect two valid messages and one invalid message (see the bold green node) to produce an error when the PersonBad node is encountered. The problem here is that the message does not conform to the Person schema imported into our envelope message schema.

<?xml version=”1.0″ encoding=”utf-8″ ?>
<ns0:People xmlns:ns0=”http://RecoverableInterchange.People”>
    <ns1:Person xmlns:ns1=”http://RecoverableInterchange.Person”>
        <firstName>simon</firstName>
        <lastName>segal</lastName>
    </ns1:Person>
    <ns1:PersonBad xmlns:ns1=”http://RecoverableInterchange.Person”>
        <firstName>mark</firstName>
        <lastName>harris</lastName>
    </ns1:PersonBad>
    <ns1:Person xmlns:ns1=”http://RecoverableInterchange.Person”>
        <firstName>steve</firstName>
        <lastName>cassidy</lastName>
    </ns1:Person>
</ns0:People>

Ok, so no surprises here, recoverable interchange is working exactly as expected.

Next I try the following message, that exhibits an altogether different problem, this time it represents malformed XML and is non-schema conforming. Note the closing People node <//ns:0People>, that contains two forward slashes and not one.

<?xml version=”1.0″ encoding=”utf-8″ ?>
<ns0:People xmlns:ns0=”http://RecoverableInterchange.People”>
    <ns1:Person xmlns:ns1=”http://RecoverableInterchange.Person”>
        <firstName>simon</firstName>
        <lastName>malformed</lastName>
    </ns1:Person>
    <ns1:Person xmlns:ns1=”http://RecoverableInterchange.Person”>
        <firstName>mark</firstName>
        <lastName>malformed</lastName>
    </ns1:Person>
    <ns1:Person xmlns:ns1=”http://RecoverableInterchange.Person”>
        <firstName>steve</firstName>
        <lastName>malformed</lastName>
    </ns1:Person>
<//ns0:People>

This results in a suspended / resumable message, which is the expected behaviour.

Our next and final message is as follows:

<?xml version=”1.0″ encoding=”utf-8″ ?>
<?xml version=”1.0″ encoding=”utf-8″ ?>
<ns0:People xmlns:ns0=”http://RecoverableInterchange.People”>
    <ns1:Person xmlns:ns1=”http://RecoverableInterchange.Person”>
        <firstName>simon</firstName>
        <lastName>malformed</lastName>
    </ns1:Person>
    <ns1:Person xmlns:ns1=”http://RecoverableInterchange.Person”>
        <firstName>mark</firstName>
        <lastName>malformed</lastName>
    </ns1:Person>
    <ns1:Person xmlns:ns1=”http://RecoverableInterchange.Person”>
        <firstName>steve</firstName>
        <lastName>malformed</lastName>
    </ns1:Person>
</ns0:People>

What’s wrong with this you ask? Look closely; yep it’s got duplicate XML declarations at the very beginning. Now if your like me then your thinking that this interchange is going to fail as a whole and therefore all three contained messages will be suspended. Guess again! When this message comes into port, it is indeed pick out as being BAD and does in fact get suspended. The Event Viewer reports the following two errors which clearly show that the XML disassembler in the pipeline cannot make sense of the message and match it to anything expected.

Looking at this you might expect that the malformed XML wont clear the gate keeper and the entire envelope will be suspended (that would have been my guess). Even though the messages contained within are correct per se, the entire original message is not valid XML, so nothing doing in gaining any of the Recoverable Interchange benefits cause the whole message is considered rubbish in this particular case (after all it’s invalid XML so fair enough). Next thing I tried was to resume this suspended message which should duly be expected to fail and become promptly re-suspended but you guessed it, no chocolates. Looking further into the issue I discovered that my message has actually fallen into a BizTalk black hole altogether and I have no evidence of my message at all except for the following two errors reported in the event viewer.

First error message indicates that the error occurred again but suffered a routing failure and then the second message tells us that an error occurred whilst trying to re-suspend the message. Unfortunately what none of this tells us that BizTalk has also gone and deleted the message and just when I wanted to recover failed messages! Upon further investigation it turns out that I had incorrectly set both subscribing send ports with exactly the same filter predicates.

What I should have done was setup one send port subscription on the receive ports name and another send port subscribing to errors on the same receive port. What I actually had done mistakenly was to set both send ports subscription filters identically and it would seem that when combined with enabling message routing for failed messages, will produce this error. Beware the combination of malformed XML, enabled routing for failed messages and RIP or your message might not Rest In Peace! Finally I want to mention that the example envelope and messages in the the RIP scenario scenario presented here is based on the example provided by Richard Blewett on his blog.

I am spending more and more time with my head wrapped around BizTalk in one way or another and very recently I am glad to say that I successfully navigated the MCTS exam and certified as a BizTalk specialist. Does this mean that I expect to do a lot of work in BizTalk from this point on? Good question. Well I do expect to do some yes and I am hoping that I will be able to offer the measured position of being from the camp that does not reach for BizTalk to solve every SOA problem looking for a solution.

There are alternatives

I do appreciate many of the features of BTS 2006 but I must confess that sometimes I do hear some absolutely mad assertions coming from those who would throw BizTalk at every problem where messaging was a clear choice. I have heard voices hold forth that ALL business rules should be managed in one central location (in BizTalk) and that all applications and line of business applications should make BizTalk their messaging target endpoint. I remember  objecting to this idea firstly on the grounds that it was out of alignment with the budget constraints of a given project and secondly that it was clearly propagating fallacy number eleven of the fallacies of distributed computing. Now in this particular case my argument was pretty solid but I should also point out that it’s not always that cut and dry. BizTalk may well be the right architectural and financial choice in many cases where we know that scaling out on mass wont be a future consideration coupled with a short delivery window imperative or perhaps a business lacking in the capability to maintain a considerable software team required to maintain a complex custom SOA application. This does not exhaust the list no doubt but gives an idea.

SOA without the bells and whistles

I may want to build my own very light weight SOA framework or use NServiceBus, either option affording me the option to deploy multiple versions of the same service at different geographical points across my LAN. In this scenario my rules should stay with my service, I don’t fancy the idea of wearing the expense of BTS for every long running (or otherwise) service I write and I have all those nicely designed domain entities that have come out of my having been Domain Driven all along the way! Heck I may even be using WCF and WF in the most out of the box fashion and even then it makes very little sense to put all my business rules in one place just because I can. Centralizing things in my architecture expose me to a single point of failure, which we know isn’t a great idea. Some will argue that it’s easier to maintain in one place and if our environment is lacking in complexity then there are times where that does hold true, but over time as developers and architects begin to pile things into one place things might start to unravel. Imagine a BizTalk environment that begins to exhibit as a massive repository for your organisations business rules! Hmm, if I change this one rule here what happens to all the logic that uses my other 1300 rules and which one of them have dependencies on the rule I just changed? I should clarify here: I am not ready to throw out the baby with the bath water just yet however, I wouldn’t even begin to consider for a moment the notion that I should build tools for mapping or adapters for some of the LOB applications out there, these very mature aspects of BizTalk can exist very nicely behind my services rather at the front of my design.

Figure 1.0

Oslo is certainly suggesting architectural decisions like that exemplified above, where the BizTalk hosting environment continue to be used for integration services and the ‘Dublin’ Server be used for application services. This distinction has me curious; will Oslo’s process server be similar to BizTalk with respect to being a Broker? In the meantime as I become more adept at using the product, I feel more at peace with making up my mind about why, when and where BizTalk makes sense.

BizTalk Express - Oh well got the name wrong - they called it Dublin. I want to make this the last post where I use the term BizTalk Express.

PDC is on our doorstep and the last minute information leak is starting to turn to a  steady stream, the one that most interested me recently came from Darren Jefford just a few days ago. Darren speaks about how Dublin (the new WCF / WF Application Server Host) will help in making both those technologies scale to enterprise without developers having to write a whole lot of infrastructure code. Darren say’s….

“So if you want to expose [WF] workflow’s via [WCF] services but ensure performance and scalability (up to enterprise scale), you can now do this without having to write the code required to host these apps on Windows Server.  Ensuring performance and scale of WCF services and WF is hard to do today, hence it’s not done very often at least in my experience and sometimes causes a tendency to twist BizTalk into doing something it wasn’t necessarily designed to do which causes problems of their own (coupling Web Sites/UI’s directly to BizTalk for synchronous processing springs to mind).

We don’t want customers in this situation to be forced into writing huge amounts of hard plumbing code to achieve this, we need a server product to do this for you, which is where Dublin comes in.  Note some of the server features announced which will be familiar to BizTalk developers (content based routing, compensation, etc.).”

So it would appear that some of the benefits of BizTalk are being borrowed and leveraged to provide application developers the ‘Server’ or ‘Host’ product and even some content based routing (both requests in the call for BizTalk Express). However part of my previous assertions was that by providing BizTalk Express (conceptually), we might become the beneficiaries of a robust and scalable host for our messaging and long running workflow technologies, there is still one point that I maintain is of significant importance that should not be overlooked and that is the requirement for Publish / Subscribe. So I am still left with this question dangling:

“How deep will all this declarative tooling go? How far does Dublin reach down into the API? Do I get publish / subscribe with my WCF / WF messages / workflow’s? What about durable messaging regardless of transport and bindings - I don’t care to lose messages on the HTTP stack?”

Whilst the information seems to be coming a little thicker and faster, some of these questions remain unanswered. I am also very interested in how flexible these new shiny toys will be? I for one don’t particularly like the RPC / CRUD styled interfaces that you see being developed ad nauseam in the Web Service space, that approach does not feel at all like messaging to me - it’s more like remote RPC API calls than it is messaging. Does Dublin and the Oslo suite of tools offer me the environment to take advantage of sending business messages or will it continue to encourage RPC styled API calls with better hosting and scale features out of the box?

I’m sure there is a lot more to the Oslo story that will be unveiled in the next month and I remain hopeful that some of my wish list is on the menu.