Living in the Tech Avalanche Generation

The most canonical example usage of WCF Transactions (according to Juval) is the following :

A client application wraps (n) service calls in a single transaction and relies on the ACID safe haven of System.Transaction to save the day. In my opinion, one of the core characteristics of WCF is that it’s design favours (and Microsoft like to demonstrate it’s usefulness via) a remote object pattern where developers program against an objects polymorphic signatures i.e. its behavioural aspects, via a proxy. This can be seen when we consider code such as proxy.UpdateAccount(); or proxy.RemoveCustomerPurchase(Guid purchaseID); for example, where proxy is a variable name for an object classed from a CustomerAccount class or more precisely implementing the ICustomerAccount interface and it’s operation contracts. Why am I highlighting this design characteristic? It’s my view that this characterstic has permeated it’s way into the entire design of WCF and it’s impact is felt an many corners of the Platform and affects us for example in the choices we are required to make on concurrency and instancing, however before we get into that let’s look at the example below as per the canonical example.

Figure 1.0

Listing 1.0

public void Call_A_B_C(Guid purchaseId)
{
    //…..create the binding and the addresses
    ChannelFactory<ServiceA> factoryA =
        new ChannelFactory<ServiceA>(bindingA, addressA);
    IServiceA channelA = factoryA.CreateChannel();

    ChannelFactory<ServiceB> factoryB =
        new ChannelFactory<ServiceA>(bindingB, addressB);
    IServiceB channelB = factoryB.CreateChannel();

    ChannelFactory<ServiceC> factoryC =
        new ChannelFactory<ServiceC>(bindingC, addressC);
    IServiceA channelC = factory.CreateChannel();

    using(TransactionScope scope = new TransactionScope())
    {
        channelA.RemoveCustomerPurchase(purchaseId);
        channelB.UpdateInventoryRequest(purchaseId);
        channelC.AmendBillingNotice(purchaseId);
        scope.Commit();
    }

    //…..etc
}

Let’s also assume that our Services A, B and C have the following Transaction flow attributes setup correctly so our client initiated (ambient) transaction flows through to the services.

[ServiceContract]
public interface IServiceA
{
    [OperationContract]
    [TransactionFlow(TransactionFlowOption.Mandatory)]
    public bool RemoveCustomerPurchase(Guid purchaseId);
}

public class ServiceA : IServiceA
{
    [OperationBehaviour(TransactionScopeRequired = true)]
    public bool RemoveCustomerPurchase(Guid purchaseId)
    {
        //…Database access etc…
    }
}

The idea here of course is that calling all three services in a single transaction will make our services behave in a way where the state of our systems (most likely kept in a database repository) will remain consistent in the case of a failure at any point through the transaction - it’s an all or nothing proposition. Combined, this aggregation of requested operations represents a discrete logical business workflow of some kind, where all operations or business actions must succeed or fail as a single ATOMIC transaction.

Again let’s note that these (three) operations (service method calls) are modelled more like traditional business object behaviours and will likely include some database interaction (we will talk about the File IO in figure 1.0 further on) and other potentially long running code. These operations typically implement a request / response message exchange pattern and by lining up all three of these requests inside a transaction and locking resources we are immediately creating a friction with our ability to scale and increasing our chances of encouraging database deadlocks and service  timeouts. Also worth noting is that the default IsolationLevel for the TransactionScope is Serializable which happens to be the most expensive Isolation Level and will lock whatever our service touches until it’s finished doing it’s work.

As a consultant I have seen my share of Web / WCF Services running out there in the wild and it’s pretty common to see them spread over a variety of IIS machines and in some Windows Services running under the SCM and furthermore some of these services are large (have hundreds of operations) and have lots of database access code and they all hit the same tables in the same DB’s in any variety of order. For example, in our example above, it’s not uncommon to see Service A, B and C all accessing the same database and tables and it’s also common for a variety of disparate applications to use all three services independently and not in a universally consistent manner. Lets say we have client X which uses Service A and C and client Y which uses Service A, B and C and client Z which uses Service A only. Imagine that these client applications use any number of permutations in aggregating service method calls within a surrounding transaction; while these long running operations are having distributed locks applied to both the services and database queries, other client requests are waiting and become more likely to timeout.

This approach does not scale. So let’s imagine we are using RepeatableRead for an isolation level (not the default Serializable) and we have multiple clients making distributed transaction calls across services, all the locking going on in the database is going to slow things up and our services will be waiting for locks to release, which in turn will make our clients wait for our services to return responses and gradually our system will grind and timeout, suffering more and more as further users come on line. Their is an underlying complexity with WCF declarative architecture, many configuration options that can be strung together to state an intent about instancing concurrency, ATOMICITY, transport characteristic, security and the list goes on. Sometimes its possible to create a collection of declarative statements that are acceptable however when applied are redundant or countermanding. It’s possible in the scenario we are discussing that service operations under load be queuing up calls and locking out other clients whilst this potentially weighty transaction is occurring across different web servers and databases and should one of them start to cave in under this load then our system is going to spiral down and every part of it will probably become non-responsive. In the meantime who knows how many important messages are being lost on the wire without any store and forward strategy. These are indeed some of the reasons that distributed transactions are a bad idea and that it may be worth considering transactional asynchronous messaging in these scenarios. Store your message and then get it across the wire under a transaction (so it’s not lost) letting each endpoint worry about storing their own messages and processing them under the steam of a different thread than that managing the Service method. You could of course use MSMQ binding but it’s a limiting design choice and if you require HTTP as the transport it’s not viable. Also, if you have to deal with state at the point of handling the message, then Separating the threads or perhaps even the processes for dealing with the concerns of receiving the message and handling messages, makes for a more scalable and performant system, lending itself well to dealing with long living service state. Perhaps your service design includes dealing with various messages over time, where the messages need to be correlated to one another as in a Saga.

Client passes message and flows transaction to >> Service A >> Service A passes message and flows transaction to >> Service B >> Service B passes message and flows transaction to >> Service C. Now if anything goes wrong in the hops between client and services here then we expect everything that happened to be rolled back right? Wrong! With ASMX or WCF out of the box bindings that don’t include MSMQ, there is no durable storage of our messages, so if the client below in Figure 2.0 sends an order, something goes wrong somewhere and we roll back, we just lost our message (hope it wasn’t an order right?). Furthermore, in Figure 2.0 we could add more clients which use any one of the services depicted exclusively, for example Client B comes along and calls Service C only, meanwhile the depicted client (A) is making an distributed transaction across aggregated calls to Service A, B and C where it currently is waiting for Service A to complete its work. Service A (on behalf of client A) is accessing the same database table as Service C (on behalf of client B) - there is a deadlock in the database, a victim is chosen and someone rolls back, however whilst they were blocking and the transactions were open on our Per Call, Single entrant WCF Services, all the other clients requesting work were held up and waiting and potentially timed out as well.

Figure 2.0

It’s generally a good idea to keep your transactions in WCF open for as short a period of time as possible and that’s why using WCF as pure message transport (i.e. absolutely no business logic or database interaction) can help you achieve better throughput, scale and be more fault resistant. Transport your messages transactionally from queues (where the message originates) and to other queues located at the message destination, this will protect against lost messages; it’s possible to use WCF entirely as the transport protocol enabler of choice and not follow the CRUD RPC styled messaging pattern.

Figure 3.0

Because we read the message off the originating queue and send it to target queue within the scope of a transaction, our message will simply roll back of there’s a problem. If we pursue the concepts outlined in Figure 3.0 to their conclusion, we might consider MSMQ or SQL Service Broker to provide us with our Queues and Service Broker does offer us some potential Smart Client benefits in being so close to our endpoints in this design scenario. Let’s be clear however, we are proposing using Service Broker for it’s queues and only it’s ability to provide queues, we do not propose to build services, contracts et al in Service Broker, thus avoiding any ensuing licensing issues which Service Broker will create if you implement your services using that technology.

BizTalk Express - Oh well got the name wrong - they called it Dublin. I want to make this the last post where I use the term BizTalk Express.

PDC is on our doorstep and the last minute information leak is starting to turn to a  steady stream, the one that most interested me recently came from Darren Jefford just a few days ago. Darren speaks about how Dublin (the new WCF / WF Application Server Host) will help in making both those technologies scale to enterprise without developers having to write a whole lot of infrastructure code. Darren say’s….

“So if you want to expose [WF] workflow’s via [WCF] services but ensure performance and scalability (up to enterprise scale), you can now do this without having to write the code required to host these apps on Windows Server.  Ensuring performance and scale of WCF services and WF is hard to do today, hence it’s not done very often at least in my experience and sometimes causes a tendency to twist BizTalk into doing something it wasn’t necessarily designed to do which causes problems of their own (coupling Web Sites/UI’s directly to BizTalk for synchronous processing springs to mind).

We don’t want customers in this situation to be forced into writing huge amounts of hard plumbing code to achieve this, we need a server product to do this for you, which is where Dublin comes in.  Note some of the server features announced which will be familiar to BizTalk developers (content based routing, compensation, etc.).”

So it would appear that some of the benefits of BizTalk are being borrowed and leveraged to provide application developers the ‘Server’ or ‘Host’ product and even some content based routing (both requests in the call for BizTalk Express). However part of my previous assertions was that by providing BizTalk Express (conceptually), we might become the beneficiaries of a robust and scalable host for our messaging and long running workflow technologies, there is still one point that I maintain is of significant importance that should not be overlooked and that is the requirement for Publish / Subscribe. So I am still left with this question dangling:

“How deep will all this declarative tooling go? How far does Dublin reach down into the API? Do I get publish / subscribe with my WCF / WF messages / workflow’s? What about durable messaging regardless of transport and bindings - I don’t care to lose messages on the HTTP stack?”

Whilst the information seems to be coming a little thicker and faster, some of these questions remain unanswered. I am also very interested in how flexible these new shiny toys will be? I for one don’t particularly like the RPC / CRUD styled interfaces that you see being developed ad nauseam in the Web Service space, that approach does not feel at all like messaging to me - it’s more like remote RPC API calls than it is messaging. Does Dublin and the Oslo suite of tools offer me the environment to take advantage of sending business messages or will it continue to encourage RPC styled API calls with better hosting and scale features out of the box?

I’m sure there is a lot more to the Oslo story that will be unveiled in the next month and I remain hopeful that some of my wish list is on the menu.

PDC announcements have been made that begin to shed some light (small aperture stuff) on the whole mystery on OSLO. However before I get into discussing what might become apparent at the PDC let me digress a moment into a discussion on what constitutes an Enterprise and whether SOA is exclusively a valid Enterprise concept.

What constitutes an Enterprise?

3 Developers, 80 staff, 12 million per annum revenue == Enterprise ?

5 Developers, 120 staff, 17 million per annum revenue == Enterprise ?

7 Developers, 200 staff, 22 million per annum revenue == Enterprise ?

10, Developers, 300 staff, 50 million per annum revenue == Enterprise ?

15 Developers, 1000 staff, etc ?

What’s interesting to me is that you will find development teams and managers in any of the categories above, that want to connect applications in the organisation  (enterprise or whatever you want to call it) and believe that they are equally entitled to be thinking about SOA as the CIO of any large Telco or public / private Utility.

Recently I posted a suggestion that if BizTalk had an express version, then it would have "crossed the chasm" en masse some time ago. The question I pose now is what are Microsoft’s plan for BizTalk licensing given the positioning they have given the product within the context of Oslo? Based on the official documentation to date and working on the assumption that there is no change to the status quo to the licensing strategy around BizTalk, then Oslo seems to be limiting it’s audience somewhat. Quite a few of the organisations (Enterprise) listed above, will have a hard time justifying the cost of BizTalk licensing and will be faced with finding alternatives. Let’s face it, there are more businesses in these categories above than any other and it’s also representative of the vast majority of Microsoft’s constituency, so the questions have to be asked, just how central to Oslo will BizTalk be and what will it cost? I note that the roadmap does not include answers to these questions and I dare say that the PDC wont devote a lot of attention to it either but here’s hoping.

The technology that lies at the heart of Oslo is being listed as BizTalk Server V6, System Centre V5, Visual Studio V10, BizTalk Services V1, and the .NET 4.0  Framework. Oslo seems constrained to providing updates to existing technology and providing some new ones, both forming the parts in creating your own SOA soup. Certainly there seems to be no mention or evidence of a framework per se and I cant help but wonder if this isn’t one of the missing elements in their general approach. Certainly it has opened up the way for both open source and 3rd party commercial frameworks / products such as nServiceBus, Simple Service Bus, Neudesic Neuron ESB, Mass Transit and Linxter amongst them. I know nServiceBus reasonable well and it  addresses some of the key problems when faced with building a fluent service based eco system in your organisation and whilst it is early days yet, I can help but notice that Redmond would appear to be taking a heavily technological approach to the problem space. My hope is that the idea of messaging being about web services ‘everywhere’ with a whole lot of orchestration, becomes consigned to the past and that we become the beneficiaries of real pub sub with WCF providing the message transports and perhaps even BizTalk Express filling in the durable pub / sub piece of the architecture?

Some of the things imagineered into Oslo so far are BizTalk deployment of Workflow to Server farms, BAM type monitoring of Workflow, more visual Workflow design tools and an online repository for storage of message contracts , workflow’s, service publications /subscriptions and discovery (ciao UDDI?).

I look forward to seeing what the PDC will unveil and hopefully Microsoft will keep the process open to some rich community input.