Living in the Tech Avalanche Generation

I have the overwhelming feeling that Microsoft will be putting more and more effort into REST, ATOM and a variety of other technologies that do not tow the SOAP / WSDL / Web Service & WS* party lines. This is extremely good for those who felt that all the technology on that stack was becoming overly complex and burdensome and to boot was not adopted by the larger internet players from whom all of us corporate behind the firewall types can learn a lot. Growing up over the past 3 or 4 years it has become more and more pronounced to me that there were these two competing worlds of technology, the one for the internet scale thinkers and doer’s and the other corporate small scale network folks, the later becoming ever so more curious about the black magic being practiced by the likes of Google and Amazon and want to come to terms with why it seemed so different and also saw benefit in borrowing their ideas and applying them to the problems in their environments.

We should take into account that Microsoft provide some highly usable and reliable tools for the corporate world, for example programming against an object (albeit a proxy) generated off a WSDL document, provided the kind of familiar comfort we feel with an old pair of socks.

Enter REST, ATOM and some of their friends and I cant help but notice the change in pitch of the new breed of Microsoft technology development teams. A lot of excitement seems to be being generated around REST and remarks like “keep using SOAP and WSDL if it works for you and your comfortable with it”, only seem to support that notion that the focus is shifting. Throw in the fact that the recruiting at Redmond also seems to be somewhat focused on a blend of great academic minds and community thinkers and celebrities, the noticeable change in culture at Microsoft is palpable. This all very good, very good indeed.

The most canonical example usage of WCF Transactions (according to Juval) is the following :

A client application wraps (n) service calls in a single transaction and relies on the ACID safe haven of System.Transaction to save the day. In my opinion, one of the core characteristics of WCF is that it’s design favours (and Microsoft like to demonstrate it’s usefulness via) a remote object pattern where developers program against an objects polymorphic signatures i.e. its behavioural aspects, via a proxy. This can be seen when we consider code such as proxy.UpdateAccount(); or proxy.RemoveCustomerPurchase(Guid purchaseID); for example, where proxy is a variable name for an object classed from a CustomerAccount class or more precisely implementing the ICustomerAccount interface and it’s operation contracts. Why am I highlighting this design characteristic? It’s my view that this characterstic has permeated it’s way into the entire design of WCF and it’s impact is felt an many corners of the Platform and affects us for example in the choices we are required to make on concurrency and instancing, however before we get into that let’s look at the example below as per the canonical example.

Figure 1.0

Listing 1.0

public void Call_A_B_C(Guid purchaseId)
{
    //…..create the binding and the addresses
    ChannelFactory<ServiceA> factoryA =
        new ChannelFactory<ServiceA>(bindingA, addressA);
    IServiceA channelA = factoryA.CreateChannel();

    ChannelFactory<ServiceB> factoryB =
        new ChannelFactory<ServiceA>(bindingB, addressB);
    IServiceB channelB = factoryB.CreateChannel();

    ChannelFactory<ServiceC> factoryC =
        new ChannelFactory<ServiceC>(bindingC, addressC);
    IServiceA channelC = factory.CreateChannel();

    using(TransactionScope scope = new TransactionScope())
    {
        channelA.RemoveCustomerPurchase(purchaseId);
        channelB.UpdateInventoryRequest(purchaseId);
        channelC.AmendBillingNotice(purchaseId);
        scope.Commit();
    }

    //…..etc
}

Let’s also assume that our Services A, B and C have the following Transaction flow attributes setup correctly so our client initiated (ambient) transaction flows through to the services.

[ServiceContract]
public interface IServiceA
{
    [OperationContract]
    [TransactionFlow(TransactionFlowOption.Mandatory)]
    public bool RemoveCustomerPurchase(Guid purchaseId);
}

public class ServiceA : IServiceA
{
    [OperationBehaviour(TransactionScopeRequired = true)]
    public bool RemoveCustomerPurchase(Guid purchaseId)
    {
        //…Database access etc…
    }
}

The idea here of course is that calling all three services in a single transaction will make our services behave in a way where the state of our systems (most likely kept in a database repository) will remain consistent in the case of a failure at any point through the transaction - it’s an all or nothing proposition. Combined, this aggregation of requested operations represents a discrete logical business workflow of some kind, where all operations or business actions must succeed or fail as a single ATOMIC transaction.

Again let’s note that these (three) operations (service method calls) are modelled more like traditional business object behaviours and will likely include some database interaction (we will talk about the File IO in figure 1.0 further on) and other potentially long running code. These operations typically implement a request / response message exchange pattern and by lining up all three of these requests inside a transaction and locking resources we are immediately creating a friction with our ability to scale and increasing our chances of encouraging database deadlocks and service  timeouts. Also worth noting is that the default IsolationLevel for the TransactionScope is Serializable which happens to be the most expensive Isolation Level and will lock whatever our service touches until it’s finished doing it’s work.

As a consultant I have seen my share of Web / WCF Services running out there in the wild and it’s pretty common to see them spread over a variety of IIS machines and in some Windows Services running under the SCM and furthermore some of these services are large (have hundreds of operations) and have lots of database access code and they all hit the same tables in the same DB’s in any variety of order. For example, in our example above, it’s not uncommon to see Service A, B and C all accessing the same database and tables and it’s also common for a variety of disparate applications to use all three services independently and not in a universally consistent manner. Lets say we have client X which uses Service A and C and client Y which uses Service A, B and C and client Z which uses Service A only. Imagine that these client applications use any number of permutations in aggregating service method calls within a surrounding transaction; while these long running operations are having distributed locks applied to both the services and database queries, other client requests are waiting and become more likely to timeout.

This approach does not scale. So let’s imagine we are using RepeatableRead for an isolation level (not the default Serializable) and we have multiple clients making distributed transaction calls across services, all the locking going on in the database is going to slow things up and our services will be waiting for locks to release, which in turn will make our clients wait for our services to return responses and gradually our system will grind and timeout, suffering more and more as further users come on line. Their is an underlying complexity with WCF declarative architecture, many configuration options that can be strung together to state an intent about instancing concurrency, ATOMICITY, transport characteristic, security and the list goes on. Sometimes its possible to create a collection of declarative statements that are acceptable however when applied are redundant or countermanding. It’s possible in the scenario we are discussing that service operations under load be queuing up calls and locking out other clients whilst this potentially weighty transaction is occurring across different web servers and databases and should one of them start to cave in under this load then our system is going to spiral down and every part of it will probably become non-responsive. In the meantime who knows how many important messages are being lost on the wire without any store and forward strategy. These are indeed some of the reasons that distributed transactions are a bad idea and that it may be worth considering transactional asynchronous messaging in these scenarios. Store your message and then get it across the wire under a transaction (so it’s not lost) letting each endpoint worry about storing their own messages and processing them under the steam of a different thread than that managing the Service method. You could of course use MSMQ binding but it’s a limiting design choice and if you require HTTP as the transport it’s not viable. Also, if you have to deal with state at the point of handling the message, then Separating the threads or perhaps even the processes for dealing with the concerns of receiving the message and handling messages, makes for a more scalable and performant system, lending itself well to dealing with long living service state. Perhaps your service design includes dealing with various messages over time, where the messages need to be correlated to one another as in a Saga.

Client passes message and flows transaction to >> Service A >> Service A passes message and flows transaction to >> Service B >> Service B passes message and flows transaction to >> Service C. Now if anything goes wrong in the hops between client and services here then we expect everything that happened to be rolled back right? Wrong! With ASMX or WCF out of the box bindings that don’t include MSMQ, there is no durable storage of our messages, so if the client below in Figure 2.0 sends an order, something goes wrong somewhere and we roll back, we just lost our message (hope it wasn’t an order right?). Furthermore, in Figure 2.0 we could add more clients which use any one of the services depicted exclusively, for example Client B comes along and calls Service C only, meanwhile the depicted client (A) is making an distributed transaction across aggregated calls to Service A, B and C where it currently is waiting for Service A to complete its work. Service A (on behalf of client A) is accessing the same database table as Service C (on behalf of client B) - there is a deadlock in the database, a victim is chosen and someone rolls back, however whilst they were blocking and the transactions were open on our Per Call, Single entrant WCF Services, all the other clients requesting work were held up and waiting and potentially timed out as well.

Figure 2.0

It’s generally a good idea to keep your transactions in WCF open for as short a period of time as possible and that’s why using WCF as pure message transport (i.e. absolutely no business logic or database interaction) can help you achieve better throughput, scale and be more fault resistant. Transport your messages transactionally from queues (where the message originates) and to other queues located at the message destination, this will protect against lost messages; it’s possible to use WCF entirely as the transport protocol enabler of choice and not follow the CRUD RPC styled messaging pattern.

Figure 3.0

Because we read the message off the originating queue and send it to target queue within the scope of a transaction, our message will simply roll back of there’s a problem. If we pursue the concepts outlined in Figure 3.0 to their conclusion, we might consider MSMQ or SQL Service Broker to provide us with our Queues and Service Broker does offer us some potential Smart Client benefits in being so close to our endpoints in this design scenario. Let’s be clear however, we are proposing using Service Broker for it’s queues and only it’s ability to provide queues, we do not propose to build services, contracts et al in Service Broker, thus avoiding any ensuing licensing issues which Service Broker will create if you implement your services using that technology.

One of the long time criticisms of WCF and ASMX web services has been the lacking of support for Publish / Subscribe. Yes I know that WS-* includes specification(s) to enable this, however if you can show me where on this wonderful planet of ours exists some decipherable prescribed guidance on putting together a store and forward architecture with durable messaging and discoverable endpoints with pub / sub capabilities, I would be very grateful. In the meantime, something that some of my colleagues and I are now considering is how we can leverage the little known and equally poorly documented System.Net.PeerToPeer library in helping us put a reliable messaging framework together that will carry messages over any of the WCF transports available. Now it’s true that WCF includes a Peer to Peer binding that leverages this library, however I have something else in mind. At this stage we envisage a design where we can deploy WCF endpoints across our network and have them use PNRP to advertise message contracts that they publish for other peers (services) to subscribe. This is very exciting and potentially removes any need to use a registry like UDDI or some home baked thing.

This wont be enough by itself however, issues such as durable messaging, the ability to support idempotent message handling and long running correlated workflow’s, all will require some more infrastructure code that we are going to have to write. The good news is that with some considerable work all these things are possible. My friends and I are planning on using SQL Server Express (Service Broker for it’s queue’s only) at the endpoints to serve a durable store, this also doubles nicely with smart client requirements should they exist in any applications that leverage such a framework. We also plan to utilise Workflow Foundation to manage the long running processes (state machine’s) inside the service. By service I am NOT referring to the WCF endpoints. The endpoints are simple entry & exit points for messages and it’s the services that host the endpoints and their message handlers that we consider make up our services.

This is what an endpoint might look like:

What this diagram doesn’t show is that there is quite a bit of infrastructure code required to glue all this together, perhaps a Bus like API where messages are published and subscribed to and where this ‘Bus’ persists it’s messages to it’s local queue. All messages will be read, sent and removed from queue to discovered endpoints inside a transaction, therefore if any transport of messages from point to point fails the transaction will rollback and return the message to the queue. I have previously defined what the WCF endpoints would potentially look like (at a very rudimentary code level) in a previous post.

To sum up, unless Oslo has something special in store that isn’t making itself plainly obvious then I am quite up beat about this design providing some value as a lite weight framework. Counting down the days but I am not holding my breath, perhaps somewhere deeper in the products release cycle.