Living in the Tech Avalanche Generation

Hot on the heels of Udi’s visit to Melbourne this year (in January), comes another chance to make good if you missed out. Plans are underway for Udi to visit us yet again, this time in Sydney. If your interested check out Udi’s post and register your interest using the link for the proposed Sydney event. If you want to know more about the course you can see the course outline here and read my review which contains links to some further reviews from other attendees in Melbourne earlier this year.

After having recently sat through Udi’s 5 day course on Advanced Distributed  Systems Design (ADSD) and after several days of quiet introspection punctuated with some random reconnecting to the web full of it’s opinion and ideas, I found myself ever more at peace with the challenges that lie ahead. During a conversation with Udi whilst he was here in Melbourne recently, he recalled a twitter comment made by my colleague Mark wherein he quoted my flippant remark that having NServiceBus at my disposal could be likened to being armed with nuclear weapons. Perhaps it wasn’t the most elegant analogy but I think it conveys the sentiment, I feel able to leverage and harness a great deal energy and power. This sentiment can also be attributed to the many great ideas that you find yourself being exposed to in the ADSD course

In these times of rampant software development technology proliferation, it would be easy to feel overwhelmed and I know from speaking with colleagues that many do, yet somehow I feel more at peace than ever. Technology stacks, frameworks, buzzwords, paradigm shifts, they are all hurtling towards us at a pace most of us have never known and the inertia alone is staggering.

Many of us need help in separating the stuff from the fluff. As Neo said to the architect, “the question is choice”! Well, is it? What are the choices? Do technology choices alone get your system built? No of course they don’t but many have fallen into the trap of behaving as though they do. Ever caught yourself saying something like “oh if I just use MVVM or ASP.NET MVC, my system will be perfect because of benefits X, Y and Z”? Other developers occasionally ask me, “what technology should I learn next” or “how do I future proof myself”? I think they are somewhat surprised by my answer; my advice is to stay as far away as possible from attempting to build value as a vendor stack expert and focus attention on maturing and refining your design and communication skills. It seems obvious really, after all once you have well developed your design and architectural skills, then you are far better placed to make appropriate technology choices. The ensuing  implementation experience will sort out any technology knowledge gaps that you feel the job’s market is causing you to feel downward pressure from. This doesn’t mean ignore technology all together, by all means pull it apart and analyze it, see how it fits but not in isolation – you need to give it it’s context. This became quite plain to me after having been through the course two years ago.

The DDD / SOA course was a lynch pin for me it was the key that has opened many doors. Without the learning in principles first, no technology will stop you from making a mess of things. Again it’s pretty obvious when you say it but I am sure I am not alone as someone who came to the course with a whole set of assumptions based on layered designs and technology fads as a bag of best practice tools and tricks.

I have now had the privilege of sitting this course three times and it never get’s old. Since I first experienced it two years ago, the content has changed a little and become even more refined. Each time I sit through the course I feel more empowered than the last time, the lessons are learned more deeply and start to become part of my psyche.

Some of my fellow attendees have posted their thoughts upon having completed the course just recently in Melbourne and I suggest you consider their feedback as well. If you haven’t heard there is now a mailing list dedicated to discussing the courses content and if you are planning on attending in London or the US sometime this year you should definitely check out the mailing list.

I would like to leave you with this; if you have not done this course and you have even the vaguest inkling that you might stand to gain something from learning what what some of the best architects in world know but rarely share in great detail, then do whatever it takes to get yourself a seat when next ADSD rolls into town or find a way of bringing Udi and the course to you.

I’m not about to cut all ties with my relational past and abandon the RDBMS. No, indeed that’s not the appropriate response – well at least not at this stage. So an exploration is what is going to be required to help in clarifying whether MongoDB can find a place in my toolbox.

Watching Michael Dirolf recently sparked my imagination, I began to see a potential abundance of goodness that could be applied in many way – possibly alongside a relational database. The potential for its implementation on the query side in a CQRS architecture for instance, serving as a store for directly bound view model data. MongoDB at first glance offers some nice features that would blend well into an architecture that included serving data from inside the web tier; BSON (binary JSON) may resonate well, it’s unceremonious and lite weight. The future promise of sharding as a product feature also has it’s appeal for obvious reasons.

Figure 1.0 – Potential Architecture

My initial enquiry was grounded in the efficacy of putting MongoDB to work in a distributed publish / subscribe architecture where (N) subscribing instances of autonomous components in a logical service boundary could effectively update N instances of a dedicated view model database. Each view model perhaps sharing an affinity to a web server allowing scaling out both in tandem. In order to sustain eventual consistency, this architecture employs transactional queues. MongoDB achieves it’s scale efficiencies by ignoring some the properties of an RDBMS that are commonly associated with the difficulties in scaling them. What’s clear is that an architecture that heavily leverages publish / subscribe, one way messaging and transactional queues (aimed at providing high availability and scale), will also generally require it’s storage to support transactional behavioural properties. MongoDB advertises it’s replication feature set as part of it’s toolbox, this helps achieve eventual consistency across a shared grid of servers but these two worlds are not about to find a way to play well together. In effect, both seem to be coming to a similar problem with a different solution, with MongoDB addressing scale through it’s built-in auto sharding, focusing more intensely on the database alone. For MongoDB to work in this environment would require the ability to enlist database calls in distributed transactions, something that is not supported by design. It would be possible to add a transactional messaging gateway to solve this problem, however there are other database alternatives that would not require me to take that extra step and additional latency.

This of course in no way invalidates assigning value to looking more deeply at MongoDB (and the like), therefore as I plant sign posts along the road about to be travelled and in light of the friction described above, I will now change track somewhat. My next move forward is to get a better grip of what the developer experience is about (not every problem requires the same solution).

Me as Developer - Starting Out

I started out with a somewhat familiar approach, running the JavaScript based shell you can run ad-hoc queries somewhat akin to query analyzer in the SQL Server world. Once installed and having put the binaries in my path environment variable, running the following commands will start up the server and the shell.

   1: c:\users\simon.segal\mongod

   2: c:\users\simon.segal\mongo

Figure 2.0 – Mongo Shell & DB

Going from download, to install, to running the shell, inserting and querying for data, is really a relatively quick exercise. With just a dash of reading from the excellent documentation (which also has a friendly readable PDF download utility), you can be writing code in the shell very fast. Code like this doesn’t need a lot of explaining to get going.

Listing 1.0 – Mongo Shell

>customer = {firstName: "Jon", lastName: "Doe", DateJoined: new Date("2010/01/30")};
>db.Customers.save(customer);
>db.Customers.find();
{ "_id" : ObjectId("4b5016e65d3c000000006393"), "firstName" : "Jon", "lastName"
: "Doe", "DateJoined" : "Sat Jan 30 2010 00:00:00 GMT+1100 (AUS Eastern Daylight
 Time)" }

The Officially Supported Drivers

• C
• C++
• Java
• Javascript
• Perl
• PHP
• Python
• Ruby

The Unofficial Community Supported Drivers

• C#
• Cold Fusion
• Erlang
• Factor
• Fantom
• F#
• Go
• Groovy
• PHP
• Powershell
• Ruby

The C# Driver

You can see that Microsoft technology is fairly well represented in these lists but I want to focus on the C# driver and talk about using MongoDB from C# specifically. Here’s some code using the C# driver to help start getting a feel for the API.

public void CreateCustomer()
{
    var mongo = new Mongo("127.0.0.1", "27017");
    mongo.Connect();
    var db = mongo.getDB("HR");
    var employees = db.GetCollection("Employees");
    var employee = new Document();
    employee["FirstName"] = "Simon";
    employee["LastName"] = "Segal";
    employee["JoinDate"] = DateTime.Now;
    employees.Insert(employee);
    var cursor = employees.FindAll();
    foreach (var emp in cursor.Documents)
    {
        Console.WriteLine(string.Format(
            "Customer ID is {0}, first name is {1} and " +
            "last name is {2}", emp["_id"],
            emp["FirstName"], emp["LastName"]));
        Console.WriteLine(emp.ToString());
    }
}

After running this code in Visual Studio we get the following output which prints a formatted string and the JSON representation as is.

Figure 1.0

Issues

The first problem with writing code like this is that it’s about as far away from being Domain Driven as I can imagine, there is no concept of an Entity, just a Document. Well it’s a Document Database you say – so why don’t I just stop whining and get on with writing code like that above? Sure transaction script might be great for some scenario’s but when I want to work with a richer domain model I don’t think I should need to change to accommodate the drivers API. Let me be clear, I don’t think there is anything wrong with the API, it’s perfectly fine and in fact I think it’s more than pliable enough for me to use it in other ways that will suit me from time to time.

I alluded earlier in the discussion, to the fact that I originally wanted to proof of concept MongoDB for playing a role as an eventually consistent storage for the Query side of an implementation of CQRS, where views are being bound directly to view model data (as described here) via a RESTFul facade. In part 2.0 I am going to look at how we can build a Context object (think DataContext or ObjectContext) to wrap up the C# driver’s functionality and enable working directly with POCO’s, removing any direct requirement to work from the MongoDB API’s Document class. That’s it for now, let’s see where we end up in Part 2.0.