Living in the Tech Avalanche Generation

Perhaps you’re the Development Manager or Architect with an ISV, where the commercial landscape plays into your technology choices constantly. Maybe your product roster is tied to a SQL vendor or possibly more than one. Meanwhile you keep hearing a nagging noise about NoSQL, its been circling the wagons for a while now and a few of the top flight developers in your team just wont stop talking about it. Until now it hasn’t been a fit, all your products are quite mature and well established, your customers are conveniently placated by your choice of SQL Server or Oracle; they too are invested in a particular database product, they have licenses, institutional skill set and divisional product and platform proliferation choices have been made on the theme of unification. Oh, and as far as your product is concerned next generation planning doesn’t seem close at hand.

Change starts within!

Eventually the CEO comes up with a new product idea that has him so excited that your team is whipped up to fever pitch on the promise of starting a new application from scratch, the time has come to start popping the greenfield champagne corks in the cubicles on the software development floor. In the earliest of design meetings the noise about NoSQL has picked up ferociously, the senior and junior developers are all beating their chests and pushing hard for the company to agree to adopt this new paradigm for data persistence but you have anticipated your bosses reaction, knowing full well he will have some serious commercial concerns about choosing a NoSQL platform to underpin your data storage; what about the clients and all their institutional dependency on SQL, what about all their quirky integration hacking between your system and someone else’s, imagine the new plethora of support requirements for clients that have barely heard of NoSQL let alone have a administrators with the capacity to manage it…the list goes on.

As a hands on kind of guy you also see the benefits of the NoSQL choice and if you have to write as much data access code as you have routinely over the years (regardless of your fancy dancy ORM) you might just loose interest in this new shiny project. At the end of the day the boss beats you down and the same old reliable safe choices get made for data access and storage, you are left wondering what has to change so this doesn’t become the default position until such time as you retire!

When you think about how long SQL has been ensconced on its throne, the ubiquity of the SQL language and its potential for universal tooling, then question begs, how long would it take to make NoSQL a transferable skill between jobs, enough to permeates IT and seep out of those cracks and into operational non IT staff skill set?

UnQL resume fodder?

The recent news on UnQL has tongues wagging about the potential for a universal language for document databases, should the vendors decide to implement it of course. Nobody in their right mind is expecting this to happen terribly quickly and even if it does eventuate and provide us with the basis of some equally powerful universal tooling that can be found in the SQL world, there is a deep drop between there and crossing the chasm of bureaucratic red tape and all the elements of resistance to change. I wonder how long it is till we see recruiters list UnQL in job ads?

Independent Software Vendors

Storage concerns that bleed into architecture choices as a direct result of commercial concerns can do so in ways that and in-house software team or web 2.0 vendor may not typically have to consider. There is a degree of user data hacking that ISV’s need to consider upon servicing a wide range of clients who possess peculiar concerns and workflows. This can lead to any given client choosing to customise the software vendors product database. Some ISV’s encourage this behaviour and others don’t, opting for building points of extensibility. Making points of extension available is useful approach but it doesn’t eliminate client resistance when the only integration avenue open to them is constrained by their institutional dependency of SQL.

As a software developer / vendor I find myself often wanting to take advantage of a schema free database but in some instances it represents a potentially commercially damaging choice, one that simply cant be made. The exception might be ancillary tooling in a product suite, applications with small surface area as far as clients are concerned and any projects that fall outside of the scope of client SQL integration needs.

NOTE: I am fully aware that we haven’t yet discussed specialised reporting or BI tooling here in this post and that its more than possible to design an architecture where SQL products are used in conjunction with a Document Database.

I’m super excited, 10Gen are currently taking a survey for registrations of interest for MongoDB training run by them in Australia. So, I would implore anyone with a eye on the NOSQL or MongoDB space to get over quickly to survey monkey to fill in the Survey and show your Mongo love!

It’s been palpable the level of interest and noise being made by the .NET community around MongoDB, however MongoDB has no special language affiliation and the Survey seeks to know your language preference  (with regards to the training), so if your simply interested in imbibing on the best MongoDB training around, then fill in the Survey and spread the word.

Here is the link to register your interest.

I’m not about to cut all ties with my relational past and abandon the RDBMS. No, indeed that’s not the appropriate response – well at least not at this stage. So an exploration is what is going to be required to help in clarifying whether MongoDB can find a place in my toolbox.

Watching Michael Dirolf recently sparked my imagination, I began to see a potential abundance of goodness that could be applied in many way – possibly alongside a relational database. The potential for its implementation on the query side in a CQRS architecture for instance, serving as a store for directly bound view model data. MongoDB at first glance offers some nice features that would blend well into an architecture that included serving data from inside the web tier; BSON (binary JSON) may resonate well, it’s unceremonious and lite weight. The future promise of sharding as a product feature also has it’s appeal for obvious reasons.

Figure 1.0 – Potential Architecture

My initial enquiry was grounded in the efficacy of putting MongoDB to work in a distributed publish / subscribe architecture where (N) subscribing instances of autonomous components in a logical service boundary could effectively update N instances of a dedicated view model database. Each view model perhaps sharing an affinity to a web server allowing scaling out both in tandem. In order to sustain eventual consistency, this architecture employs transactional queues. MongoDB achieves it’s scale efficiencies by ignoring some the properties of an RDBMS that are commonly associated with the difficulties in scaling them. What’s clear is that an architecture that heavily leverages publish / subscribe, one way messaging and transactional queues (aimed at providing high availability and scale), will also generally require it’s storage to support transactional behavioural properties. MongoDB advertises it’s replication feature set as part of it’s toolbox, this helps achieve eventual consistency across a shared grid of servers but these two worlds are not about to find a way to play well together. In effect, both seem to be coming to a similar problem with a different solution, with MongoDB addressing scale through it’s built-in auto sharding, focusing more intensely on the database alone. For MongoDB to work in this environment would require the ability to enlist database calls in distributed transactions, something that is not supported by design. It would be possible to add a transactional messaging gateway to solve this problem, however there are other database alternatives that would not require me to take that extra step and additional latency.

This of course in no way invalidates assigning value to looking more deeply at MongoDB (and the like), therefore as I plant sign posts along the road about to be travelled and in light of the friction described above, I will now change track somewhat. My next move forward is to get a better grip of what the developer experience is about (not every problem requires the same solution).

Me as Developer - Starting Out

I started out with a somewhat familiar approach, running the JavaScript based shell you can run ad-hoc queries somewhat akin to query analyzer in the SQL Server world. Once installed and having put the binaries in my path environment variable, running the following commands will start up the server and the shell.

   1: c:\users\simon.segal\mongod

   2: c:\users\simon.segal\mongo

Figure 2.0 – Mongo Shell & DB

Going from download, to install, to running the shell, inserting and querying for data, is really a relatively quick exercise. With just a dash of reading from the excellent documentation (which also has a friendly readable PDF download utility), you can be writing code in the shell very fast. Code like this doesn’t need a lot of explaining to get going.

Listing 1.0 – Mongo Shell

>customer = {firstName: “Jon”, lastName: “Doe”, DateJoined: new Date(“2010/01/30″)};
>db.Customers.save(customer);
>db.Customers.find();
{ “_id” : ObjectId(“4b5016e65d3c000000006393″), “firstName” : “Jon”, “lastName”
: “Doe”, “DateJoined” : “Sat Jan 30 2010 00:00:00 GMT+1100 (AUS Eastern Daylight
 Time)” }

The Officially Supported Drivers

• C
• C++
• Java
• Javascript
• Perl
• PHP
• Python
• Ruby

The Unofficial Community Supported Drivers

• C#
• Cold Fusion
• Erlang
• Factor
• Fantom
• F#
• Go
• Groovy
• PHP
• Powershell
• Ruby

The C# Driver

You can see that Microsoft technology is fairly well represented in these lists but I want to focus on the C# driver and talk about using MongoDB from C# specifically. Here’s some code using the C# driver to help start getting a feel for the API.

public void CreateCustomer()
{
    var mongo = new Mongo(“127.0.0.1″, “27017″);
    mongo.Connect();
    var db = mongo.getDB(“HR”);
    var employees = db.GetCollection(“Employees”);
    var employee = new Document();
    employee["FirstName"] = “Simon”;
    employee["LastName"] = “Segal”;
    employee["JoinDate"] = DateTime.Now;
    employees.Insert(employee);
    var cursor = employees.FindAll();
    foreach (var emp in cursor.Documents)
    {
        Console.WriteLine(string.Format(
            “Customer ID is {0}, first name is {1} and “ +
            “last name is {2}”, emp["_id"],
            emp["FirstName"], emp["LastName"]));
        Console.WriteLine(emp.ToString());
    }
}

After running this code in Visual Studio we get the following output which prints a formatted string and the JSON representation as is.

Figure 1.0

Issues

The first problem with writing code like this is that it’s about as far away from being Domain Driven as I can imagine, there is no concept of an Entity, just a Document. Well it’s a Document Database you say – so why don’t I just stop whining and get on with writing code like that above? Sure transaction script might be great for some scenario’s but when I want to work with a richer domain model I don’t think I should need to change to accommodate the drivers API. Let me be clear, I don’t think there is anything wrong with the API, it’s perfectly fine and in fact I think it’s more than pliable enough for me to use it in other ways that will suit me from time to time.

I alluded earlier in the discussion, to the fact that I originally wanted to proof of concept MongoDB for playing a role as an eventually consistent storage for the Query side of an implementation of CQRS, where views are being bound directly to view model data (as described here) via a RESTFul facade. In part 2.0 I am going to look at how we can build a Context object (think DataContext or ObjectContext) to wrap up the C# driver’s functionality and enable working directly with POCO’s, removing any direct requirement to work from the MongoDB API’s Document class. That’s it for now, let’s see where we end up in Part 2.0.