Sunday, March 29, 2009

Web 2.0 Expo travel

We have just been incredibly fortunate this far. I last flew commercial soon after 9/11, and the experience was harsh to say the least. Anotinio Gramsci once wrote that the downfall of a secular democracy was the continuous surrendering of freedoms in exchange for protection from our fellow citizens. Our travel experience to SF this weekend has taught me how much the human spirit can shine despite and perhaps paradoxically because of difficult circumstance. We have been graced by the goodwill of so many of our fellow citizens that I am tacitly aware of Gramsci's error if unable to put words to it.

Today has been incredible, but I must save my emotive prose for another time.




Wednesday, March 18, 2009

Sex-Software Simile

Using software vendors' technology demos as the metric for your software development is like measuring your sexual prowess by watching porn films: accurate for edge cases but ultimately not relevant.

Friday, March 13, 2009

Why ORM? Why Cloud Computing?

I think it was Fred Brooks that once wrote that someone will inevitably look at any software system and ask, "how did we get here?" And, the answer is invariably, "one logical choice at a time."  Rather than answer the questions, "Why ORM?" and "Why Cloud Computing?", I will endeavor to elucidate those logical choices.

The monolithic database model (MDM) has served the IT sector very well for about fifty years.  I will explicitly define it as a model of software development where myriad applications are integrated through a shared database, or--more accurately--more than one user interface is provided to a database.  In this model the database itself is where business processes and domain modeling exist.  Enterprises often expanded these databases to include all aspects of the business, or perhaps a few databases integrated via ETL, until the number of objects exceeded any one person's capacity to understand the system.  EDI, data warehousing, and data marts are all extensions of the (MDM) for dealing with M&A, e-biz partners, etc.  Given a large number of well-trained database professionals with rigorous communication and documentation requirements, the monolithic database model is efficacious and expedient.

As object-oriented programming techniques became more well-known, developers began modeling the domain and business process using advanced general purpose programming languages like Lisp, Smalltalk, C++, etc.  They used object databases to persist these models and all was good and productive.  Unfortunately, few developers grok those technologies, and even fewer have a mastery of the techniques needed to use them well, so it didn't catch on.  Perhaps the biggest impediment to adoption was the success of the MDM. The MDM was working and understood.  The deficiencies of the model were insufficient to move IT development en masse to the object crowd.

Still, a lot of software was being written outside of IT.  Embedded developers loved object programming and had no need to store loads of data.  So, the hardcore programmers--those that relish writing life support systems, flight control systems, nuclear power plant control systems, etc.--were exclusively "object guys".  Meanwhile, Sun Microsystems was building a runtime for embedded systems to simplify programming across hardware--Java.  A 4GL, Java had all the object-oriented bells and whistles, and Sun had written runtimes for Unix, Solaris, and even Windows!  These runtimes came bundled with AWT, allowing developers to nominally "write once, run anywhere."

Suddenly there was the WWW.  I say suddenly because even Bill Gates once purportedly remarked that it was irrelevant.  As IT understood the implications of Metcalfe's Law for the Internet, it became the elephant in every boardroom.  No single company in the world wanted to be a loser in the new, connected economy.  All IT decisions were tempered by the Internet revolution.

Now the cohesive, insular world of the MDM was really looking problematic.  Still, that's where the data was, and ultimately the Internet was about access to information.  So, every IT vendor that wanted to compete was pitching platforms that claimed to do one central task better than anyone else: turn relational data into web pages.  These platforms were either backed by scripting or 4GL, and DBI, JDBC, ODBC, ADO, etc., in all their versions and iterations were simply libraries used by the object guys to get at the data.

Concurrently, universities had been turning out software developers who started writing C++ or Java in their freshman year, graduating to "real" object languages in subsequent years.  The worldwide demand for IT developers skyrocketed, and technical colleges met the challenge with programs designed to get people into the market with the minimal set of skills to create web applications.  At this point the technologies for building web applications were infantile relative to RDBMS technologies.  The result was a lot of really bad developers developing really bad applications, slowly.

The good developers, the object guys with strong Lisp kung fu, really resented having to interface with databases--to go from their object model to a data model.  IT programmers oft complained of the inordinate amount of time they spent move data in and out of databases.  Architects started designing systems built around messaging again, calling them web services this time, and the canon said that these messages were serialization of object state.  So smart people started writing libraries for interfacing with databases in a tacit fashion, attempting to abstract away the "implementation details" of data storage and retrieval: object-relational mappers (ORM).  The "real" model of business entities and processes, after all, were their objects.

Now, the old IT systems in DB2 and Cobol were "legacy" systems to be tolerated until they could eventually be re-written.  New IT initiatives would set out in their own silo (usually an application server), with their own persistence engines that utilized ORM, and integrate with the legacy systems via message-oriented middleware (MOM): Biztalk, SOA, ESB, etc.  Vendors were happy because MOM was expensive; object-guys were happy because their jobs now matched up with their education; and business owners were hopefully optimistic because mainframe vendors all operate on an annuity model, and the owners' hope was to escape from a world of multi-million dollar support contracts.

Lo, the MDM guys sneered, since the whole enterprise--the business itself--was still built upon their systems, and every attempt to re-write their systems was an incredibly expensive failure.

Business owners were getting pretty angry, too.  All these application servers needed lots of new hardware, and the new servers needed to be partitioned by firewalls, and they needed new database servers too, since RDBMS was still the persistence engine of choice, and of course all of this needed fail-over and clustering.  Meanwhile, what the developers were producing wasn't sufficient to run the entire business on, so they now had to maintain two systems in parallel, putting increased strain on the middleware servers.  Further, the developer's velocity was slow, since everything had to be integrated, and years and years of process refinements implicit in the MDM had to be re-imagined and re-discovered.  This integration stuff was expensive.

Again, though, not all software was being written in IT shops.  There were companies building applications hosted on web servers that were available on-demand, so called Application Service Providers (ASP).  These companies didn't have any legacy systems and drew the smartest object guys.  Many of them were wildly successful, but when the dot-com bubble burst, the idea that a company that you relied upon for mission critical systems could just disappear caused many IT shops to reject the idea out-of-hand.  The ASPs that survived were those companies that offered important but not "mission-critical" (like email, production data management, etc.) services, such as sales force automation or marketing support.  Another class of ASPs that won were those that offered hosted services to SME customers who could not or chose not to afford the sunk costs of providing their own infrastructure.

In the U.S., capital was king; the financial markets were providing great returns and due to foreign investment and interest rate cuts the cost of capital was low.  Businesses were now faced with enormous capital expenditures from their IT operations during a time when capital investments were the most prudent choice, but no one was willing to give up the very real productivity gains that had been made steadily in the last decade.  The solution to the dilemma was to defer expenditures, to lease.

At the same time, the business owner's frustration with the lack of agility of their IT operations peaked.  The costs were high, the pace of development was slow, the requirements gathering and definition process was pain-staking, and the failure rate was abominable.  The maintenance costs alone for all the manifold servers, languages, and platforms was choking the life out of new business opportunities before they could be explored.

So, the market itself demanded cloud-computing.  Rather than buying the infrastructure and paying to provision it, they wanted to "pay as you go."  Elastic demand on IT systems meant provisioning for the worst-possible scenario; cloud-computing meant scaling-up (and down) on demand.  Rather than having to pay for and maintain application servers, database servers, load-balancers, etc., they could just write the app and see if it sticks.

Yea, though I walk through the valley of the shadow of integration, I shall fear no SQL.

The problem of integration remains.  No cloud platform fits any enterprise's IT needs completely, so now the challenge is how to integrate a world where the web application server platform is considered a legacy.  Microsoft's solution is to position all ASP.NET development as "cloud" development, making it easy to keep the application on-premise or move it to the cloud (Azure).  Ironically, the major impediment to this approach is the persistence engine: the database. 

Moving a database that fully exploits the RDBMS platform to the cloud is non-trivial.  In other words, developers that wrote stored procedures, utilized event brokers or extended stored procedures, created ETL jobs, made it unlikely that they can move their applications to Azure.  If they instead had only used pure ORM (LINQ2SQL), this could be done mechanically.

This property of not violating the ORM abstraction, namely the ease with which the backing data store can be mechanically replaced, is the key reason why ORM advocates promulgate the principle that the application should never accept a dependency on the data store.  In other words, no stored procedures allowed.  The Entity Data Model is an attempt to abstract the universality of the data model--arguably the best feature of MDM--away from the persistent storage engine(s), in order to ensure the portability of the data model in the enterprise and beyond, i.e. to the cloud.

Wednesday, March 4, 2009

How We Will Develop for the Web in Five Years

Okay, I admit that I'm predisposed to prognostication, but I'm less inclined to blog if I'm not confident.  So comeback in five years so I can say, "Told ya so!"

In five years, nearly everyone who writes code for the web will be using a web-based IDE to create those applications.  All of the libraries that you use to create your UI will be hosted on a CDN, freely available to all, and your IDE will be aware of the CDN.  You'll add new widgets to your toolbox by specifying CDN URLs.

The MVCC for your web project will be hosted by your cloud infrastructure provider.  Unit tests and code coverage requirements will be imposed on check-in, and continuous integration will be de rigueur.  The other distributed developers on your team will receive a notification of failed builds in the web-based IDE, and workgroup chat capabilities in the IDE will be used to coordinate a fix.

On the server-side, the distinction between persistent storage and in-flight objects will have all but disappeared.  No longer will there be a separate notion of a data store and middle-tier objects.  Ad-hoc object composition will be supported via interfaces in the object runtime, as opposed to a query language.  Backups and disaster recovery will be outmoded ideas, as the cloud infrastructure provider will ensure redundancy and edge computing performance in myriad datacenters.  Data replication between these points-of-presence will be completely automatic and unnoticeable.

The standard application architecture will be server-coordinated real-time continuations.  The majority of applications will have just a handful of screens in the UI, with one screen generally accounting for 80% of interactions.  These screens will be HTML pages generated primarily using JavaScript UI toolkits.  That is, the server will not generate individual pages, but will deliver client-side applications.  These applications/screens will interact with the server via RESTful AJAX calls returning JSON that will move forward a state-machine facilitated by continuations on the server.  Browser extensions like AIR/Flex, Gears, and Silverlight will be used primarily for their ability to allow offline application interactions, not for flashy graphics or rich-media, but this usage will eventually be deprecated by browser implementations of offline storage in HTML 5.

State machines and workflows will be specified using a constraint language layered on top of the fluent interface/DSL available ipso facto on a platform that requires an object model to get any work done.  So, business rules will be clearer and organized, allowing more transparency in the development process.

This state of affairs will require domain-driven design a priori.  While user interfaces and the mechanics of interactions could be built ahead of other development, all meaningful interactions will require the construction of workflows out of domain models.

Much of these ideas already exist.  Seaside is a great example of browser-based development and server-side continuations.  Salesforce.com's Force.com platform imposes code coverage restrictions for production code. LINQ provides an object-runtime method of programming without an explicit query language.  Google hosts jQuery and other tools on its CDN.  ASP.NET 4.0 stands poised to take client-side templating to the masses. AWS, Azure, AppEngine, and RailYard are providing the infrastructure we need.  The pieces exist, what I wanted to do was articulate a vision for the future where these technologies are mainstream, integrated, and proven.