There is a revolution happening quietly in the development of web computing infrastructure. To those who have been involved in the development of large scale distributed computing, i.e., cluster and grid computing, the concepts and applications of the revolution are decades old. To the computation science community, including weather forecasters, climate change scientists, numerical ecologists, artificial intelligence experts, bomb developers, etc., these types of efforts have been at the forefront of super computing technology development. I have been involved in various types of cluster computers for oceanic ecological modeling, and some of our collaborators at Rutgers University are experts in the field of distributed computing for coupled atmospheric and oceanic modeling. However, for the average person, terms like cluster or grid computing have little or no tactile meaning. Perhaps a few could tie it to the SETI grid computing project, but even these few might not understand the implications at the average business or consumer level.

One of the favorite terms today for large scale distributed computing at the business and consumer level is “Web-Scale Computing”. You see it in the sessions for a couple of the O’Reilly Conferences (ETech and Web 2.0 Expo) mainly discussing Amazon Web Services’ (AWS) EC2 and S3 services. AWS is one of the first mainstream applications that put the power of cluster computing into the hands of commercial web application developers. With these services, and those that will surely follow, we as a society/culture/business community move one step closer to the concept of on-demand purchasing of computer cycles, and the development of markets in these cycles.

These services, what I will refer to as Commodity Computer Cycles (C3), are different from commodity computing. In commodity computing you are still responsible for assembling the components and the network of processors into a cluster for your distributed processing application. You are still required to pay for the power, cooling and maintenance, as well as, the personnel involved in development, care, and security of the systems. These expenses are upfront and continuing throughout the life of the business, regardless of total computational use. With C3, you buy the FLOPS or the storage space needed for your application, on-demand.

With C3, your business can then focus on developing better applications and services for your customers, rather than the development of the in-house infrastructure to rack, cool, and take care of your computers. If the outsourcing of FLOPS and storage makes business sense (which I truly believe it does), we should expect that the demand of C3 services will increase, leading to the building of more C3 infrastructure and therefore feeding virtuously into the creations of evermore efficient web applications and services. If the revolution seriously takes roots and spreads across the whole of the business and consumer communities, it will affect us all.

To bring this discussion back home, we at WeoGeo are trying to change the dynamics of quantitative mapping. Our own maps are terabytes in size, and require petaFLOPS of processing. We developed our web exchange and server application on EC2 and S3 for many reasons, including the costs associated with the growth in our computing needs and the requirement to automatically scale as a function of computing cycle demand. In addition, by developing on a fully scalable C3 model (see below), we could pass the infrastructure savings directly to our user community. This should help enable them to develop new markets for their mapping products and hopefully lead them into a new model for generating revenue in the field of geospatial maps, services, and technologies.

The EC2 version of C3 marks the beginning of the widespread commercial use of on-demand distributed computing. I believe it is a harbinger of things to come. For our purposes, EC2 was not quite ready for prime time and we had to overlay additional intelligent management software to provide stability and optimized scaling to take full advantage of the C3 potential offered by AWS (see this WeoCEO blog post, as well as this AWS forum post by Robert Banfield). I am sure that our solution is but one of the first of many to come. The important thing to recognize is that the delivery of scalable, fully optimized, Commodity Computing Cycles is happening right now and will only get better, easier, and cheaper with time. I believe that the next phase of productivity enhancement in the business and consumer markets begins now, and it is truly exciting to be a part of this wave.