How do you make a geospatial exchange a reality? You find great content providers to bring their wares to the market. Aerials Express (AEX) is one of those great content providers. With 420,000 square miles of high resolution aerial imagery over major metropolitan areas in the US (see map below), AEX brings base map content to “prime-the-pump” in the derivative product marketplace.

Christopher Warren and Bill Landis at AEX have been great. Their listings of AEX products address a big niche in our industry. High resolution imagery that can be physically acquired and manipulated with an explicit license to resell derivative works. Bill’s quote from the Press Release -

WeoGeo is an excellent opportunity for our company, said Bill Landis, President of Aerials Express. We are looking to WeoGeo’s advanced technology and unique distribution model to enhance the availability of our products into a wider range of GIS related markets.

It says a lot about the potential of an exchange-based market for our industry.

We will do our absolute best to make the market technology easy to use for search, discovery, and product acquisition. Its success will increase productivity and margins for all of its participants. Today, we mark its beginning.

There is a nice discussion happening on James Fee’s Blog about Web Mapping Systems and Services and the future of hosted mapping services. I was reading it and thought back to an interesting Wall Street Journal article on Monday about Circuit City that said same store sales in December fell by 12% in the US. While this news was depressing for the stock market, the silver lining for the geo-community was that navigational products were the only product line with increasing sales over the period.

Geo-devices are becoming more ubiquitous. The shear number of curious and talented people moving into our industry combined with these devices will drive product and service innovation in directions that may not be completely clear at the moment.

Converging with the mass market penetration of geo-devices and geo-content (geoware?) is the cloud computing efforts by AWS (and soon to be others). While the production of quality mapping today may require high end desktop workstations and servers, I think that Moore’s Law is eventually going to allow our field to produce geo-content and services far more easily, leading to a feedback into future product innovation. How we in the professional community create products and services today may be radically different in the future.

I offer this anecdote - today, after 10 years of running a Microsoft Exchange Server for our email requirements, we switched to Google Mail Premium. Over the 10 year period, we incurred costs of $10,000s, possibly greater than $100,000. These costs included licensing, hardware, server room, service personnel, etc. Our spam filter alone on the MSFT Exchange Server costs us $35 per year per mailbox. Our costs for Google Mail Premium service is $50 a mailbox per year. It is an easier to use, cheaper to implement, and offers more robust service than the Exchange product.

I think there might be parallels for our industry in this anecdote. It is probably a good exercise to be thinking about what products might be replacing the ones we are using today.

The future of GIS, geo-content, geo-entertainment, etc. will belong to those who can think outside of the traditional methods of production and product delivery. For historical evidence of the difference between companies that focus on the future and those that focus on their current narrow niche, look at the change in market capitalization of Trimble (TRMB) and Garmin (GRMN) over the last decade.

Above Chart taken from Google Finance

I am writing this in Seattle as we prepare for the finals of the Amazon Start-Up Challenge. We are truly excited to be a part of this Challenge. It is an amazing opportunity to be recognized for our technology and business model.

I am in love with our technology and how easy it makes it for all players, large and small, to participate in a global mapping market based on the quality of their skills and geo-content. WeoGeo will make it easy for all of us in the geospatial field to do our jobs, while at the same time increase our operating margins and productivity.

A really exciting part of this Challenge is that our business model is also being recognized by an internet service company with a $39 billion market capitalization. Everyone in our field has seen the explosion of interest in geospatial over the last few years. One only has to look at the >200 million downloads of Google Earth, as well as the consolidation by Tele Atlas and NavTeq to know that our industry is moving mainstream. This is excellent new for all of us, for it will provide more resources and revenues for our field, which translates into better opportunities for us all.

Come see the videos of the finalists in this Challenge and vote for your favorite (preferably WeoGeo!). But make sure you see our video. I hope the passion for what we are trying to accomplish is evident. I also hope that you will find what we are attempting to accomplish as exciting as we do.

Big thanks to Adena Schutzberg at All Points Blog and James Fee at Spatially Adjusted for helping us rock the vote!

The only thing I can say is, “Wow!” Followed by the biggest grin you have ever seen on my face. As one of 7 finalists, Amazon expresses their confidence in our technology and business strategy. In all honesty, I am humbled and honored by the selection, and truly thank them for their selection of us as one of the 7 finalists.

I believe (passionately) in what we are trying to create. I believe that WeoGeo will change the paradigm in how we discover and access geo-content. I believe that we (the geospatial industry) as a community will more easily synthesize new mapping products that will help us create a better world. But these are my beliefs, and I tend to view everything we do through these rose colored glasses.

The selection as a finalist by Amazon Web Services (AWS) means that someone else out there sees the same potential for the mapping and geo-content industry as we do. It provides validation for the people who have worked so hard on this project beyond anything that I could offer, and for this I am eternally grateful.

In addition, Amazon will offer the winner of this contest a venture investment. I believe this says a lot about the geospatial industry, as well as WeoGeo. For WeoGeo to be among those considered a suitable investment opportunity by a $32 billion dollar company, we must have (1) a great business plan, (2) a great set of technology, and (3) be in an industry with high growth potential. Our industry, the geospatial industry, is now recognized by a leader in internet services industry as having high growth potential.

I’ve been grinning so much, my face hurts…

WeoGeo has created a scalable, fault-tolerant infrastructure to manage its use of Amazon Web Services Elastic Compute Cloud (EC2) operations. I’ve written about it a couple of times (see this link for a listing of the Amazon tagged blogs). The latest version of WeoCEO (Version 0.1.0) is ready for release and with it we are moving from private to public Beta.

This version includes the Assistant to back up WeoCEO (see this feature described in this Amazon Web Services StartUp Event Slide Show). WeoCEO Version 0.1.0 also provides enhancements to the stable IP addressing, failure detection, and automatic scaling and load balancing. These enhancements include automatic emailing to your site administrator during trouble events and detailed logging capabilities.

WeoCEO Version 0.1.0 (including the load-balancing and auto-scaling capabilities) will be free of charge at least until December 1, 2007. It will continue to be free if you only use the stable IP addressing and auto-recovery features for a single client instance.

There will be a charge for the load-balancing and auto-scaling features of WeoCEO, which support running multiple EC2 instances and optimizing your network. The charge for these features will be $0.05 per managed client instance per hour. The charge will be on the average usage over an hour, calculated at <15 minute intervals.

You can obtain a WeoCEO ISO with the setup and installation instructions, by visiting http://www.WeoCEO.com and clicking the “Signup” button, or by clicking the Signup button below. We are still in beta, so constructive comments on any of the components that make up this service will be met with exuberance and free goodies.

I have had a number of questions from our private beta Providers that basically ask, “What maps should I be making?” To be honest, I wish I knew. In reality, WeoGeo Market was established to answer this very question.

We set up WeoGeo to lower the risks of creating and selling mapping products (most importantly by reducing marketing and transaction costs). We believe that by lowering the risk of creating and selling geo-content, more products could be created at lower prices. By combining more products at lower prices with a greater ability to find and customize those products, Users of those products would be more apt to purchase more geo-content. The overall goal is to create a truly functioning marketplace for geo-content. The end result would be a collective intelligence expressed through the market that would help all of us focus on making the most valuable geospatial products.

Answering the question of what product to create is one of the hardest parts of running FERI’s operations. While FERI is a research and development organization, we still had to perform the basic sales and marketing efforts of finding paying customers to support our development of value-added mapping products. It is a very time consuming and difficult process, requiring a lot of telephone calls and a lot of travel to search out programs that would value our imaging and mapping efforts. Such is the nature of sales and marketing, and every good salesman would tell you that is just the way you have to generate business.

However, as a scientist I want to focus on the generation of new mapping products. While I could (and still do) focus on sales and marketing, my real interest is in generating new mapping products that could help people make decisions with their resources or help save lives. With the hyperspectral imagery, we could develop maps focused on a variety of topics. These maps could range from Harmful Algal Blooms (HABs or more commonly called red tides) to Submerged Aquatic Vegetation (SAV) to detecting probable locations of Improvised Explosive Devices (IEDs). Yet, finding sufficient demand for these products to overcome the high initial production cost of creating these products is difficult. (I have a whole other story on the IEDs and how the DoD does business with contractors and appropriation earmarks that I’ll save for another time.)

Over the years, we have watched with great interest how the internet has impacted other businesses. One of the most interesting impacts that we have seen is the rise of shared intelligence from the accumulation of individual choices. For example, search engines have used the individual linkages of web page creators to develop a collective intelligence estimate of the most likely desired result for a search term (and a new industry of search engine optimization). In particular, we were fascinated by eBay’s ability to enable millions of people to develop larger markets for their niche products.

By establishing a functioning marketplace for these niche goods, eBay created liquidity and demand for products that previously had limited marketability. In the process of creating a market whose niches could be efficiently filled, they also provided opportunities for entrepreneurs to develop new markets. In effect, eBay created a platform that enabled individuals to make choices, create products, and satisfy the needs of others, which in turn created a positive feedback mechanism for everyone who participated. This led to the creation of whole businesses that did not exist prior to eBay, and the rise of the valuation of goods that previously had limited market enetration, and thus, underdetermined recognized value.

The increased liquidity of products and the collective actions of many individuals led to a self-sustaining marketplace that enriches all of the participants. eBay is a lesson in economic theory, and gives truth to the concept that “a rising tide lifts all boats.”

So what does this have to do with answering the question from our Providers about which maps to produce? The answer to that question is that I am not sure, but I can make sure that the Provider’s risk is low enough for them to make some reasonable choices, and to give them the agility to respond to market demands. Through this process, I believe that our collective intelligence will point Providers in the most profitable direction.

This marketplace will give those with the skills and those with the content the ability to connect as never before possible. The new network of connections will lead to the creation of new geo-content that will enhance and enrich the lives of our community. And our community will profit from it because we will know which maps to make.

With the Open Geospatial Consortium (OGC) planning adoption of KML as a standard, we’ve focused our efforts in its support on varied platforms. KML’s previous proprietary nature was a cause for concern as Raj Singh aptly describes:

And one crucial point that I think a lot of people miss is the legal intellectual property aspect. Bringing KML into OGC isn’t just about what features end up in that XML format. It’s just as much about making sure the format is royalty-free to use forever.

We use KML as an additional means to help our Users (map consumers) and Providers (map sellers) use the Market. A reduced resolution KML is created for free whenever a dataset is listed by the Provider.

For Providers, a reduced resolution KML button on Panel 4 offers a link to their listing’s KML file. An example of this feature for FERI’s hyperspectral map listing can been seen in the red circle on Figure 1. The link in the red circle provides access to a KML file, which can be used by the Provider as another method to market their map listing through other mapping programs. These links (the one to the listing or the one to the listing’s KML) or the reduced KML itself can be used in any of the provider’s marketing material or blog posts.


Figure 1. WeoGeo Market listing for FERI’s St. Joseph Bay hyperspectral mapping product (click on the image to go the Market listing). The red circle in the bottom left hand corner links to this data product’s reduced resolution KML. The blue circle on the bottom right of Panel 5 is for high resolutions KML products, which are fully tiled and regionized products that the Provider may offer for sale (to be covered in a future post).

For the User, this feature helps them decide whether this map is appropriate for their needs. By viewing the KML in their favorite map viewer, the User can quickly see if this listing intersects with their area of interest or other mapping products. As an example of this feature, here is the KML from the listing in Figure 1 seen in ArcGIS Explorer, World Wind, Google Maps, and Google Earth.

Figure 2. KML listing of FERI’s St. Joseph Bay hyperspectral mapping product offering on WeoGeo Market as seen in ArcGIS Explore v 9.2, build 410. Currently, the KML needs to be saved as a file to your local machine from the link and imported into Explorer. In the balloon is a link directly back to the specific WeoGeo Market listing.

Figure 3. KML listing of FERI’s St. Joseph Bay hyperspectral mapping product offering on WeoGeo Market as seen in Google Maps. In the balloon is a link directly back to the specific WeoGeo Market listing.


Figure 4. KML listing of FERI’s St. Joseph Bay hyperspectral mapping product offering on WeoGeo Market as seen in Google Earth. In the balloon is a link directly back to the specific WeoGeo Market listing.

Figure 5. KML listing of FERI’s St. Joseph Bay hyperspectral mapping product offering on WeoGeo Market as seen in NASA World Wind v 1.4. The KML needs to be saved as a file to your local machine from the link and imported into World Wind.

Adena Schutzberg did a podcast with me last week on the business model for WeoGeo. It was my first podcast and I hope that I made sense to people (I welcome comments and/or critiques in the comments section here). I would like to thank Adena for giving us the opportunity to tell our story.

However, I am not sure I was as clear as I could have been about our history and the importance that history in the development of WeoGeo. I could not quite put my finger on what was missing until after the AWS StartUp Event - Boston (see here as well for my comments) when someone asked how many man-years of effort went into developing the site.

My first response was to take the number of years that FERI was in operation times the number of people involved at FERI. Kind of silly, I know. But when I think of why we built WeoGeo, this response seems relevant. Their response, of course, was, “no really, how much technical development time?” I understood the question; the person was trying to ascertain how difficult it would be to recreate what we are doing.

Our technical development on this project did start back around 2001 with a project called Hyperspectral Data Repository On-line (HyDRO). This was our first distribution system, developed to help alleviate the problems associated in delivering HSI data to our customers. This concept and technology eventually evolved into the WeoGeo Server (see post here as well). Between 2001 and 2005 we had 4 PhDs and masters-trained personal spending a portion of their time on HyDRO because it was a critical element of our research programs. In the last couple of years, we increased the number of people working on WeoGeo Market/Server, to >12 currently if you include outside contractors. For the most part, they are highly trained GIS and MIS/CIS/CS personnel.

The technology is hot, no question about it. I am amazed on a daily basis what our group of people has developed for mapping on both commodity computers and utility computing systems. Yet, here is the rub to this type of man-years calculation. I really believe that the reasons for WeoGeo, and its associated development time, stem from our history at FERI, which makes such calculations difficult. The “technical development time” is not just time spent coding; it includes the needs assessment and the development of the system architecture to address critical problems and/or pain. What we have developed at WeoGeo is a direct function of two critical needs of our operations as a research and imagery services organization.

These two critical needs were (and still are):
1) Delivery of our survey grade, high volume mapping content;
2) Finding and acquiring other survey grade mapping content to fuse with ours to create value-added geocontent for our clients.

WeoGeo was built to solve these two critical problems (there are others, but not nearly as critical to our organization as these). If you have never been faced with these problems, then you might not appreciate the depth of the solutions we have built to service these needs (and its potential). But if you have, then you have felt our pain - and I hope value our solution.

This weekend was a bit crazy for some of the AWS EC2 users. EC2’s “management software erroneously terminate[d] a small number of user’s instances” (from the AWS forum post). Some of our instances were among them providing an opportunity to test the fail-safe mechanisms in WeoCEO. We received the following email:

From: Amazon Web Services
Sent: Saturday, September 29, 2007 5:46 PM
To: David Kohler
Subject: Amazon EC2 Notification of Terminated Instances

Hello,

This is just a quick note to let you know that some of your instances were erroneously terminated today. We have resolved the underlying issue, and the service is fully available.

You can find a summary of the issue here:
http://developer.amazonwebservices.com/connect/thread.jspa?messageID=68169𐩉

These are your affected instances:
i-8004e0e9
i-681ef101

We apologize for this inconvenience.

Sincerely,

The Amazon EC2 Team

If we had not prepared for this by building WeoCEO, this could have been a real issue for us. We would have needed to scramble staff at 6 AM on a Saturday morning. Fortunately, WeoCEO recovered from the failure and it was not until Monday afternoon that we notice that it happened to a lot of other people.

From WeoCEO’s architect, Bob Banfield’s, forum post:

Here is a quick shot from our WeoCEO logs. We told WeoCEO that regardless of usage we want a minimum of two instances running, so that is the initial number of instances at 6am in the morning, even though we are receiving next to no traffic. At 6:09, i-681ef101 stops responding (the first of five allowed consecutive failures). At 6:10 it still hasn’t responded, and at 6:11 both it and instance i-52907e3b have now stopped responding. Instance i-52907e3b comes back up in another 2 minutes, but instance i-681ef101 is ruled dead after 5 failures. It is automatically terminated and a new one is brought up in its place.

(SSS) Sat Sep 29 06:07:24 2007 Weoceo[6562]: Overall usage = 0% NumInstances = 2
(SSS) Sat Sep 29 06:08:25 2007 Weoceo[6562]: Overall usage = 0% NumInstances = 2
(EEE) Sat Sep 29 06:09:25 2007 Weoceo[6562]: Instance i-681ef101 has not reported statistics (1/5)
(SSS) Sat Sep 29 06:09:25 2007 Weoceo[6562]: Overall usage = 0% NumInstances = 2
(EEE) Sat Sep 29 06:10:25 2007 Weoceo[6562]: Instance i-681ef101 has not reported statistics (2/5)
(SSS) Sat Sep 29 06:10:25 2007 Weoceo[6562]: Overall usage = 0% NumInstances = 2
(EEE) Sat Sep 29 06:11:26 2007 Weoceo[6562]: Instance i-681ef101 has not reported statistics (3/5)
(EEE) Sat Sep 29 06:11:26 2007 Weoceo[6562]: Instance i-52907e3b has not reported statistics (1/5)
(EEE) Sat Sep 29 06:11:26 2007 Weoceo[6562]: No instances have reported statistics.
(EEE) Sat Sep 29 06:12:26 2007 Weoceo[6562]: Instance i-681ef101 has not reported statistics (4/5)
(EEE) Sat Sep 29 06:12:26 2007 Weoceo[6562]: Instance i-52907e3b has not reported statistics (2/5)
(EEE) Sat Sep 29 06:12:26 2007 Weoceo[6562]: No instances have reported statistics.
(EEE) Sat Sep 29 06:13:26 2007 Weoceo[6562]: Instance i-681ef101 has not reported statistics (5/5)
(EEE) Sat Sep 29 06:13:26 2007 Weoceo[11310]: Terminating i-681ef101 due to lack of statistics
(SSS) Sat Sep 29 06:13:26 2007 Weoceo[6562]: Overall usage = 0% NumInstances = 1
(III) Sat Sep 29 06:13:26 2007 Weoceo[6562]: Launching 1 instance(s)
(III) Sat Sep 29 06:13:26 2007 Weoceo[11310]: Terminating 1 instance
(SSS) Sat Sep 29 06:14:28 2007 Weoceo[6562]: Overall usage = 0% NumInstances = 1
(SSS) Sat Sep 29 06:15:28 2007 Weoceo[6562]: Overall usage = 0% NumInstances = 1
(SSS) Sat Sep 29 06:16:29 2007 Weoceo[6562]: Overall usage = 0% NumInstances = 1
(SSS) Sat Sep 29 06:17:29 2007 Weoceo[6562]: Overall usage = 0% NumInstances = 1
(SSS) Sat Sep 29 06:18:29 2007 Weoceo[6562]: Overall usage = 0% NumInstances = 1
(III) Sat Sep 29 06:19:05 2007 Weoceo[11351]: Added ID=i-94ce20fd, PublicHost=ec2-67-202-13-222.z-1.compute-1.amazonaws.com, Host=domU-12-31-36-00-1D-B4.z-1.compute-1.internal, PublicIP=67.202.13.222, IP=10.253.34.66
(SSS) Sat Sep 29 06:19:32 2007 Weoceo[6562]: Overall usage = 0% NumInstances = 2
(SSS) Sat Sep 29 06:20:32 2007 Weoceo[6562]: Overall usage = 0% NumInstances = 2

Email warnings were delivered to me 6am on Saturday alerting me to the problem, however I was fast asleep and WeoCEO corrected identified and corrected the problem.

We believe in the future of scalable utility computing. Dealing with events such as these is just a part of the issues with these types of systems that we’ll all have to overcome to make this future work. Our goal is that we can share what we are creating for WeoGeo in a way that helps other overcome such problems.

I do not wish to minimize the impact of this API outage, but it would be unrealistic to assume that this type of event will not happen in the future. We should all consider this in building our virtual computing architectures. The use of AWS means that you are outsourcing your metal infrastructure. This means that your system design must be organic and self-healing (see also slideshare link).

WeoCEO was built to help us at WeoGeo survive these types of outages. We are completing our private beta shortly, and are releasing the latest version of WeoCEO that we will be bringing into open beta. Contact us at WeoCEO [at] WeoGeo [dot] com if you would like to participate. Open beta will provide the stable IP addressing and recovery options for one instance for free.

Our solution is simple to use and operate, but does expect that you have some working knowledge of EC2. There are others who can help in building these types of architectures on AWS from the ground up (some of those contributed to the above AWS Forum thread including Thorsten at RightScale and Reuven at Enomaly).

Please be aware of the limitation of utility computing, as well as the promise. Planning for these outages will be a requirement for safely outsourcing your metal resources.

I was out of town last week. I’ll try and catch up on a number of subjects this week.

One of the reasons I was out of town was that I was invited by AWS to present at their StartUp event in Boston.

A copy of the presentation may be seen on Slideshare.net (or just click on the image). It was a great event, and I enjoyed sharing the stage with the talented people from AideRSS, Praxeon, and Geezeo. It was good to interact with others who are building (and bootstrapping) new web services using AWS.

I truly believe that utility computing is going to change the way businesses get started and (eventually) operate. However, we are going to have to build systems that are organic in how they handle resources, i.e. scale up and down as a function of load. In addition, these systems need to be self-healing by automatically addressing processor and storage outages.

The importance of self-healing will be evident in the next post.