Google is not a web search company, but rather Google is an advertising company (they generated 97% of their revenues from advertising in 2009).  Search is the candy they give away to you so they can market advertising services.  Granted without search they have no advertising sales, so it is hard to separate Google from search.  But the point is that the company uses search as the marketing material to sell its products.

In previous posts (here and here), I have made the case that the mapping industry will change because players like Google and Microsoft (and Apple) will create mapping services embedded in their products as a feature enhancement.  The dollars that these companies are willing to spend on creating mapping products as features to their main business lines create “gravitational” distortions in the geospatial industry.  (For example TomTom stock is down ~60% since Google canceled its Tele Atlas contract in October 2009 and released its own turn-by-turn navigation system for mobile phones using the Android operating system. )

What was a bit unexpected was that other professional software companies would follow this free “Data-as-a-Feature” business model.  Yet this is exactly what ESRI has done.  Over the past couple of years, they bought a big piece of i-cubed to provide raster imagery products to their customers, and they have cut deals with data vendors, such as Microsoft Bing Maps, DeLorme, Tele Atlas, and others.  They have brought free data to light from federal agencies like USGS.  And they are combining these sources (with their stellar cartographic capabilities) to create derivative products that are extremely useful and appealing.

Just look at this topographic synthesis product from ArcGIS Online.  At the FedUC, Jack showed 1:1000 scale topographic data for selected cities within a worldwide map.  Nice and oh and by the way, it is free to users of ArcGIS Desktop products who maintain their license.

Data-as-a-Feature for ArcGIS Users

ESRI is a software company and generates most of its revenues via software sales and maintenance.  In fact, I’ve heard ESRI claim they have >90% of the core professional GIS market.  Data is the “new” free candy that ESRI gives away to maintain its market dominance in GIS software.  You have to admire the strategy to leverage their market dominance.  Data is a necessary “feature” for working in the GIS field.  Getting quality data, styled with beautiful cartography, as part of your content creation tool is a great benefit to users of that tool.

Software customers may be happy with data-as-a-feature; and anecdotal stories from GIS software integrators and solution providers suggest their customers are quite happy with the free regional products ESRI.  However, for independent data vendors it is a scary prospect.  If you produce regional or worldwide data, you can either sell your product to ESRI at “their” price, or face the prospect of losing access to their ecosystem of GIS developers by virtue of ESRI (re)creating the free product.

Life for independent data vendors has become more difficult.  On one hand they have to worry about Google, Microsoft, and Apple creating free worldwide data sets and applications to enhance their products.  Yet, I think these data vendors may have gotten use to this new paradigm and sought refuge in the “professional” data quality niche.  However, another hand is now in the picture (and in their till) with free professional data.  That hand is ESRI with its free Data-as-a-Feature.

OK, lets be honest, talking about content management is about exciting as talking about metadata, or XML, or Canadian fiscal policy. I mean really, who cares about managing your information in ways that make it discoverable, usable and most importantly sharable.

Oh right, maybe this stuff isn’t as boring at we thought. I’ll tell you what is the most boring thing in the world. Watching your desktop search strain to find data on your local drives with wacky search strings like water.*. Think about how you find GIS data on your hard drives? Search or browse. Heck, you might even go back to an old project and try and find the dataset that was stored in my MXD or other map document.

Now this doesn’t mean that using search tools to find data isn’t valuable. I know many folks that use Google Desktop Search to index their GIS files. It just isn’t efficient and it sure as heck doesn’t allow you to share it with multiple users. Using these private search indexes is just like putting the data on your desktop, sure you can find it, but no one else can. It gets to the core why content management of geospatial data is so important and why it is actually really fun to talk about.

WeoGeo’s approach to content discoverability is to use the one thing that makes finding maps simple. A map! If I want to find hydrography layers in Wisconsin, I pan/zoom to Wisconsin and then search for hydro tags. If I want a worldwide country boundary layer, I zoom to the world and search tags for country. If I want to find demographic data from Arizona, I geocode that in my WeoGeo search and zoom right to that area.

WeoGeo manages content using maps

Textual based search is horrible at finding spatial data. Does “washington” mean the state, which of 31 counties, which of almost 50 cities and the countless lakes, islands, streets, neighborhoods and other place-names? A map allows you to narrow that down to the Washington that you are interested in and not have to parse though hundreds of records for other “Washingtons”.

Sharing Your Geo-Data

OK, so we’ve all got these issues finding data on your hard drives. That is part of the problem though. Sharing this data amongst your peers is hard as well. Not everyone has that GIS data hard drive mapped, the VPN is problematic and slow causing off-site users to not have access and it is too large to email to your clients who want to see it. This makes it very hard to get the most value out of this dataset because you are fighting to share it as much as you are fighting to find it.

What a content management system such as WeoGeo brings into play with our web-based front end is a simple way for many people to access the same dataset from many different places. Across the hall, across the building, across the state and across the world. With user access controls, you can set which datasets are available to which users. Grant rights for your clients to gain access to your geodata, but see only those datasets that they are supposed to see. No worries about them rummaging through your FTP site looking for proprietary information that shouldn’t be seen.

Add users and give them access to datasets

Geo-Formats In and Out

One of the greatest fears I always had with archiving data was will it be usable in the future. I just found a CD the other day with a ton of Corel Draw documents on it. Nothing I had on my laptop could read those old .cdr files despite claims from the software package that it could. What about GIS data? I’m pretty sure that your shapefiles and GeoTIFFs will be readable in the future, but what about more obscure data formats? Will you be able to read that vector or raster format in 10 years? Who knows right? But WeoGeo has the industry standard on reading proprietary data files under our hood. Safe Software FME drives all our data format transformations. Safe FME allows WeoGeo to transform data stored in our Library to any format they support. So if you need to get your data out of Integraph MGE, because your company no longer supports it, into Autodesk SDF format, you can do that with FME and WeoGeo. Protecting your archival, system of record, datasets from obsolescence is just as important as making sure that you can find it.

Content Management Within Your Workflows

There are tons of content management systems out there and for good reason. Companies want to keep control of their data because they can get more value out of it than they would storing it on DVDs in your desk drawer or on a DLT tape at Iron Mountain. Neither of those methods is a form of content management and neither makes your data discoverable or shareable.

What sets WeoGeo apart from most content management systems is that we are first and foremost a GEO content management system. We are designed from the ground up to allow you to manage all your geo-content; whether it be shapefiles, ERDAS or anything that has a reference to place (PDF, Word documents, video). We’ve also worked hard to integrate WeoGeo into your workflows with our new WeoGeo Tools for ArcGIS and our WeoApp which runs on Windows, Mac OS X or Linux. Using content management shouldn’t be hard or require lots of training. We’ve made it as simple as “upload to WeoGeo” and “browse WeoGeo” to get data in and out of our content management system.

WeoGeo Tools for ArcGIS in action

Your return on investment putting your data in WeoGeo will be realized quickly when you no longer have to spend hours finding your data or invest in expensive, closed, proprietary content management systems. You can start using WeoGeo for Content Management today for free and see how it can change how you manage your geodata.

crowd sourced data is playing an integral role in relief efforts. It is a role, though, that complements the traditional sources of geospatial information. The largest impacts we’ve seen is in the fusion of the two sources. The combination has incredible value providing baselines, context and temporal adaptability – resulting in a malleable abstract that can be molded to solve a myriad of disparate challenges.

I agree with Sean’s sentiments. There is no need to try and draw arbitrary bounds around the type or method of collection of geospatially-aware data. The data’s use defines its value. Let’s focus on solving challenges and creating value for our neighbors, stakeholders, citizens, and customers.

Photo by Alfred Eisenstaedt, taken on V-J Day, 1945 (from Life Magazine)

One thing about using a “cloud based” data store is that you have to leave your tools to start using it. This means that you break any work-flows you are in and lets be honest, the last thing you want to happen when you are being productive is to head to the internet.

Seriously though, being able to stay in your content creation tools and work with SaaS based content management is a great idea. The reality of the situation is that there hasn’t been any good tools to accomplish this yet.

Enter WeoGeo Tools

Today we are releasing the first of our planned content creation tools that bring the SaaS WeoGeo Library and WeoGeo Market as well as any WeoGeo Library Appliances to ArcGIS Desktop users. With WeoGeo Tools for ArcGIS you can connect to these libraries and discover datasets in your area of interest. Inside ArcMap this area of interest is defined by your view window. So if you’ve got a project you are working on and want to see what is available in your work area, WeoGeo Tools takes your extents and returns only datasets in that region.

Text-based Search is Wrong for Geo-Content

WeoGeo has worked hard at changing the way we search for geospatial datasets. Rather than requiring users to type in keywords or other text search terms into a search query, we allow users to discover data via maps. You can see this in our WeoGeo Market. What is natural though about content creation tools like ArcGIS Desktop is that your canvas is a map. Why not take that bounding box extent and return data in that area? Our MapRank technology also returns datasets that best match your zoom level (if you are zoomed into a neighborhood you’ll get those datasets listed first and likewise if you are at the country level, you’ll get those datasets before neighborhood ones.

Uploading From Inside ArcGIS Desktop

Getting your content up to your Library or the Market should be just as easy. Why do you have to save out your data and then open your web browser and then proceed to fill out data forms and finally upload? Yep, that is what we call a workflow killer. With WeoGeo Tools for ArcGIS, ESRI users can easily upload datasets from within ArcMap with just a click of a button. Everything is handled for you. Just highlight the dataset in your layers list and click the upload to WeoGeo button.

Download WeoGeo Tools for ArcGIS

I encourage everyone to download WeoGeo Tools for ArcGIS and give it a shot. You can browse the WeoGeo Market without having to have an account so you can see how easy it is to bring data in. If you want to try out our Library, we offer free Library tiers and 30 day trials for the rest. Just use signup code: weogeolib5769. Lastly if you need any help, don’t hesitate to contact us.

This is our first release of WeoGeo Tools for ArcGIS and we will continue to work hard on improving it, so any feedback is welcomed. We are also working at more WeoGeo Tools for other GIS software packages. You can also create your own versions of WeoGeo Tools for your own software packages by using our open APIs.

Visit Us at the FedUC

If you are at the 2010 ESRI Federal User Conference, feel free to drop by out booth (#141) and get a full demo of WeoGeo Tools for ArcGIS as well as all our SaaS and appliance products.

Here is a high resolution WeoGeo icon to use with Fluid. (Just click to download)

Paul, Dan and James will all be there on the floor and even if you aren’t going to the FedUC, we’d be happy to meet with you in the area to talk about WeoGeo Library and Market.  The exhibit floor is open Wednesday, February 17 from 3:00 – 6:00 PM and Thursday, February 18, from 10:00AM – 4:00 PM.

Worried about the weather?  ESRI says don’t let that stop you.  It isn’t going to stop us from showing up.

After creating the cartography, I export out the preview images and use our WeoApp to upload them.  The WeoApp is our graphical upload tool that can handle terabytes of data and runs natively on Windows, Mac OS X and Linux.  Once the WeoApp finishes uploading the data, one can now view it in your WeoGeo Library.  This is our access control content management system; so feel free to securely share your data with your friends and colleges.  The Library is available as SaaS or as an appliance.  Because I want to share this data with people outside my private library (and let you guys see it), I pushed it to our WeoGeo Market.

While I could set a price to this dataset, I decided to give it away free.  Now you might be thinking that why would someone want to sell public domain data on WeoGeo Market.  Well you probably wouldn’t make much money doing that because people would quickly realize that you were charging them for free data and rate your products poorly.  BUT, if you added value to that data by creating a complex cartographic styleset and spent weeks creating it, why wouldn’t you want to charge money for that?  People should value the hard work you put into your symbology even if the data is free.  Plus with our WeoGeo License, derivative works are supported so others can remix your work and resell it as well, but you’d get a cut of the sale yourself.

Now that this dataset is freely available on the WeoGeo Market, you may now try our customization features yourself.  You could subscribe to a GeoRSS feed that would keep you up to date on any public domain datasets that become available on the WeoGeo Market.  You can share the preview KML with others so they can see the dataset in Google Earth (and then get to WeoGeo Market to download it).  You can even use our Datasets API to get the details of the entry in JSON.

Now I’m a vector guy a heart, so vector support in WeoGeo is just about the most exciting thing I’ve been a part of.  With WeoGeo’s ability to perform server side transformations, users can get what data they want in formats and projects they want.  No more worries about users wanting data in everything from Old Hawaiian to EPSG:4326.  And best of all if you don’t like our UI, you can roll your own with our APIs.  Build your own front end to our SaaS and Appliance Library to meet the needs of your clients and users.

Best of all, stay tuned next week to hear about a great new product we will be releasing to take your geo-content management beyond where you thought it could go.

Next week my friends!

First, let’s take a look at the dataset to be updated.  This Listing on my WeoGeo Library has dummy descriptive data for the name, description, and tags:

Using the GET Dataset as WeoFile API in a web-browser, we can view this descriptive data as XML:

For the new values, I’ve created an XML file that contains the dataset tag and child nodes for each of the fields that I want updated:

I’ve saved that file locally to my local file system and will read it from the following Python script:

import base64
import string
import httplib, urllib

# Programmer defined variables
usern = "your api key"
passw = "" # Null string when API Key is used
library = "yourLibraryDomain" # the %s part of %s.weogeo.com
token = "the dataset token"
new_xml = "./data/new_xml.xml"  #path in my file-system
# No more programmer changes

# set up the authentication string
auth = 'Basic ' + string.strip(base64.encodestring(usern + ':' + passw))

# create the header with the authentication string
headers = {
 "Content-Type": "application/xml" ,
 "Authorization": auth,
 }

# read the XML data into body and print it
body = open(new_xml).read()
print body

# create the connection
conn = httplib.HTTPSConnection("%s.weogeo.com" % library)

# Construct the request with headers and XML in the body
conn.request("PUT","/datasets/%s.weo" % token, body, headers)

# call the URI
response = conn.getresponse()

#check status of response
print response.status, response.reason

# store the response in data
data = response.read()

# close the connection
conn.close()

# output the results
print data

This is very similar to the last example but note the following changes:

1. the HTTP verb PUT is used in place of POST
2. the variable body is used on line 29 where a null string was used before

Calling this Python script from the console outputs the following:

 S:\>python put_example.py
<?xml version="1.0" encoding="UTF-8"?>
<dataset>
 <name>Counties in the Southeast</name>
 <description>
 <![CDATA[
 <h1>Big Header<h1>
 Some <i>HTML</i> elements like <b>bold text</b> and
 <a href="http://blogs.weogeo.com/dandye">links</a> are allowed.
 ]]>
 </description>
 <tags>TIGER shapefile political_boundaries counties</tags>
</dataset>
200 OK

S:\>

The 200 response indicates that all went well with the update and we can verify that by looking at the updated listing in the Preview Panel:

First, a little background: Dataset Tokens are Universally Unique Identifiers.  There is one Dataset Token for each Dataset Listing in your WeoGeo Library.  If your Dataset is also listed on the WeoGeo Market, the same Token is used there.  Although the WeoApp can create Dataset Tokens for you when it creates a Dataset WeoFile, sometimes developers find it is useful to create a number of them in advance.

The Developer API Documentation shows that the curl request for Dataset Token generation is:

curl -u ${api_key}: -X POST https://${hostname}/datasets/tokens?count=2

There are three things of note here:

1. For authentication, only the API Key is used
2. The HTTP method is POST where previous examples have used GET
3. This API uses https rather than http

Here is a bare-bones Python example demonstrating the same API call:

import base64
import string
import httplib, urllib

# Programmer defined variables
usern = "your API key goes here"
passw = "" # Null string when API Key is used
library = "YourLibraryName" # the %s part of %s.weogeo.com
n_tokens = 2
# No more programmer changes

# set up the authentication string
auth = 'Basic ' + string.strip(base64.encodestring(usern + ':' + passw))

# Generate Tokens Curl Example (POST via HTTPS)
# curl -u #{api_key}: -X POST https://#{hostname}/datasets/tokens?count=2

# create the header with the authentication string
headers = {
 "Content-Type": "application/xml" ,
 "Authorization": auth,
 }

# set parameters for the request
params =  urllib.urlencode( { 'count':n_tokens } )

# create the connection
conn = httplib.HTTPSConnection("%s.weogeo.com" % library)

# Construct the request with headers amd add paramters to the URI
conn.request("POST","/datasets/tokens?%s" % params, "", headers)

# call the URI close the connection
response = conn.getresponse()

#check status of result
print response.status, response.reason

#check status of result
data = response.read()

# close the connection
conn.close()

# output the results
print data

For the three things of interest noted above:

1. API Key is set on line 5 and used on lines 11, 17, and 28
2. Line 28 sets the HTTP Request Method to POST
3. Line 25 uses httplib.HTTPSConnection (previous examples used httplib.HTTPConnection)

That’s it!  Next time, I plan to show the PUT method as used in the Update Dataset API.

Httplib is a little more complicated but it handles authentication slightly better and, more importantly, specifying the headers will enable more sophisticated requests (that will be required for later API examples).

The curl example for the API call is below:

And here is a bare-bones example of that request in Python using httplib:

import base64
import string
import httplib, urllib

usern = "your_username" # or your API key
passw = "your_password" # if API Key is used: = ""
library = "the * part of your Library's domain name in *.weogeo.com"

# Construct the Basic authentication string
auth = 'Basic ' + string.strip(base64.encodestring(usern + ':' + passw))

# List Datasets (GET) curl example
# curl -H 'Content-Type: application/xml' http://#{hostname}/datasets.weo?page=1&east=0&north=90&south=0&west=-180&scale=5

# Optional parameters:
params = urllib.urlencode({
 'page': '1',
 'per_page': '15',
 'north': '90.0',
 'south': '-90.0',
 'east': '180.0',
 'west': '-180.0',
 'date_from': '1990-01-01',
 'date_to': '2010-01-01',
 'data_type': 'RASTER',
 'min_provider_rating': '0',
 'min_dataset_rating': '0',
 'tags': 'test'
})

# create the header with the authentication
headers = {
 "Content-Type": "application/xml" ,
 "Authorization": auth,
 }

# create teh connection
conn = httplib.HTTPConnection("dandye.weogeo.com")

# Construct the request with headers amd add paramters to the URI
conn.request("GET","/datasets.weo?%s" % params, "", headers)

# call the URI
response = conn.getresponse()

# check status of result
print response.status, response.reason

# read the data retreieved
data = response.read()

# close the connection
conn.close()

# print the data retreieved
print data

Notice on Line 9 that your username and password are base64 encoded.  This is basic access authentication and encoding != encrypting.  So for added security, you may use your Library API Key in place of your username and leave your password blank (password = “”).

Again, for extra credit, try changing the extension (on line 40) from “.weo” to [".kml",".json",".rss"].