In this blog series, we’ve looked at WeoGeo API examples using the GET and POST HTTP Verbs. These were used to GET a List of Datasets and to POST a request for new Tokens in a WeoGeo Library. Today, we’re going to PUT an XML file in order to update the Name, Description, and Tags associated with a Dataset.
First, let’s take a look at the dataset to be updated. This Listing on my WeoGeo Library has dummy descriptive data for the name, description, and tags:
For the new values, I’ve created an XML file that contains the dataset tag and child nodes for each of the fields that I want updated:
I’ve saved that file locally to my local file system and will read it from the following Python script:
import base64
import string
import httplib, urllib
# Programmer defined variables
usern = "your api key"
passw = "" # Null string when API Key is used
library = "yourLibraryDomain" # the %s part of %s.weogeo.com
token = "the dataset token"
new_xml = "./data/new_xml.xml" #path in my file-system
# No more programmer changes
# set up the authentication string
auth = 'Basic ' + string.strip(base64.encodestring(usern + ':' + passw))
# create the header with the authentication string
headers = {
"Content-Type": "application/xml" ,
"Authorization": auth,
}
# read the XML data into body and print it
body = open(new_xml).read()
print body
# create the connection
conn = httplib.HTTPSConnection("%s.weogeo.com" % library)
# Construct the request with headers and XML in the body
conn.request("PUT","/datasets/%s.weo" % token, body, headers)
# call the URI
response = conn.getresponse()
#check status of response
print response.status, response.reason
# store the response in data
data = response.read()
# close the connection
conn.close()
# output the results
print data
This is very similar to the last example but note the following changes:
the HTTP verb PUT is used in place of POST
the variable body is used on line 29 where a null string was used before
Calling this Python script from the console outputs the following:
S:\>python put_example.py
<?xml version="1.0" encoding="UTF-8"?>
<dataset>
<name>Counties in the Southeast</name>
<description>
<![CDATA[
<h1>Big Header<h1>
Some <i>HTML</i> elements like <b>bold text</b> and
<a href="http://blogs.weogeo.com/dandye">links</a> are allowed.
]]>
</description>
<tags>TIGER shapefile political_boundaries counties</tags>
</dataset>
200 OK
S:\>
The 200 response indicates that all went well with the update and we can verify that by looking at the updated listing in the Preview Panel:
The Developer Documentation for WeoGeo’s API provides examples using curl. In this blog series, I’m providing similar examples using Python. In WeoGeo API: Python Examples, Part 1, I showed how to invoke the “GET Dataset as WeoFile” API from Python using urllib. In Part 2, I showed the same GET Dataset request using httplib. This time, we’re going to generate Dataset Tokens, which will require use of POST and https.
First, a little background: Dataset Tokens are Universally Unique Identifiers. There is one Dataset Token for each Dataset Listing in your WeoGeo Library. If your Dataset is also listed on the WeoGeo Market, the same Token is used there. Although the WeoApp can create Dataset Tokens for you when it creates a Dataset WeoFile, sometimes developers find it is useful to create a number of them in advance.
The Developer API Documentation shows that the curl request for Dataset Token generation is:
curl -u ${api_key}: -X POST https://${hostname}/datasets/tokens?count=2
There are three things of note here:
For authentication, only the API Key is used
The HTTP method is POST where previous examples have used GET
This API uses https rather than http
Here is a bare-bones Python example demonstrating the same API call:
import base64
import string
import httplib, urllib
# Programmer defined variables
usern = "your API key goes here"
passw = "" # Null string when API Key is used
library = "YourLibraryName" # the %s part of %s.weogeo.com
n_tokens = 2
# No more programmer changes
# set up the authentication string
auth = 'Basic ' + string.strip(base64.encodestring(usern + ':' + passw))
# Generate Tokens Curl Example (POST via HTTPS)
# curl -u #{api_key}: -X POST https://#{hostname}/datasets/tokens?count=2
# create the header with the authentication string
headers = {
"Content-Type": "application/xml" ,
"Authorization": auth,
}
# set parameters for the request
params = urllib.urlencode( { 'count':n_tokens } )
# create the connection
conn = httplib.HTTPSConnection("%s.weogeo.com" % library)
# Construct the request with headers amd add paramters to the URI
conn.request("POST","/datasets/tokens?%s" % params, "", headers)
# call the URI close the connection
response = conn.getresponse()
#check status of result
print response.status, response.reason
#check status of result
data = response.read()
# close the connection
conn.close()
# output the results
print data
For the three things of interest noted above:
API Key is set on line 5 and used on lines 11, 17, and 28
Line 28 sets the HTTP Request Method to POST
Line 25 uses httplib.HTTPSConnection (previous examples used httplib.HTTPConnection)
That’s it! Next time, I plan to show the PUT method as used in the Update Dataset API.
The Developer Documentation for WeoGeo’s API provides examples using curl. In this blog series, I’m providing similar examples but using Python. In WeoGeo API: Python Examples, Part 1, I showed how to invoke the “GET Dataset as WeoFile” API from Python using urllib. This time, I’ll show the same request using httplib.
Httplib is a little more complicated but it handles authentication slightly better and, more importantly, specifying the headers will enable more sophisticated requests (that will be required for later API examples).
The curl example for the API call is below:
And here is a bare-bones example of that request in Python using httplib:
import base64
import string
import httplib, urllib
usern = "your_username" # or your API key
passw = "your_password" # if API Key is used: = ""
library = "the * part of your Library's domain name in *.weogeo.com"
# Construct the Basic authentication string
auth = 'Basic ' + string.strip(base64.encodestring(usern + ':' + passw))
# List Datasets (GET) curl example
# curl -H 'Content-Type: application/xml' http://#{hostname}/datasets.weo?page=1&east=0&north=90&south=0&west=-180&scale=5
# Optional parameters:
params = urllib.urlencode({
'page': '1',
'per_page': '15',
'north': '90.0',
'south': '-90.0',
'east': '180.0',
'west': '-180.0',
'date_from': '1990-01-01',
'date_to': '2010-01-01',
'data_type': 'RASTER',
'min_provider_rating': '0',
'min_dataset_rating': '0',
'tags': 'test'
})
# create the header with the authentication
headers = {
"Content-Type": "application/xml" ,
"Authorization": auth,
}
# create teh connection
conn = httplib.HTTPConnection("dandye.weogeo.com")
# Construct the request with headers amd add paramters to the URI
conn.request("GET","/datasets.weo?%s" % params, "", headers)
# call the URI
response = conn.getresponse()
# check status of result
print response.status, response.reason
# read the data retreieved
data = response.read()
# close the connection
conn.close()
# print the data retreieved
print data
Notice on Line 9 that your username and password are base64 encoded. This is basic access authentication and encoding != encrypting. So for added security, you may use your Library API Key in place of your username and leave your password blank (password = “”).
Again, for extra credit, try changing the extension (on line 40) from “.weo” to [".kml",".json",".rss"].
This week we released an update to the WeoGeo Library and Market services. 2.0.10 is now up and running and you can find the details on our change log.
One of the biggest features is the new free tier to WeoGeo Library. Yep, we now have free WeoGeo Library accounts. You can now start using WeoGeo for free, the only limitations are number of users and storage size. You can of course easily upgrade to paid tiers when you need to. So try out our WeoGeo Library for free (no credit card needed) and see what it can do for you. When signing up use the invite code: freeweogeolib
curl example in WeoGeo Developer Documentation (wiki.weogeo.com)
However, after fielding some questions from Developers, I’m thinking some Python (and C#?) examples might be useful too. So here is the first installment.
This is a bare-bones example of the List Datasets API, which returns a list of Datasets within the (optional) defined parameters. Possible parameters include a Minimum Bounding Rectangle (MBR), date range, ratings, and/or Tags.
import urllib
usern = [your WeoGeo username]
passw = [your WeoGeo password]
library = [the * part of your Library's domain name *.weogeo.com]
# List Datasets (GET) curl example:
# curl -H 'Content-Type: application/xml' http://#{hostname}/datasets.weo?page=1&east=0&north=90&south=0&west=-180&scale=5
# Optional parameters:
params = urllib.urlencode(
{
'page': 1,
'per_page': '15',
'north': '90.0',
'south': '-90.0',
'east': '180.0',
'west': '-180.0',
'date_from': '1990-01-01',
'date_to': '2010-01-01',
'data_type': 'RASTER',
'min_provider_rating': '0',
'min_dataset_rating': '0',
'tags': 'test'
}
)
# construct the URI for the API with parameters:
url = "https://%s:%s@%s.weogeo.com/datasets.weo?%s" % (usern,passw,library,params)
# call the URI
f = urllib.urlopen(url)
# print results to the CLI
print f.read()
For extra credit, try changing the extension from “.weo” to [".kml",".json",".rss"].
Next time, I’ll show the same WeoGeo API call in Python but use httplib instead of urllib. Although, urllib is simpler, httplib handles authentication better and enables the more sophisticated requests that will be required for later API examples.
Last week our 2.0.9 release went live enabling a couple of new features and cleaning up a few stragglers from the last release. You can always read the details at our wiki page, but I’ll highlight a couple here that are worth noting.
Human Readable URLs: Prior to this release, a dataset had a token in the URL such as “4a7df1a5-af77-ab5a-a0b9-59a97b6d03d4″. This token was a unique identifier to the dataset, but you couldn’t tell what the token was unless you clicked on it. Now URLs are based on the name of the dataset. Thus if I have a dataset called “City Roads”, the URL will now use that in the URL.
You can still use the tokens as before, but the URLs will default to the name of the dataset.
CSV (comma separated values): You can now convert vector datasets into CSV files. This means that users that don’t have access to GIS software can get at the data values behind the shapefiles.
GeoRSS Feeds Improved: Our last release introduced GeoRSS files. Now at 2.0.9 the RSS feed gives you updated datasets in your area of interest defined by the bounding box you set either in the the WeoGeo website or by customizing the URL yourself.
Performance improvements: Added caching to improve the performance of the web app.
Improved user activity stats for administrators: Library administrators can get better analytics to determine how datasets are being downloaded and used.
If you are interested in taking the WeoGeo Library or Market for a test spin, please contact us and let us know you’d like to try it out. Also please let us know if there are any enhancements or features you’d like to see enabled into our Library/Market or APIs.
The GeoMonkey is already working with the WeoGeo APIs to convert MapInfo TAB to CSV.
We’ve gone ahead and added GeoRSS support for our listings in both the private Library and the open Market. On the top of Panel 3 you’ll see a RSS icon which leads to a GeoRSS feed that gives updates to whatever search criteria you have set including the bounding box (your area of interest), any keywords search, modified dates, file type (vector/raster/other) or price (on Market).
Panel 3 now has a GeoRSS Link
By zooming into the state of California, you can see the GeoRSS feed includes only datasets in that area of interest. Any time a new dataset is loaded, you’ll know about it because it will appear in your RSS aggregator or even in your GIS applications. Or how about this feed for ATLIS Geomatics which shows all their datasets in Calgary, Alberta. Providers of data on WeoGeo can give their users GeoRSS feeds to keep them updated on new uploads.
The OpenGeoData Blog had a very interesting post last week on what they called the cake test. From the blog:
“What is the Cake Test? Easy: A set of geodata, or a map, is libre only if somebody can give you a cake with that map on top, as a present.”
I actually like this test. It is simple and easy to see. Can you download geodata and have your local baker reproduce it with icing. But there is a little problem with their definition; it is way too narrow. Most of the data in the world is not libre or free. It is either locked up at one of the various stages of local and federal government, or it can be owned by a company/individual who doesn’t want to give it away for free. The cake test does work in this aspect, if you can take a dataset and put it on a map without signing anything or paying for anything, it is most likely libre. But the reality of the situation is that most data isn’t free and probably won’t be in the near term. This means we need another way to license geodata on top of that cake.
Derivative Work from geodata is very liberating
We at WeoGeo have been thinking about this for a long time. In fact we realize that without a good license for allowing users to create derivative products from geodata, that information is probably going to stay locked up behind restrictive licensing and cause people to create their own datasets without them. Our WeoGeo License Agreements for Providers and Users both oversee how content can be licensed to allow derivative works to be created. If you’d like to use this aerial image of download Calgary, you will get a license that allows you to create the single use from this aerial which would include putting it on a cake for a (yummy) mapping party.
Now of course this aerial isn’t libre, and you are still bound by the license agreement. However this license also explicitly states that you could sell this cake as a derivative product to others and a potion of that sale would go back to ATLIS Geo. But what it does do is give users of geodata a clear and consistent way to actually use content to create derivative products and still give the original rights holder a cut of future sales. So if you list data on the WeoGeo Market, you can give buyers of your geodata the rights to create derivative work based upon it and gain an additional revenues from those downstream sales. Remember EVS-Islands and DigitalGlobe from a while back? All he wanted to do is license derivative work off of the imagery. Because DigitalGlobe has no such license, we can’t enjoy his hard work and DigitalGlobe gives up some revenue that they wouldn’t have had otherwise.
The cake test as proposed by the OpenGeoData blog is an all or nothing test that results in the many users gaining little benefit. Sure it is nice that OpenStreetMap, TIGER and other datasets can be thrown on top of a cake, but most data available to users is not libre. If the owners of this data were to enable derivative works using the WeoGeo Content License Agreement or similar type licenses, you can only imagine some of the really great derivative products people would be able to create and help to continue funding further development of these datasets. Don’t we all want our geodata to pass the cake test and ensure that you have something tasty to celebrate on Geography Awareness Week?
There have been some interesting conversations across the blogosphere on Google’s entrance into the basemap and navigation business. From an interested spectator point-of-view, I am a bit of a fan of the sheer audacity of Google’s mapping effort; creating a US basemap from scratch, building a navigation application, and rolling out a real-time sensor network (with 76 million devices) disguised as a mobile phone, all in the space of a couple of years. Well, that’s just crazy. However, as much as I am a fan, this event is just one more example of the impacts of rapidly changing technology in our field (Part 1), making previous niches obsolete and forcing individuals (James Fee) and companies (e.g. Cloudmade) to adjust business models to just survive.
Geo-powered apps are changing the game for developers
There were a couple of themes from these conversations that I found interesting. The first was represented by Peter Batty (and links therein), which could be broadly interpreted as “Content Matters”. That is, the cost (or lack there of) of the navigation application was less than important than the base content within it. The second theme was summarized by a podcast at Directions Magazine, which posited that the data didn’t matter. Adena and Joe’s point of view was that the application was going to overshadow the importance of the content. They felt that the application cost, functionality, and ease of use would generate more relative value than the content. I personally believe that both these points are two sides of the same business coin, and to be successful in this space you will have to have both great content, and a great marketing and delivery application.
Let’s start with the Batty et al.’s assertion that “Content is King”. In a frictionless marketplace of ideas and data, where the process of disintermediation has reached its peak, the lines of competition will be drawn around product price, branding (think Crest versus Colgate) and quality (think GM versus Lexus). This suggests that our market will continue to separate between the aggregation of (free) commodity data (10 meter USGS Digital Elevation Maps) and premium data (nationwide parcel data). I have been a proponent of premium content for a couple of years and I see it as a bright spot for geospatial professionals because good content tends to be localized providing niche geo-opportunities locally, across the globe.
Wall Street Journal's premium content drives growth
The Wall Street Journal has maintained its subscription-based revenue model all through the “content-should-be-free” era of the internet. It is now the only major newspaper in the US with a growing circulation, and is the nation’s largest newspaper by circulation. They demonstrate that the quality of content matters, and will continue to matter moving forward. Looking more closely at the difference in content provided by the #1 Wall Street Journal and the #2 USA Today, the Wall Street Journal might be considered a “premium” niche player, compared to the generic “consumer” news offer by the USA Today. Generic news may be found in a lot of places (for free). In the spatial industry, we might compare this to the “free” generic DEMS offered by the USGS and the “premium” high-resolution LiDAR surveys by companies such as Merrick. Clearly, quality content has value and people are willing to pay for it.
But Adena and Joe have a point. We consume digital content differently than other commodities such as toothpaste. Even if today’s premium navigation data becomes a commodity by virtue of the “less-the-free” data offered by Google, the data must still be delivered via a service and application that people wish to use. The consumption of content, and the value-added services built on top of the content, will depend directly on the quality of the application that delivers that content. In this case, the application will matter as much as the content. If either fails to deliver, the whole effort will fail.
While we may currently consider ourselves either data providers or application developers/integrators, I think our industry will move towards a Data-as-a-Service (DaaS) industry, where the data and applications are inextricably tied. I believe this will be the case for all platforms, including web, mobile, and desktop. And in this new DaaS era, the geospatial integrators may have an edge because they are used to thinking about data and applications simultaneously.
In addition, the economic process that is currently squeezing the revenues of integrators and GIS professionals may be turned to work in their favor, as the many sources of free, low cost, or open source software becomes their application supply chain. In the past, these integrators thought about their customer’s data and its integration within a proprietary mapping software stack. In the future, they have an opportunity to become vendors of DaaS, either for their customers, or for themselves. The combination of accessible data (low-cost or free) with low-cost applications and hosting will enable these professional to create premium DaaS niches, recouping the profit margins lost to the commodization of their IT skill set.
This is one path towards a new, hopeful future where the geospatial professional controls the fate of their destiny, rather than trying to survive on the leavings of giants.
James Fee made an eloquent case for why he made the leap to WeoGeo. While I would like to claim the powers of a Jedi knight, I think the true motivation of his choice was the hard economic realities of the spatial IT business. As James mentioned, the pricing pressures in basic spatial IT integrations are increasing, which are resulting in a falling revenue flow for many integrators.
The future of Spatial IT as a technology sub-discipline.
I think this is happening for many reasons, but here is the major one. Spatial technology is becoming more robust and easier to use for repetitive business functions, i.e. building a slippery map that shows points-of-interests (POIs), lines, and polygons no longer requires a specialized GIS technology stack. Just getting an organization’s information on a map that can be viewed internally (intra-net) or externally (inter-net) used to pay a lot of bills. The problem is that it doesn’t anymore. To a large extent, we are the victims of our own success at demonstrating the power of spatially-enabled business content and services.
In addition, for other types of higher-order analysis, display, and spatial enterprise operations, you don’t need a proprietary specialized database any more (i.e. Oracle Spatial 11g ) as Microsoft SQL 2008 and others built geometry and geography natively into their applications. And of course there are the plethoras of open source options that allow you to avoid proprietary databases all together (e.g. PostGIS). With these databases, one does not need the spatial data engine abstraction layers (e.g. ESRI’s SDE, which might be why they quit selling it as a stand-alone product) to expose your organization’s spatial data to those that need it or other applications to consume it. These spatially enabled databases also provide for some high-order geospatial analysis to be preformed without the need of desktop- or server-side products (like ArcGIS Desktop or Server), and in many cases without the need for GIS Professionals to run that analysis.
What does that mean for the specialized services of the spatial technology integrator? The simple mapping stuff will be just part of the web programming stack, with little separation between web programming and spatial web programming. Spatial technology integrators will have to evolve to create more value from enterprise technology operations, where spatial is just one part of their enterprise project. This can be successfully done, and one only has to look at Dave Bouwman’s group DTSAgile (which is just kickin’ it) to see that it can be accomplished.
However, the competition will be fierce because the specialized spatial IT stack will evolve into the plain vanilla IT stack, with more competitors and easier-to-use technology. This will mean much lower margins per spatial project; and that is just the way of the economic “force”. In addition, I have been hearing stories of increased competition from specialized software vendors like ESRI for consulting revenues on increasingly smaller and smaller projects. This suggests that the integrators will be squeezed from multiple directions, setting up the potential for a shakeout for integrators in our industry.