Background, Remote Sensing, Hyperspectral, WeoGeo, FERI, mapping, BigTIFF
What file format do you use for a 40GB image? (BigTIFF!)
Large imagery files are a problem. In the hyperspectral world, we send things via ENVI’s file format (BSQ, BIL, or BIP). ENVI was designed by folks doing HSI remote sensing and was optimized to easily handle large raster images. The use of this file format allows us to deliver extremely large raster files, with a separate header that described all the channels, bands, or layers in the image.
Unfortunately, not everyone owns a copy of ENVI. It is an expensive image processing package. While other remote sensing and GIS packages claimed to handle multi-band imagery data, we found that support for imagery with bands n > 3 was difficult at best. So if our customers at FERI didn’t have ENVI, the transport of the imagery had to be accomplished in another file format. The most common format other than the ENVI format for us was GeoTIFF.
Unfortunately, the GeoTIFF format is limited to 4 GB. This is clearly problematic for the image shown here in Figure 1.
Figure 1. HSI imagery of St. Joseph Bay, FL (click on the image to see the data set at WeoGeo Market.)
This image is 156 band hyperspectral mosaic. The entire image at is native spatial resolution equals 40 GB in size. Cutting this data into 10 tiles of 4 GB a piece would be one way to deliver this data set. But this is problematic for both us and the receiver of the images, as the time, energy, and effort to tile and then re-mosaic is less than efficient.
You could also say that for the most part that HSI data is a relatively small backwater of the remote sensing community, so why worry about it. To this I would respond with this imagery that we collected at the same time in Figure 2.
Figure 2. 3-Band DSS imagery of St. Joseph Bay, FL (click on the image to see the data set at WeoGeo Market.)
This is a 3-band RGB from an Applanix DSS. The resolution was about 1/6 the spatial resolution of the HSI sensor. The higher spatial resolution makes this image nearly as large as the HSI image. We actually incurred the pain of tiling the full image set for our original customer because they had only ESRI software with which to analysis this image.
Our friends at GDAL asked us about sponsoring a new file format, BigTIFF, which would be based on extending the TIFF format. We were happy to step up to help make this happen. I believe that the other sponsors had similar file storage and distribution issues, and we look forward to broad acceptance of this file format.
It will certainly make our distribution issues easier.
28 Aug 2007 Paul Bissett
6 Comments on “What file format do you use for a 40GB image? (BigTIFF!)”
-
06 Sep 2007 at 10:52 am 1 Fiducial Marks » 40 GB Imagery File Redux
[…] An obvious question that drops out of yesterday’s post on the right file format to use to distribute large raster files is, “How do you distribute a 40 GB file?” The distribution of a single 40 GB file would overwhelm the bandwidth of many small businesses. That was one of the reasons we originally developed the WeoGeo Server. […]
I would use GDAL’s virtual raster (VRT) to point to multiple ‘normal’ TIFFs. From that I can for instance generate one big ECW. Only problem in your case is that AFAIK ECW doesn’t support hyperspectral.
If you have generated overlays for your tiff, several image tools will handle any image GDAL reads (since thats what they are using), including VRT and this would make it somewhat efficient to work with, without even compressing it.
Hi!
Can’t GDAL read ENVI files? We use an in-house app to regularly open ENVI files directly through GDAL bindings without the need to export to GeoTIFF or BigTIFF. Never tried with hyperspectral data but I can’t see why it would not work
@ Morten - I had not heard of GDAL VRT structure until now. Looks interesting.
But in regard to your main point, Paul can transport the 40GB file directly using ENVI’s format currently. That format is just a flat binary file with corresponding ASCII header file. His real issue is that he is limited to people he can easily deliver to because they need ENVI or a program that can read ENVI files. Swapping ENVI for ERMapper causes the same problem: end-users need proprietary software to work with the format.
Our hope is that BigTIFF will be adopted as a non propriety standard so that the transport and delivery of very large raster images will be as easy as distributing GeoTIIFFs is currently.
@ Alexandre - We are heavy users of GDAL too. And we use it to read ENVI files (amongst a whole host of others). However, the issue is not are there formats and readers that can read large images, but rather are these universally adopted.
Our experience is that GeoTIFF is this format - read by every package and understood by all levels of geospatial professionals. Its main drawback is its file size limitation.
BigTIFF extends GeoTIFF beyond the 4GB limit. And we hope that the format is adopted by software and users alike. If so, it will make difficult job of transferring and using large file rasters a little bit easier.
Hi there
Where can I get hold of java code to split a geotiff image into several tiles?
@Chetna
We are giving our Providers the ability to serve low resolution KML and sell tiled, high resolution KMZ files. We are not providing the code to tile your own GeoTIFF images. I have seen some blog posts about free tiling code. You may wish to try http://freegeographytools.com/.