Switching from ArcGIS to GDAL
Locked
 
|
Hi All,
I've recently been tasked to cache several large imagery datasets using ArcGIS and have found that my bottleneck has been generating overviews for the datasets prior to caching. ArcGIS appears to be limiting itself to a single core while building overviews, which seems ridiculous on a 12 core machine ... which is why I am here for some help. I am using mosaic datasets in ArcGIS to manage my images. Footprints for the images have been created to clip out any bounding black pixels. Am I able to define overviews in ArcGIS and then use GDAL to build them with multiple cores ... Or at least utilize the footprints I've already defined in my mosaic dataset? Does GDAL have something similar to a mosaic dataset to manage images that I should be using? I'm extremely new to the thought of using GDAL, so any advice on how to go about utilizing it or at least pointing me in the right direction to some tutorials is much appreciated. Thanks Jon. |
Re: Switching from ArcGIS to GDAL
|
|
25.02.2012 22:21, jgamble пишет:
> Hi All, > > I've recently been tasked to cache several large imagery datasets using > ArcGIS and have found that my bottleneck has been generating overviews for > the datasets prior to caching. ArcGIS appears to be limiting itself to a > single core while building overviews, which seems ridiculous on a 12 core > machine ... which is why I am here for some help. > > I am using mosaic datasets in ArcGIS to manage my images. Footprints for > the images have been created to clip out any bounding black pixels. Am I > able to define overviews in ArcGIS and then use GDAL to build them with > multiple cores ... Or at least utilize the footprints I've already defined > in my mosaic dataset? Does GDAL have something similar to a mosaic dataset > to manage images that I should be using? > > I'm extremely new to the thought of using GDAL, so any advice on how to go > about utilizing it or at least pointing me in the right direction to some > tutorials is much appreciated. > > Thanks > Jon. > > -- > View this message in context: http://osgeo-org.1560.n6.nabble.com/Switching-from-ArcGIS-to-GDAL-tp4507244p4507244.html > Sent from the GDAL - Dev mailing list archive at Nabble.com. > _______________________________________________ > gdal-dev mailing list > [hidden email] > http://lists.osgeo.org/mailman/listinfo/gdal-dev > > Best regards, Dmitry _______________________________________________ gdal-dev mailing list [hidden email] http://lists.osgeo.org/mailman/listinfo/gdal-dev |
|
|
Does gdaladdo support mosaic datasets? If so, are there examples of how to use gdaladdo with mosaic datasets?
|
Re: Switching from ArcGIS to GDAL
|
|
25.02.2012 23:02, jgamble пишет:
> Does gdaladdo support mosaic datasets? If so, are there examples of how to > use gdaladdo with mosaic datasets? > > > -- > View this message in context: http://osgeo-org.1560.n6.nabble.com/Switching-from-ArcGIS-to-GDAL-tp4507244p4507323.html > Sent from the GDAL - Dev mailing list archive at Nabble.com. > _______________________________________________ > gdal-dev mailing list > [hidden email] > http://lists.osgeo.org/mailman/listinfo/gdal-dev > > Best regards, Dmitry _______________________________________________ gdal-dev mailing list [hidden email] http://lists.osgeo.org/mailman/listinfo/gdal-dev |
Re: Switching from ArcGIS to GDAL
|
|
In reply to this post by jgamble
On 12-02-25 11:02 AM, jgamble wrote:
> Does gdaladdo support mosaic datasets? If so, are there examples of how to > use gdaladdo with mosaic datasets? Jon, GDAL does not support ArcGIS virtual mosaic format. The gdaladdo utility can be used on conventional raster formats. It is also possible to build a GDAL virtual raster file which might be a mosaic of raster files and then build overviews on it. I believe gdalbuildvrt is normal utility used to build .vrt mosaics. However, whether this is suitable for your purposes will depend on what you want to use the result for. If you just want a freestanding overview image of a set of images then you would be better off just doing something like: gdal_merge.py -o mosaic.tif -ps 20 20 in*.tif Assuming you want a mosaic of all in*.tif files with the output overview mosaic with a 20m by 20m pixel size. Note that .vrt's and their corresponding overviews may be usable in ArcGIS (since it uses GDAL) but I am not certain and it will likely depend on the version. Best regards, -- ---------------------------------------+-------------------------------------- I set the clouds in motion - turn up | Frank Warmerdam, [hidden email] light and sound - activate the windows | http://home.gdal.org/warmerda and watch the world go round - Rush | Geospatial Software Developer _______________________________________________ gdal-dev mailing list [hidden email] http://lists.osgeo.org/mailman/listinfo/gdal-dev |
RE: Switching from ArcGIS to GDAL
|
|
In reply to this post by jgamble
Hi Jon - alas for we ArcGIS users, which even here in 2012 is still a
single-threaded application. Who needs more than one CPU core anyway? So your ArcGIS choice is to split your batch up into 12 groups and run the processes simultaneously (which will then take all your CPUs), or use something else. Might I suggest a pretty simple solution which uses GDAL and MapServer instead? In the state of Maine we now serve all of our enterprise web mapping services (WMS) for imagery using GDAL and MapServer. Using gdalwarp and gdal_translate we convert whatever the source imagery is into JPEG2000 compressed tiles, and generate overview images using gdalwarp. This system takes our 3.5TB of source imagery down to about 700GB of JP2 files and is pretty darn fast. You could set something like this up on a simple PC (but don't use a USB drive for the data - put it on a fast internal drive) for a small group of users inside your organization, or on servers for a wide distribution to a large # of folks. Our system is designed to support about 2000 end-users and about 10,000 requests per hour, using a dual-CPU (quad-core) HP server. We use MS4W which is super-easy to install, and then scripts to generate the tiles. I'd be happy to share some of these with you as examples. ====================================== Michael Smith State GIS Manager, Maine Office of GIS State of Maine, Office of Information Technology michael.smith _at_ maine.gov 207-215-5530 Board Member, Maine GeoLibrary Board Member, Maine GIS Users Group State Rep, National States Geographic Information Council | MEGIS | GeoLibrary | MEGUG | NSGIC | State House Station 174 264 Civic Center Drive Augusta, ME 04333-0174 69 o 47' 49.5"W 44 o 20' 54.5"N Confidentiality Notice: This email message, including any attachments, is for the sole use of the intended recipients and may contain confidential and privileged information. If you are not the intended recipient, or an authorized agent of the intended recipient, please immediately contact the sender by reply email and destroy/delete all copies of the original message. Any unauthorized review, use, copying, forwarding, disclosure, or distribution by other than the intended recipient or authorized agent is prohibited. -----Original Message----- From: [hidden email] [mailto:[hidden email]] On Behalf Of jgamble Sent: Saturday, February 25, 2012 1:21 PM To: [hidden email] Subject: [gdal-dev] Switching from ArcGIS to GDAL Hi All, I've recently been tasked to cache several large imagery datasets using ArcGIS and have found that my bottleneck has been generating overviews for the datasets prior to caching. ArcGIS appears to be limiting itself to a single core while building overviews, which seems ridiculous on a 12 core machine ... which is why I am here for some help. I am using mosaic datasets in ArcGIS to manage my images. Footprints for the images have been created to clip out any bounding black pixels. Am I able to define overviews in ArcGIS and then use GDAL to build them with multiple cores ... Or at least utilize the footprints I've already defined in my mosaic dataset? Does GDAL have something similar to a mosaic dataset to manage images that I should be using? I'm extremely new to the thought of using GDAL, so any advice on how to go about utilizing it or at least pointing me in the right direction to some tutorials is much appreciated. Thanks Jon. -- View this message in context: http://osgeo-org.1560.n6.nabble.com/Switching-from-ArcGIS-to-GDAL-tp4507 244p4507244.html Sent from the GDAL - Dev mailing list archive at Nabble.com. _______________________________________________ gdal-dev mailing list [hidden email] http://lists.osgeo.org/mailman/listinfo/gdal-dev _______________________________________________ gdal-dev mailing list [hidden email] http://lists.osgeo.org/mailman/listinfo/gdal-dev |
|
|
In reply to this post by Frank Warmerdam
Frank,
Thanks for the response. I would like to take the overviews and use them to cache the imagery into a web service ... I am assuming i'd be able to use the overviews to cache. I'll look into gdalbuildvrt command.
Do GDAL mosaics allow you to mask tile boundaries at all? I'd like to remove bounding black pixels along the borders prior to buliding overviews. Thanks Jon. On Sun, Feb 26, 2012 at 11:55 AM, Frank Warmerdam [via OSGeo.org] <[hidden email]> wrote:
|
|
|
In reply to this post by Smith, Michael-2
Hey Michael,
Good tips on the GDAL/MapServer combo. I may have a look into such a setup for comparison purposes. At the current moment though, I am in need of a solution that uses ArcGIS Server as the cache that I create will eventually be incorporated back into existing tiled services hosted on ArcGIS Server. I was hoping that there was a solution that would allow me to incorporate GDAL overviews into an ArcGIS mosaic dataset, though at the moment I am not sure how to do such a thing. I'll keep looking at let you know if I find anything. Thanks for your suggestion concerning MapServer though ... Seems like a lot of folks are headed towards alternatives to ArcGIS Server these days. Cheers, Jon. On Mon, Feb 27, 2012 at 8:56 AM, Smith, Michael-2 [via OSGeo.org] <[hidden email]> wrote:
|
«
Return to GDAL - Dev
|
1 view|%1 views
| Free forum by Nabble | Edit this page |