Switching from ArcGIS to GDAL

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Switching from ArcGIS to GDAL

jgamble
Hi All,

I've recently been tasked to cache several large imagery datasets using ArcGIS and have found that my bottleneck has been generating overviews for the datasets prior to caching.  ArcGIS appears to be limiting itself to a single core while building overviews, which seems ridiculous on a 12 core machine ... which is why I am here for some help.

I am using mosaic datasets in ArcGIS to manage my images.  Footprints for the images have been created to clip out any bounding black pixels.  Am I able to define overviews in ArcGIS and then use GDAL to build them with multiple cores ... Or at least utilize the footprints I've already defined in my mosaic dataset?  Does GDAL have something similar to a mosaic dataset to manage images that I should be using?

I'm extremely new to the thought of using GDAL, so any advice on how to go about utilizing it or at least pointing me in the right direction to some tutorials is much appreciated.

Thanks
Jon.      
Reply | Threaded
Open this post in threaded view
|

Re: Switching from ArcGIS to GDAL

Dmitry Baryshnikov
25.02.2012 22:21, jgamble пишет:

> Hi All,
>
> I've recently been tasked to cache several large imagery datasets using
> ArcGIS and have found that my bottleneck has been generating overviews for
> the datasets prior to caching.  ArcGIS appears to be limiting itself to a
> single core while building overviews, which seems ridiculous on a 12 core
> machine ... which is why I am here for some help.
>
> I am using mosaic datasets in ArcGIS to manage my images.  Footprints for
> the images have been created to clip out any bounding black pixels.  Am I
> able to define overviews in ArcGIS and then use GDAL to build them with
> multiple cores ... Or at least utilize the footprints I've already defined
> in my mosaic dataset?  Does GDAL have something similar to a mosaic dataset
> to manage images that I should be using?
>
> I'm extremely new to the thought of using GDAL, so any advice on how to go
> about utilizing it or at least pointing me in the right direction to some
> tutorials is much appreciated.
>
> Thanks
> Jon.
>
> --
> View this message in context: http://osgeo-org.1560.n6.nabble.com/Switching-from-ArcGIS-to-GDAL-tp4507244p4507244.html
> Sent from the GDAL - Dev mailing list archive at Nabble.com.
> _______________________________________________
> gdal-dev mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
Do you try http://gdal.org/gdaladdo.html ?

Best regards,
Dmitry
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Switching from ArcGIS to GDAL

jgamble
Does gdaladdo support mosaic datasets?  If so, are there examples of how to use gdaladdo with mosaic datasets?
Reply | Threaded
Open this post in threaded view
|

Re: Switching from ArcGIS to GDAL

Dmitry Baryshnikov
25.02.2012 23:02, jgamble пишет:

> Does gdaladdo support mosaic datasets?  If so, are there examples of how to
> use gdaladdo with mosaic datasets?
>
>
> --
> View this message in context: http://osgeo-org.1560.n6.nabble.com/Switching-from-ArcGIS-to-GDAL-tp4507244p4507323.html
> Sent from the GDAL - Dev mailing list archive at Nabble.com.
> _______________________________________________
> gdal-dev mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
Do you mean new in ArcGIS 10 raster mosaic in geodatabase?

Best regards,
Dmitry
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Switching from ArcGIS to GDAL

Frank Warmerdam
In reply to this post by jgamble
On 12-02-25 11:02 AM, jgamble wrote:
> Does gdaladdo support mosaic datasets?  If so, are there examples of how to
> use gdaladdo with mosaic datasets?

Jon,

GDAL does not support ArcGIS virtual mosaic format.

The gdaladdo utility can be used on conventional raster formats.  It is
also possible to build a GDAL virtual raster file which might be a mosaic
of raster files and then build overviews on it.  I believe gdalbuildvrt
is normal utility used to build .vrt mosaics.

However, whether this is suitable for your purposes will depend on
what you want to use the result for.  If you just want a freestanding
overview image of a set of images then you would be better off just
doing something like:

   gdal_merge.py -o mosaic.tif -ps 20 20 in*.tif

Assuming you want a mosaic of all in*.tif files with the output overview
mosaic with a 20m by 20m pixel size.

Note that .vrt's and their corresponding overviews may be usable in
ArcGIS (since it uses GDAL) but I am not certain and it will likely
depend on the version.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, [hidden email]
light and sound - activate the windows | http://home.gdal.org/warmerda
and watch the world go round - Rush    | Geospatial Software Developer

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

RE: Switching from ArcGIS to GDAL

Smith, Michael-2
In reply to this post by jgamble
Hi Jon - alas for we ArcGIS users, which even here in 2012 is still a
single-threaded application.  Who needs more than one CPU core anyway?
So your ArcGIS choice is to split your batch up into 12 groups and run
the processes simultaneously (which will then take all your CPUs), or
use something else.

Might I suggest a pretty simple solution which uses GDAL and MapServer
instead?  In the state of Maine we now serve all of our enterprise web
mapping services (WMS) for imagery using GDAL and MapServer.  Using
gdalwarp and gdal_translate we convert whatever the source imagery is
into JPEG2000 compressed tiles, and generate overview images using
gdalwarp.  This system takes our 3.5TB of source imagery down to about
700GB of JP2 files and is pretty darn fast.

You could set something like this up on a simple PC (but don't use a USB
drive for the data - put it on a fast internal drive) for a small group
of users inside your organization, or on servers for a wide distribution
to a large # of folks.

Our system is designed to support about 2000 end-users and about 10,000
requests per hour, using a dual-CPU (quad-core) HP server.  We use MS4W
which is super-easy to install, and then scripts to generate the tiles.
I'd be happy to share some of these with you as examples.


======================================
Michael Smith
State GIS Manager, Maine Office of GIS
State of Maine, Office of Information Technology
michael.smith _at_ maine.gov
207-215-5530

Board Member, Maine GeoLibrary
Board Member, Maine GIS Users Group
State Rep, National States Geographic Information Council

| MEGIS | GeoLibrary | MEGUG | NSGIC |

State House Station 174
264 Civic Center Drive
Augusta, ME 04333-0174

69 o 47' 49.5"W   44 o 20' 54.5"N

Confidentiality Notice: This email message, including any attachments,
is for the sole use of the intended recipients and may contain
confidential and privileged information. If you are not the intended
recipient, or an authorized agent of the intended recipient, please
immediately contact the sender by reply email and destroy/delete all
copies of the original message. Any unauthorized review, use, copying,
forwarding, disclosure, or distribution by other than the intended
recipient or authorized agent is prohibited.

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of jgamble
Sent: Saturday, February 25, 2012 1:21 PM
To: [hidden email]
Subject: [gdal-dev] Switching from ArcGIS to GDAL

Hi All,

I've recently been tasked to cache several large imagery datasets using
ArcGIS and have found that my bottleneck has been generating overviews
for
the datasets prior to caching.  ArcGIS appears to be limiting itself to
a
single core while building overviews, which seems ridiculous on a 12
core
machine ... which is why I am here for some help.

I am using mosaic datasets in ArcGIS to manage my images.  Footprints
for
the images have been created to clip out any bounding black pixels.  Am
I
able to define overviews in ArcGIS and then use GDAL to build them with
multiple cores ... Or at least utilize the footprints I've already
defined
in my mosaic dataset?  Does GDAL have something similar to a mosaic
dataset
to manage images that I should be using?

I'm extremely new to the thought of using GDAL, so any advice on how to
go
about utilizing it or at least pointing me in the right direction to
some
tutorials is much appreciated.

Thanks
Jon.      

--
View this message in context:
http://osgeo-org.1560.n6.nabble.com/Switching-from-ArcGIS-to-GDAL-tp4507
244p4507244.html
Sent from the GDAL - Dev mailing list archive at Nabble.com.
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Switching from ArcGIS to GDAL

jgamble
In reply to this post by Frank Warmerdam
Frank,

Thanks for the response.  I would like to take the overviews and use them to cache the imagery into a web service ... I am assuming i'd be able to use the overviews to cache.  I'll look into gdalbuildvrt command. 

Do GDAL mosaics allow you to mask tile boundaries at all?  I'd like to remove bounding black pixels along the borders prior to buliding overviews.


Thanks
Jon.

On Sun, Feb 26, 2012 at 11:55 AM, Frank Warmerdam [via OSGeo.org] <[hidden email]> wrote:
On 12-02-25 11:02 AM, jgamble wrote:
> Does gdaladdo support mosaic datasets?  If so, are there examples of how to
> use gdaladdo with mosaic datasets?

Jon,

GDAL does not support ArcGIS virtual mosaic format.

The gdaladdo utility can be used on conventional raster formats.  It is
also possible to build a GDAL virtual raster file which might be a mosaic
of raster files and then build overviews on it.  I believe gdalbuildvrt
is normal utility used to build .vrt mosaics.

However, whether this is suitable for your purposes will depend on
what you want to use the result for.  If you just want a freestanding
overview image of a set of images then you would be better off just
doing something like:

   gdal_merge.py -o mosaic.tif -ps 20 20 in*.tif

Assuming you want a mosaic of all in*.tif files with the output overview
mosaic with a 20m by 20m pixel size.

Note that .vrt's and their corresponding overviews may be usable in
ArcGIS (since it uses GDAL) but I am not certain and it will likely
depend on the version.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, [hidden email]
light and sound - activate the windows | http://home.gdal.org/warmerda
and watch the world go round - Rush    | Geospatial Software Developer

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev



If you reply to this email, your message will be added to the discussion below:
http://osgeo-org.1560.n6.nabble.com/Switching-from-ArcGIS-to-GDAL-tp4507244p4510600.html
To unsubscribe from Switching from ArcGIS to GDAL, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: Switching from ArcGIS to GDAL

jgamble
In reply to this post by Smith, Michael-2
Hey Michael,

Good tips on the GDAL/MapServer combo.  I may have a look into such a setup for comparison purposes.

At the current moment though, I am in need of a solution that uses ArcGIS Server as the cache that I create will eventually be incorporated back into existing tiled services hosted on ArcGIS Server.  I was hoping that there was a solution that would allow me to incorporate GDAL overviews into an ArcGIS mosaic dataset, though at the moment I am not sure how to do such a thing. 

I'll keep looking at let you know if I find anything.

Thanks for your suggestion concerning MapServer though ... Seems like a lot of folks are headed towards alternatives to ArcGIS Server these days.

Cheers,
Jon.

On Mon, Feb 27, 2012 at 8:56 AM, Smith, Michael-2 [via OSGeo.org] <[hidden email]> wrote:
Hi Jon - alas for we ArcGIS users, which even here in 2012 is still a
single-threaded application.  Who needs more than one CPU core anyway?
So your ArcGIS choice is to split your batch up into 12 groups and run
the processes simultaneously (which will then take all your CPUs), or
use something else.

Might I suggest a pretty simple solution which uses GDAL and MapServer
instead?  In the state of Maine we now serve all of our enterprise web
mapping services (WMS) for imagery using GDAL and MapServer.  Using
gdalwarp and gdal_translate we convert whatever the source imagery is
into JPEG2000 compressed tiles, and generate overview images using
gdalwarp.  This system takes our 3.5TB of source imagery down to about
700GB of JP2 files and is pretty darn fast.

You could set something like this up on a simple PC (but don't use a USB
drive for the data - put it on a fast internal drive) for a small group
of users inside your organization, or on servers for a wide distribution
to a large # of folks.

Our system is designed to support about 2000 end-users and about 10,000
requests per hour, using a dual-CPU (quad-core) HP server.  We use MS4W
which is super-easy to install, and then scripts to generate the tiles.
I'd be happy to share some of these with you as examples.


======================================
Michael Smith
State GIS Manager, Maine Office of GIS
State of Maine, Office of Information Technology
michael.smith _at_ maine.gov
<a href="tel:207-215-5530" value="+12072155530" target="_blank">207-215-5530

Board Member, Maine GeoLibrary
Board Member, Maine GIS Users Group
State Rep, National States Geographic Information Council

| MEGIS | GeoLibrary | MEGUG | NSGIC |

State House Station 174
264 Civic Center Drive
Augusta, ME 04333-0174

69 o 47' 49.5"W   44 o 20' 54.5"N

Confidentiality Notice: This email message, including any attachments,
is for the sole use of the intended recipients and may contain
confidential and privileged information. If you are not the intended
recipient, or an authorized agent of the intended recipient, please
immediately contact the sender by reply email and destroy/delete all
copies of the original message. Any unauthorized review, use, copying,
forwarding, disclosure, or distribution by other than the intended
recipient or authorized agent is prohibited.

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of jgamble
Sent: Saturday, February 25, 2012 1:21 PM
To: [hidden email]
Subject: [gdal-dev] Switching from ArcGIS to GDAL

Hi All,

I've recently been tasked to cache several large imagery datasets using
ArcGIS and have found that my bottleneck has been generating overviews
for
the datasets prior to caching.  ArcGIS appears to be limiting itself to
a
single core while building overviews, which seems ridiculous on a 12
core
machine ... which is why I am here for some help.

I am using mosaic datasets in ArcGIS to manage my images.  Footprints
for
the images have been created to clip out any bounding black pixels.  Am
I
able to define overviews in ArcGIS and then use GDAL to build them with
multiple cores ... Or at least utilize the footprints I've already
defined
in my mosaic dataset?  Does GDAL have something similar to a mosaic
dataset
to manage images that I should be using?

I'm extremely new to the thought of using GDAL, so any advice on how to
go
about utilizing it or at least pointing me in the right direction to
some
tutorials is much appreciated.

Thanks
Jon.      

--
View this message in context:
http://osgeo-org.1560.n6.nabble.com/Switching-from-ArcGIS-to-GDAL-tp4507
244p4507244.html
Sent from the GDAL - Dev mailing list archive at Nabble.com.
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________
gdal-dev mailing list
If you reply to this email, your message will be added to the discussion below:
http://osgeo-org.1560.n6.nabble.com/Switching-from-ArcGIS-to-GDAL-tp4507244p4514858.html
To unsubscribe from Switching from ArcGIS to GDAL, click here.
NAML