Raster Catalogs (aka Tileindexes)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Raster Catalogs (aka Tileindexes)

Frank Warmerdam
Folks,

Jason Birch and I are interested in some mechanism to support raster
catalogs (or tileindexes in mapserver-speak) for the GDAL raster provider.
That is a way of listing a whole bunch of files, along with their spatial
footprint in a single catalog so that the raster provider only has to actually
open the files of interest for the current request.

Does such a thing exist for the other (ATIL based) raster provider?

Is there a standard scheme suggested for implementing such a thing?

My thinking was that it should be possible to use any FDO feature source
as the catalog and that the catalog would primarily devolve down to two
things.

   - A column with the filename for the raster (possibly a relative path)
   - A polygon geometry with is the footprint of the raster.

The already existing "gdaltindex" utility can create shapefiles in this
form for a set of rasters.  I'd suggest just hardcoding things to use
shapefiles, but I suspect some people will want to use more sophisticated
raster catalogs in an RDBMS and it just doesn't seem like the right
FDO way of doing things.

As long as we come up with a scheme that isn't too complicated, I'm happy
to implement it for the GDAL based raster provider.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, [hidden email]
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
Reply | Threaded
Open this post in threaded view
|

RE: Raster Catalogs (aka Tileindexes)

Traian Stanev
[fdo-internals] Raster Catalogs (aka Tileindexes)
Hi Frank,
 
Here my thoughts on this. A raster catalog does not need to be exposed at the API level. For example, one way to do this would be to "connect" the raster provider to a directory containing a large number of rasters. Internally, the provider could index these in whatever way it wants.
 
I would personally use an SDF file for the spatial index -- like you say, insert a polygon corresponding to the bounds of the raster and use the raster name (or index) as ID property. An SHP file will probably work fine too, I've seen lots of raster catalogs stored in shp files before.
 
One thing I always wondered is whether it's worth it to persist the index into a file or is it easier to initialize it the first time an FDO connection is opened and then keep it in memory until the connection is closed. Given that FDO connections remain open for a while, it seems like keeping it in memory is not such a bad idea, and avoids the trouble of dealing with an index file.
 
Traian
-----Original Message-----
From: [hidden email] on behalf of Frank Warmerdam (External)
Sent: Wed 1/24/2007 10:42 PM
To: FDO Internals Mail List
Cc: Jason Birch
Subject: [fdo-internals] Raster Catalogs (aka Tileindexes)

Folks,

Jason Birch and I are interested in some mechanism to support raster
catalogs (or tileindexes in mapserver-speak) for the GDAL raster provider.
That is a way of listing a whole bunch of files, along with their spatial
footprint in a single catalog so that the raster provider only has to actually
open the files of interest for the current request.

Does such a thing exist for the other (ATIL based) raster provider?

Is there a standard scheme suggested for implementing such a thing?

My thinking was that it should be possible to use any FDO feature source
as the catalog and that the catalog would primarily devolve down to two
things.

   - A column with the filename for the raster (possibly a relative path)
   - A polygon geometry with is the footprint of the raster.

The already existing "gdaltindex" utility can create shapefiles in this
form for a set of rasters.  I'd suggest just hardcoding things to use
shapefiles, but I suspect some people will want to use more sophisticated
raster catalogs in an RDBMS and it just doesn't seem like the right
FDO way of doing things.

As long as we come up with a scheme that isn't too complicated, I'm happy
to implement it for the GDAL based raster provider.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, [hidden email]
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals


_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
Reply | Threaded
Open this post in threaded view
|

Re: Raster Catalogs (aka Tileindexes)

Frank Warmerdam
Traian Stanev wrote:
> Hi Frank,
>  
> Here my thoughts on this. A raster catalog does not need to be exposed
> at the API level. For example, one way to do this would be to "connect"
> the raster provider to a directory containing a large number of rasters.
> Internally, the provider could index these in whatever way it wants.

Traian,

Sometimes the rasters may be spread in a variety of locations, so one
advantage of a catalog is allowing this as opposed to the current
requirement (I guess unless a config.xml file is used) to have them all
in one directory.

But yes, I can see that "auto-indexing" could be built quite easily, and
would not complicate the user experience at all.

BTW, I hadn't intended to expose this all as new API calls - but rather
as some sort of extra config variables or something like that for the
provider.

> I would personally use an SDF file for the spatial index -- like you
> say, insert a polygon corresponding to the bounds of the raster and use
> the raster name (or index) as ID property. An SHP file will probably
> work fine too, I've seen lots of raster catalogs stored in shp files before.

Well, I suggest shapefiles because I have lots of experience using
them as catalogs, and because gdaltindex can already create them.
The problem with SDF at this point is I don't know how to do anything
with them short of creating a custom FDO application.  Clearly it would
be a step forward for me to build an OGR-FDO driver, so I can use my
normal OGR tools to read, create and manipulate FDO feature stores
like SDF.

I've also had this idea that SDF support needs to be extracted from
FDO and made into an easy to use standalone library.  But I haven't
had time to even contemplate this which would be a substantial project.
And, as has been pointed out to me, forking a copy of the SDF code
from FDO would make it harder for the SDF format to evolve.

> One thing I always wondered is whether it's worth it to persist the
> index into a file or is it easier to initialize it the first time an FDO
> connection is opened and then keep it in memory until the connection is
> closed. Given that FDO connections remain open for a while, it seems
> like keeping it in memory is not such a bad idea, and avoids the trouble
> of dealing with an index file.

This of course is something I have trouble wrapping my mind around
coming from the very stateless (or perhaps better to say all-state-in-files)
world of mapserver.  You are correct that indexing in memory would go a
long ways towards resolving performance issues with large numbers of files
in a directory.

I guess someone would need to restart the mapguide service if they wanted
to ensure things were re-indexed?

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, [hidden email]
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
Reply | Threaded
Open this post in threaded view
|

RE: Raster Catalogs (aka Tileindexes)

Greg Boone
In reply to this post by Frank Warmerdam
Hi All,

The FDO team faces a similar issue with the ATIL based raster provider
found in the MapGuide Enterprise release. At this time we are planning
to fix this issue for the next release of MapGuide Enterprise. I would
hope that we could find a solution that is common to both the ATIL based
and GDAL based providers.

The current proposal advocates that the ATIL Raster provider's XML
configuration document be updated to include the bounds of the
referenced raster images. Along with this change would come an
associated change to the API that controls the creation and access of
the XML configuration document. These additional 'bounds' definitions
would be used by the provider to filter the list of initial raster
images based on a spatial filter supplied by the user in an FDO select
command. Once the selection set of raster images has been determined,
the raster provider will directly access the images and extract the
geo-reference information to determine which images to return to the
client application.

Since the ATIL and GDAL overrides API and configuration documents are
virtually identical, a common approach could be agreed on for both
providers.

Please find the proposal attached. Comments and feedback are welcome.

Greg

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Frank
Warmerdam (External)
Sent: Wednesday, January 24, 2007 10:43 PM
To: FDO Internals Mail List
Cc: Jason Birch
Subject: [fdo-internals] Raster Catalogs (aka Tileindexes)

Folks,

Jason Birch and I are interested in some mechanism to support raster
catalogs (or tileindexes in mapserver-speak) for the GDAL raster
provider.
That is a way of listing a whole bunch of files, along with their
spatial
footprint in a single catalog so that the raster provider only has to
actually
open the files of interest for the current request.

Does such a thing exist for the other (ATIL based) raster provider?

Is there a standard scheme suggested for implementing such a thing?

My thinking was that it should be possible to use any FDO feature source
as the catalog and that the catalog would primarily devolve down to two
things.

   - A column with the filename for the raster (possibly a relative
path)
   - A polygon geometry with is the footprint of the raster.

The already existing "gdaltindex" utility can create shapefiles in this
form for a set of rasters.  I'd suggest just hardcoding things to use
shapefiles, but I suspect some people will want to use more
sophisticated
raster catalogs in an RDBMS and it just doesn't seem like the right
FDO way of doing things.

As long as we come up with a scheme that isn't too complicated, I'm
happy
to implement it for the GDAL based raster provider.

Best regards,
--
---------------------------------------+--------------------------------
------
I set the clouds in motion - turn up   | Frank Warmerdam,
[hidden email]
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo,
http://osgeo.org

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals


RasterBoundsProposal.doc (57K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Raster Catalogs (aka Tileindexes)

Frank Warmerdam
Greg Boone wrote:

> Hi All,
>
> The FDO team faces a similar issue with the ATIL based raster provider
> found in the MapGuide Enterprise release. At this time we are planning
> to fix this issue for the next release of MapGuide Enterprise. I would
> hope that we could find a solution that is common to both the ATIL based
> and GDAL based providers.
>
> The current proposal advocates that the ATIL Raster provider's XML
> configuration document be updated to include the bounds of the
> referenced raster images. Along with this change would come an
> associated change to the API that controls the creation and access of
> the XML configuration document. These additional 'bounds' definitions
> would be used by the provider to filter the list of initial raster
> images based on a spatial filter supplied by the user in an FDO select
> command. Once the selection set of raster images has been determined,
> the raster provider will directly access the images and extract the
> geo-reference information to determine which images to return to the
> client application.
>
> Since the ATIL and GDAL overrides API and configuration documents are
> virtually identical, a common approach could be agreed on for both
> providers.

Greg,

The proposal looks good to me.  The only downsides I can see are
that linear scanning of the XML config doc may not scale up to hundreds
of thousands of tiles as well as might be hoped, and that folks not
using MapGuide Studio may not have a convenient way of producing the
config files.  However, I don't see either of these as a significant
issue.

Will you or someone at Autodesk be migrating the changes into the
GDAL provider once the change is made for the ATIL based provider?
I'm willing to do some work on this, but I don't honestly grok all this
override stuff and I'd hesitate to do it from scratch.

I would like to help the webstudio guys add raster support including
building the config doc.  It should be pretty easy.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, [hidden email]
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
Reply | Threaded
Open this post in threaded view
|

RE: Raster Catalogs (aka Tileindexes)

Jason Birch
In reply to this post by Frank Warmerdam
CONTENTS DELETED
The author has deleted this message.
Reply | Threaded
Open this post in threaded view
|

Re: Raster Catalogs (aka Tileindexes)

Frank Warmerdam
Jason Birch wrote:

> ------------
> I would like to help the webstudio guys add raster support including
> building the config doc.  It should be pretty easy.
> ------------
>
> My concern with this would be those of us who are using MapGuide Studio
> with MapGuide Open Source.  If you are going to do this, could you
> please consider making it modular enough that I could easily adapt it to
> a standalone page that generates the config file, creates a new
> unmanaged raster data source, and uploads the config file into the
> resource content?

Jason,

I was assuming that the Studio developers would be doing something for
generating this config file in studio.  Actually, I have no idea how people
normally create these config files.

Having said earlier that creating the config file should be easy, I must
confess that on sober second thought I'm not exactly sure how I'm going to
approach it.  The GDAL PHP bindings and I'm not sure we want Web Studio
having a dependency on GDAL PHP bindings anyways.

In theory I suppose Web Studio can use FDO itself to get the bounds for
individual files and then build the config file based on that.  I presume
that is what Studio does?

So, I find myself backing off a bit and thinking perhaps I would offer
an output switch for my existing gdaltindex program that would provide
XML config files instead of shapefiles.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, [hidden email]
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
Reply | Threaded
Open this post in threaded view
|

RE: Raster Catalogs (aka Tileindexes)

Greg Boone
In reply to this post by Frank Warmerdam
There are a couple of options for generating the bounds information in
the configuration file:

1) The FDO team can provide a small application or tool to create the
configuration file based on a set of input directories. Studio or
Studio developers could then use this tool to create the XML
configuration file.
2) Studio could use the FDO DescribeSchemaMapping command to generate
the schema mappings for a default raster connection made to a single
directory of images. However, as stated, this mechanism is limited in
that it only generates mappings for a single directory.

Greg

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Frank
Warmerdam (External)
Sent: Thursday, January 25, 2007 12:55 PM
To: FDO Internals Mail List
Subject: Re: [fdo-internals] Raster Catalogs (aka Tileindexes)

Jason Birch wrote:
> ------------
> I would like to help the webstudio guys add raster support including
> building the config doc.  It should be pretty easy.
> ------------
>
> My concern with this would be those of us who are using MapGuide
Studio
> with MapGuide Open Source.  If you are going to do this, could you
> please consider making it modular enough that I could easily adapt it
to
> a standalone page that generates the config file, creates a new
> unmanaged raster data source, and uploads the config file into the
> resource content?

Jason,

I was assuming that the Studio developers would be doing something for
generating this config file in studio.  Actually, I have no idea how
people
normally create these config files.

Having said earlier that creating the config file should be easy, I must
confess that on sober second thought I'm not exactly sure how I'm going
to
approach it.  The GDAL PHP bindings and I'm not sure we want Web Studio
having a dependency on GDAL PHP bindings anyways.

In theory I suppose Web Studio can use FDO itself to get the bounds for
individual files and then build the config file based on that.  I
presume
that is what Studio does?

So, I find myself backing off a bit and thinking perhaps I would offer
an output switch for my existing gdaltindex program that would provide
XML config files instead of shapefiles.

Best regards,
--
---------------------------------------+--------------------------------
------
I set the clouds in motion - turn up   | Frank Warmerdam,
[hidden email]
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo,
http://osgeo.org

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals