[Geoserver-devel] Ingestion Engine proposal

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

[Geoserver-devel] Ingestion Engine proposal

Alessio Fabiani
Hi all,
I will explain in this email our proposal for a GeoServer Ingestion Engine.

The Ingestion Engine we would like to implement for GeoServer should
be configured as a PlugIn that an Administrator can plug into
GeoServer and use as an alternative to the web interface to manage the
configuration files, i.e. the "catalog.xml" which is where NameSpaces,
DataStores and CoverageFormats parameters are stored and the different
"info.xml" associated to each GeoServer features and coverages which
is where all the information relative to the FeatureType or
GridCoverage are stored.

In order to achieve this objective, we do not want to modify the
actual GeoServer configuration concept, at this moment every time an
Administrator wants to add a new FeatureType or Coverage to GeoServer
he has to follow several steps:

Step 1: Defining the parameters and the ID for a new DataStore or
Format. In the new release of GeoServer-WCS experiment we have renamed
DataStore as FeatureStore and Format as CoverageStore because they are
theoretically the same thing respectively for Vectorial and Gridded
data. GeoServer stores all those informations in the catalog.xml.

Step 2: Creating a new FeatureType or GridCoverage starting from the
Store created in the Step 1. GeoServer creates a new directory with
the same name of the Store ID and Feature/Coverage name and stores
inside an info.xml file containing all the metadata associated to the

Notice that GeoServer makes the configuration files persistent only
after the Administrator does a Save action by clicking over the button

The Ingestion Engine we have in mind should be able to perform Step 1,
Step 2 and Save configuration automatically.

We have two main objectives to achieve:

1. Building something that is pluggable and unplaggable to GeoServer
2. Building something that allows GeoServer to automatically modify
the configuration performing the above steps without removing the
actual GeoServer configuration management system

To achieve the first objective we think about building a Servlet with
his own classes that the administrator can add/remove, configure and
enable/disable by GeoServer web.xml. This Servlet will work on a
temporal based schedule by simply checking the file system structure
for changes.
To achieve the second objective the Servlet simply will automatize the
Administrator steps for each change.

How the servlet works:
First of all we do not want to force users to maintain a predefined
file system structure. We think about a system that mainly leaves
unaltered the file system manually created by the Administrator using
the web interface but creates and maintains it's own structure for the
subdirectories automatically managed, compatible with the first one.

Suppose that the Administrator wants to create a new subdirectory
automatically managed by the GeoServer Ingestion Engine for a set of
files belonging to a particular Store.
What he has to do is creating this subdirectory and placing inside it
a particular xml file which describes the Store type the common
parameters and the metadata that the Ingestion Engine will use. An
external tool, that we want to create too, can be used to create this
configuration file. The Ingestion Servlet will scan the directory and
every time it will encounter a new compatible file it will create a
new subdirectory where this file (and all related) will be moved and
the relative info.xml will be created. For FeatureStores like postgis
those files can be just xml files containing the parameters named like
the final FeatureType. If the Administrator deletes one or more of
those subdirectories, the Ingestion Servlet will remove the
Features/Coverages (and the associated Store) from the GeoServer
configuration. Notice that the Administrator can even manually remove
those features/coverages by using the web interface.

Attached there are two images that show how the Ingestion Engine
should work on an "auto-managed" subdirectory.

Moreover notice that by adding few more metadata informations we can
even handle WMS nested layers. We don't need to reflect the exact File
System tree structure, we can even build a virtual WMS layer tree
structure by handling some metadata. I will explain in detail in the
next email.

engine_dir_one.gif (16K) Download Attachment
engine_dir_two.gif (17K) Download Attachment