R: [Geoserver-devel] Ingestion Engine proposal

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

R: [Geoserver-devel] Ingestion Engine proposal

P.Rizzi Ag.Mobilità Ambiente
Hi Alessio,
we here have, like many others I think, the same need to automatically
configure GeoServer, without passing through its Web Admin interface.
We are, until now, only working with DataStores, but I believe there
should be no much differences for the Coverages part.

I read what you are thinking to do to solve this, and I should say
I personally wouldn't go in that direction, but you have for sure
your needs and for sure your best bet is to follow them.

Anyway here follow my vision of this matter.

---------------------------------------------------------------------------
I think the configuration part in GeoServer is not its best one,
because it intermixes three concerns that should instead be separated:
   - the config data itself
   - the config data persistence
   - the servlet

The configuration data, for me, is the MEMORY IMAGE a given running instance
of GeoServer has in any given instant in time, and that has nothing to do
with how that information was generated, beeing it from an XML file, from a
DB, etc.
So each configuration mechanism will persist its own data in the form it
likes more, there's no need to save and load it from a single source.

What I'm saying is that the configuration data must be separated from it's
persistence.
Now GeoServer uses several files and directories to store its configuration
and it's
only able to load the configuration from that files.
Instead the catalog.xml and all others should be only one of the many
different
persistence forms for GeoServer's configuration data. That is, the
catalog.xml
file and the pieces of Java code able to manage it, should be no different
from the ingestion engine or from whatever other mechanism one can invent.

The memory image of the configuration data is kept by GeoServer inside
the ServletContext, so that it's globally accessible from any servlet
running inside a given GeoServer instance and from any HTTP request/response
beeing serviced by that servlet. So you can modify that memory image
on-the-fly
using a servlet filter, without any need to store it using the same
persistence mechanism that GeoServer uses now.

Anyway, if one would prefer to persist all the config info inside the
same store, I won't use files and directories for that, I'd use DataStores.
GeoServer uses GeoTools to access data, and GeoTools uses DataStores
to that purpose. Configuration data are no different from other data,
so I believe the right place to store it is inside DataStores.
There's a GMLDataStore (although it's read-only at the moment), so one
can use it if he likes to store config info inside XML files.

I think that the main concern is not how one persist the config info,
but it's the config info itself, that is the Catalog.
Now GeoServer uses FeatureTypeInfo and related classes to construct
its memory image of configuration. Are that classes good enough
to support coverages too??? Are they flexible/extensible enough
to support future services??? Are they "compatible" with classes
used by other systems (like uDig for example)???
I feel this is what we should think about, the mean by which these
catalog classes are persisted should be irrilevant.

One distinction must be made here. I think that there're two different
levels of config info. One is about DataStores, that is how to connect
to each source of data. The other is about FeatureTypes, that is which
data is available, how it is structured, how it must be validated,
who has the rights to read or modify it, etc.
DataStore config info can be in simple XML files or whatever, because
you only have to say how to connect to that source of data, so the
connection params are basically all that's needed.
FeatureType config info are a different story, or better they're no
config info at all, they are metadata and they belong to the Catalog.
In our vision, and in what we have implemented so far, to add new data
to GeoServer, you simply have to add an XML file with the connection params
of the DataStore. The metadata for each FeatureType contained in that
DataStore
is read from the DataStore itself. That is, for each DataStore you want to
add
to GeoServer you MUST also add what we call a MetaStore, that is a
persistent
form of the metadata for the FeatureTypes contained in the DataStore itself.
Together with the MetaStore you MUST specify a Loader able to load that
persisted metadata in a memory image, that is a Catalog. We have a few
Loader
implemented, one is able to load metadata from FeatureSources (aka DB
tables)
with a specific structure, another one is able to infer a minimum set
of metadata from the DataStore itself, so that you can also add a DataStore
for which you actually don't have a proper MetaStore.
So we have this catalog in memory and we can use it to configure GeoServer
on-the-fly using a servlet filter (actually it is a Tomcat Valve) that
directly
"talks" to the GeoServer's catalog building DTOs on the fly.

But there's more to it...

We're also more and more convinced that some of the things GeoServer is able
to do now should be moved to GeoTools. Validation and Transactionality
should be in GeoTools, probably even the GetFeature operation should be in
GeoTools.
The central point of it all is basically about be able to operate
(querying, reading, writing, validating, etc.) against a set of DataStores
instead that against each one separately, and this capability should be in
GeoTools,
not in GeoServer.

If things were like that, many of the config info now used by GeoServer,
the ones regarding FeatureTypes metadata, would go into the GeoTools catalog
configuration.

GeoServer will then have only the config info relevant to each OGC service
it exposes (WFS, WMS, WCS, etc.) and it only needs to have references to the
metadata configured inside the GeoTools catalog.

So each service plugged into GeoServer will have it's own configuration
mechanism,
for its own configuration info. GeoServer will only need a system to
configure the plugins,
data will instead be configured inside the GeoTools catalog.
---------------------------------------------------------------------------

...that was a very long dissertation, I'm sorry...
And implementing it is much more impacting then what you're proposing,
so it may take a while to do (even if we already implemented a certain part
of it).
Also in this very moment we have other more urgent aspects to see after,
so I'm afraid I won't be able to heavily work on this for a while.
I'm very sorry about this, because I'd like to see others using what
we've done so far, I'd have to find time to make it general enough
and to publish it...

Bye
Paolo Rizzi



> -----Messaggio originale-----
> Da: Alessio Fabiani [mailto:[hidden email]]
> Inviato: martedì 19 luglio 2005 11.30
> A: [hidden email];
> [hidden email]
> Oggetto: [Geoserver-devel] Ingestion Engine proposal
>
>
> Hi all,
> I will explain in this email our proposal for a GeoServer
> Ingestion Engine.
>
> The Ingestion Engine we would like to implement for GeoServer should
> be configured as a PlugIn that an Administrator can plug into
> GeoServer and use as an alternative to the web interface to manage the
> configuration files, i.e. the "catalog.xml" which is where NameSpaces,
> DataStores and CoverageFormats parameters are stored and the different
> "info.xml" associated to each GeoServer features and coverages which
> is where all the information relative to the FeatureType or
> GridCoverage are stored.
>
> In order to achieve this objective, we do not want to modify the
> actual GeoServer configuration concept, at this moment every time an
> Administrator wants to add a new FeatureType or Coverage to GeoServer
> he has to follow several steps:
>
> Step 1: Defining the parameters and the ID for a new DataStore or
> Format. In the new release of GeoServer-WCS experiment we have renamed
> DataStore as FeatureStore and Format as CoverageStore because they are
> theoretically the same thing respectively for Vectorial and Gridded
> data. GeoServer stores all those informations in the catalog.xml.
>
> Step 2: Creating a new FeatureType or GridCoverage starting from the
> Store created in the Step 1. GeoServer creates a new directory with
> the same name of the Store ID and Feature/Coverage name and stores
> inside an info.xml file containing all the metadata associated to the
> latter.
>
> Notice that GeoServer makes the configuration files persistent only
> after the Administrator does a Save action by clicking over the button
> associated.
>
> The Ingestion Engine we have in mind should be able to perform Step 1,
> Step 2 and Save configuration automatically.
>
> We have two main objectives to achieve:
>
> 1. Building something that is pluggable and unplaggable to
> GeoServer
> 2. Building something that allows GeoServer to automatically modify
> the configuration performing the above steps without removing the
> actual GeoServer configuration management system
>
> To achieve the first objective we think about building a Servlet with
> his own classes that the administrator can add/remove, configure and
> enable/disable by GeoServer web.xml. This Servlet will work on a
> temporal based schedule by simply checking the file system structure
> for changes.
> To achieve the second objective the Servlet simply will automatize the
> Administrator steps for each change.
>
> How the servlet works:
> First of all we do not want to force users to maintain a predefined
> file system structure. We think about a system that mainly leaves
> unaltered the file system manually created by the Administrator using
> the web interface but creates and maintains it's own structure for the
> subdirectories automatically managed, compatible with the first one.
>
> Suppose that the Administrator wants to create a new subdirectory
> automatically managed by the GeoServer Ingestion Engine for a set of
> files belonging to a particular Store.
> What he has to do is creating this subdirectory and placing inside it
> a particular xml file which describes the Store type the common
> parameters and the metadata that the Ingestion Engine will use. An
> external tool, that we want to create too, can be used to create this
> configuration file. The Ingestion Servlet will scan the directory and
> every time it will encounter a new compatible file it will create a
> new subdirectory where this file (and all related) will be moved and
> the relative info.xml will be created. For FeatureStores like postgis
> those files can be just xml files containing the parameters named like
> the final FeatureType. If the Administrator deletes one or more of
> those subdirectories, the Ingestion Servlet will remove the
> Features/Coverages (and the associated Store) from the GeoServer
> configuration. Notice that the Administrator can even manually remove
> those features/coverages by using the web interface.
>
> Attached there are two images that show how the Ingestion Engine
> should work on an "auto-managed" subdirectory.
>
> Moreover notice that by adding few more metadata informations we can
> even handle WMS nested layers. We don't need to reflect the exact File
> System tree structure, we can even build a virtual WMS layer tree
> structure by handling some metadata. I will explain in detail in the
> next email.
>


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. <a href="http://ads.osdn.com/?ad_idt77&alloc_id492&op=click">http://ads.osdn.com/?ad_idt77&alloc_id492&op=click
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: R: [Geoserver-devel] Ingestion Engine proposal

Alessio Fabiani
Hi Paolo,
ok I understand what you're saying ... I agree with you, but actually I cannot figure out how we can persist the configuration somewhere. We cannot exclude the Web interface ... let me do an example. Consider that we have different ways to populate the GeoServer Memory Image and at a certain point we need to manually modify through the web interface the metadata associated with a certain DataStore or FeatureType. My question is, how can we persist changes at the end? We should preserve even informations about the config plugin or something like that ... I missing something?

Therfore now we have to make a choice between two ways I think:
1) The fastest: building a plugin integrated with the actual GeoServer implementation
2) The slowest: start to think about a better GeoServer configuration system
 
By the way, for the WCS we absolutely needs an Ingestion Engine capable to recognize file system changes, so in any case we need a plugin that writes DTOs by performing a periodic file system scan or something like this.
 
I'd like to ear something as soon as possible, eventually you can send me your code so I can take a look and try to find a common solution to the problem.
 
Dave, I'd like to ear something from you too before starting to implement something unuseful.
 
On 7/19/05, P. Rizzi Ag.Mobilità Ambiente <[hidden email]> wrote:
Hi Alessio,
we here have, like many others I think, the same need to automatically
configure GeoServer, without passing through its Web Admin interface.
We are, until now, only working with DataStores, but I believe there
should be no much differences for the Coverages part.

I read what you are thinking to do to solve this, and I should say
I personally wouldn't go in that direction, but you have for sure
your needs and for sure your best bet is to follow them.

Anyway here follow my vision of this matter.

---------------------------------------------------------------------------
I think the configuration part in GeoServer is not its best one,
because it intermixes three concerns that should instead be separated:
  - the config data itself
  - the config data persistence
  - the servlet

The configuration data, for me, is the MEMORY IMAGE a given running instance
of GeoServer has in any given instant in time, and that has nothing to do
with how that information was generated, beeing it from an XML file, from a
DB, etc.
So each configuration mechanism will persist its own data in the form it
likes more, there's no need to save and load it from a single source.

What I'm saying is that the configuration data must be separated from it's
persistence.
Now GeoServer uses several files and directories to store its configuration
and it's
only able to load the configuration from that files.
Instead the catalog.xml and all others should be only one of the many
different
persistence forms for GeoServer's configuration data. That is, the
catalog.xml
file and the pieces of Java code able to manage it, should be no different
from the ingestion engine or from whatever other mechanism one can invent.

The memory image of the configuration data is kept by GeoServer inside
the ServletContext, so that it's globally accessible from any servlet
running inside a given GeoServer instance and from any HTTP request/response
beeing serviced by that servlet. So you can modify that memory image
on-the-fly
using a servlet filter, without any need to store it using the same
persistence mechanism that GeoServer uses now.

Anyway, if one would prefer to persist all the config info inside the
same store, I won't use files and directories for that, I'd use DataStores.
GeoServer uses GeoTools to access data, and GeoTools uses DataStores
to that purpose. Configuration data are no different from other data,
so I believe the right place to store it is inside DataStores.
There's a GMLDataStore (although it's read-only at the moment), so one
can use it if he likes to store config info inside XML files.

I think that the main concern is not how one persist the config info,
but it's the config info itself, that is the Catalog.
Now GeoServer uses FeatureTypeInfo and related classes to construct
its memory image of configuration. Are that classes good enough
to support coverages too??? Are they flexible/extensible enough
to support future services??? Are they "compatible" with classes
used by other systems (like uDig for example)???
I feel this is what we should think about, the mean by which these
catalog classes are persisted should be irrilevant.

One distinction must be made here. I think that there're two different
levels of config info. One is about DataStores, that is how to connect
to each source of data. The other is about FeatureTypes, that is which
data is available, how it is structured, how it must be validated,
who has the rights to read or modify it, etc.
DataStore config info can be in simple XML files or whatever, because
you only have to say how to connect to that source of data, so the
connection params are basically all that's needed.
FeatureType config info are a different story, or better they're no
config info at all, they are metadata and they belong to the Catalog.
In our vision, and in what we have implemented so far, to add new data
to GeoServer, you simply have to add an XML file with the connection params
of the DataStore. The metadata for each FeatureType contained in that
DataStore
is read from the DataStore itself. That is, for each DataStore you want to
add
to GeoServer you MUST also add what we call a MetaStore, that is a
persistent
form of the metadata for the FeatureTypes contained in the DataStore itself.
Together with the MetaStore you MUST specify a Loader able to load that
persisted metadata in a memory image, that is a Catalog. We have a few
Loader
implemented, one is able to load metadata from FeatureSources (aka DB
tables)
with a specific structure, another one is able to infer a minimum set
of metadata from the DataStore itself, so that you can also add a DataStore
for which you actually don't have a proper MetaStore.
So we have this catalog in memory and we can use it to configure GeoServer
on-the-fly using a servlet filter (actually it is a Tomcat Valve) that
directly
"talks" to the GeoServer's catalog building DTOs on the fly.

But there's more to it...

We're also more and more convinced that some of the things GeoServer is able
to do now should be moved to GeoTools. Validation and Transactionality
should be in GeoTools, probably even the GetFeature operation should be in
GeoTools.
The central point of it all is basically about be able to operate
(querying, reading, writing, validating, etc.) against a set of DataStores
instead that against each one separately, and this capability should be in
GeoTools,
not in GeoServer.

If things were like that, many of the config info now used by GeoServer,
the ones regarding FeatureTypes metadata, would go into the GeoTools catalog
configuration.

GeoServer will then have only the config info relevant to each OGC service
it exposes (WFS, WMS, WCS, etc.) and it only needs to have references to the
metadata configured inside the GeoTools catalog.

So each service plugged into GeoServer will have it's own configuration
mechanism,
for its own configuration info. GeoServer will only need a system to
configure the plugins,
data will instead be configured inside the GeoTools catalog.
---------------------------------------------------------------------------

...that was a very long dissertation, I'm sorry...
And implementing it is much more impacting then what you're proposing,
so it may take a while to do (even if we already implemented a certain part
of it).
Also in this very moment we have other more urgent aspects to see after,
so I'm afraid I won't be able to heavily work on this for a while.
I'm very sorry about this, because I'd like to see others using what
we've done so far, I'd have to find time to make it general enough
and to publish it...

Bye
Paolo Rizzi



> -----Messaggio originale-----
> Da: Alessio Fabiani [mailto:[hidden email]]
> Inviato: martedì 19 luglio 2005 11.30
> A: [hidden email];
> [hidden email]
> Oggetto: [Geoserver-devel] Ingestion Engine proposal
>
>
> Hi all,
> I will explain in this email our proposal for a GeoServer
> Ingestion Engine.
>
> The Ingestion Engine we would like to implement for GeoServer should
> be configured as a PlugIn that an Administrator can plug into
> GeoServer and use as an alternative to the web interface to manage the
> configuration files, i.e. the "catalog.xml" which is where NameSpaces,
> DataStores and CoverageFormats parameters are stored and the different
> "info.xml" associated to each GeoServer features and coverages which
> is where all the information relative to the FeatureType or
> GridCoverage are stored.
>
> In order to achieve this objective, we do not want to modify the
> actual GeoServer configuration concept, at this moment every time an
> Administrator wants to add a new FeatureType or Coverage to GeoServer
> he has to follow several steps:
>
> Step 1: Defining the parameters and the ID for a new DataStore or
> Format. In the new release of GeoServer-WCS experiment we have renamed
> DataStore as FeatureStore and Format as CoverageStore because they are
> theoretically the same thing respectively for Vectorial and Gridded
> data. GeoServer stores all those informations in the catalog.xml .
>
> Step 2: Creating a new FeatureType or GridCoverage starting from the
> Store created in the Step 1. GeoServer creates a new directory with
> the same name of the Store ID and Feature/Coverage name and stores
> inside an info.xml file containing all the metadata associated to the
> latter.
>
> Notice that GeoServer makes the configuration files persistent only
> after the Administrator does a Save action by clicking over the button
> associated.
>
> The Ingestion Engine we have in mind should be able to perform Step 1,
> Step 2 and Save configuration automatically.
>
> We have two main objectives to achieve:
>
> 1.    Building something that is pluggable and unplaggable to
> GeoServer
> 2.    Building something that allows GeoServer to automatically modify
> the configuration performing the above steps without removing the
> actual GeoServer configuration management system
>
> To achieve the first objective we think about building a Servlet with
> his own classes that the administrator can add/remove, configure and
> enable/disable by GeoServer web.xml. This Servlet will work on a

> temporal based schedule by simply checking the file system structure
> for changes.
> To achieve the second objective the Servlet simply will automatize the
> Administrator steps for each change.
>
> How the servlet works:
> First of all we do not want to force users to maintain a predefined
> file system structure. We think about a system that mainly leaves
> unaltered the file system manually created by the Administrator using
> the web interface but creates and maintains it's own structure for the
> subdirectories automatically managed, compatible with the first one.
>
> Suppose that the Administrator wants to create a new subdirectory
> automatically managed by the GeoServer Ingestion Engine for a set of
> files belonging to a particular Store.
> What he has to do is creating this subdirectory and placing inside it
> a particular xml file which describes the Store type the common
> parameters and the metadata that the Ingestion Engine will use. An
> external tool, that we want to create too, can be used to create this
> configuration file. The Ingestion Servlet will scan the directory and
> every time it will encounter a new compatible file it will create a
> new subdirectory where this file (and all related) will be moved and
> the relative info.xml will be created. For FeatureStores like postgis
> those files can be just xml files containing the parameters named like
> the final FeatureType. If the Administrator deletes one or more of
> those subdirectories, the Ingestion Servlet will remove the
> Features/Coverages (and the associated Store) from the GeoServer
> configuration. Notice that the Administrator can even manually remove
> those features/coverages by using the web interface.

>
> Attached there are two images that show how the Ingestion Engine
> should work on an "auto-managed" subdirectory.
>
> Moreover notice that by adding few more metadata informations we can
> even handle WMS nested layers. We don't need to reflect the exact File
> System tree structure, we can even build a virtual WMS layer tree
> structure by handling some metadata. I will explain in detail in the
> next email.
>

Reply | Threaded
Open this post in threaded view
|

Re: R: [Geoserver-devel] Ingestion Engine proposal

Chris Holmes-2
In reply to this post by P.Rizzi Ag.Mobilità Ambiente
---------------------------------------------------------------------------

> I think the configuration part in GeoServer is not its best one,
> because it intermixes three concerns that should instead be
> separated:
>    - the config data itself
>    - the config data persistence
>    - the servlet
>
> The configuration data, for me, is the MEMORY IMAGE a given running
> instance
> of GeoServer has in any given instant in time, and that has nothing
> to do
> with how that information was generated, beeing it from an XML file,
> from a
> DB, etc.
> So each configuration mechanism will persist its own data in the form
> it
> likes more, there's no need to save and load it from a single source.
>
> What I'm saying is that the configuration data must be separated from
> it's
> persistence.
> Now GeoServer uses several files and directories to store its
> configuration
> and it's
> only able to load the configuration from that files.
This isn't true.  It's also able to 'load' its configuration from the
web admin tool.  We are actually completely independant from the xml
files, it's just the only way we have to _persist_.  But the structures
are all there to take different ways to load and store files.  This was
a side effect of me being anal about separation with the building of
the web admin tool.  All loading is done through DTO objects.  The xml
loader creates DTO objects, as does the web admin tool.  If you want a
datastore loader, you just have to make it create the appropriate dto
files.  You just have to write some code to make the initial geoserver
start up get its DTO's from some other source.  Shouldn't be that hard
to code, could probably just pass it in as a web.xml param, default to
xml, but also allow the others.  You may have to add one slight layer
of abstraction, but the core is there, loading the memory image of
geoserver is completely abstracted, it just needs to get into DTOs.

> Instead the catalog.xml and all others should be only one of the many
> different
> persistence forms for GeoServer's configuration data. That is, the
> catalog.xml
> file and the pieces of Java code able to manage it, should be no
> different
> from the ingestion engine or from whatever other mechanism one can
> invent.
Agreed.  But I think we already have a decent start to this structure.
Granted it could use a nice rewrite, but I think you can do a smaller
incremental step with the ingestion engine - it just writes out DTOs
and says 'load'.  That can then be adjusted with the web admin tool.
And if you want you can use the current structure and write out a
different persistance mechanism.

>
> The memory image of the configuration data is kept by GeoServer
> inside
> the ServletContext, so that it's globally accessible from any servlet
> running inside a given GeoServer instance and from any HTTP
> request/response
> beeing serviced by that servlet. So you can modify that memory image
> on-the-fly
> using a servlet filter, without any need to store it using the same
> persistence mechanism that GeoServer uses now.
Or you can pass it a DTO object.  I admit that that DTO's may be one
level of abstraction too far, could use the memory image directly, but
the structure is there to do what you want.

>
> Anyway, if one would prefer to persist all the config info inside the
> same store, I won't use files and directories for that, I'd use
> DataStores.
> GeoServer uses GeoTools to access data, and GeoTools uses DataStores
> to that purpose. Configuration data are no different from other data,
> so I believe the right place to store it is inside DataStores.
> There's a GMLDataStore (although it's read-only at the moment), so
> one
> can use it if he likes to store config info inside XML files.
Cool, you can code this up and we can all try it out.  Just have it
create and save DTOs.  If it ends up better and more flexible, we can
adopt it.  This is what dave was getting at with a 'new' web admin
tool, we can still make use of the old one.  And you can do the same
with persisting to data stores.  And if it fully does the xml files
right, then we can switch it all over.

>
> I think that the main concern is not how one persist the config info,
> but it's the config info itself, that is the Catalog.
> Now GeoServer uses FeatureTypeInfo and related classes to construct
> its memory image of configuration. Are that classes good enough
> to support coverages too??? Are they flexible/extensible enough
> to support future services??? Are they "compatible" with classes
> used by other systems (like uDig for example)???
> I feel this is what we should think about, the mean by which these
> catalog classes are persisted should be irrilevant.
>
> One distinction must be made here. I think that there're two
> different
> levels of config info. One is about DataStores, that is how to
> connect
> to each source of data. The other is about FeatureTypes, that is
> which
> data is available, how it is structured, how it must be validated,
> who has the rights to read or modify it, etc.
> DataStore config info can be in simple XML files or whatever, because
> you only have to say how to connect to that source of data, so the
> connection params are basically all that's needed.
> FeatureType config info are a different story, or better they're no
> config info at all, they are metadata and they belong to the Catalog.
> In our vision, and in what we have implemented so far, to add new
> data
> to GeoServer, you simply have to add an XML file with the connection
> params
> of the DataStore. The metadata for each FeatureType contained in that
> DataStore
> is read from the DataStore itself. That is, for each DataStore you
> want to
> add
> to GeoServer you MUST also add what we call a MetaStore, that is a
> persistent
> form of the metadata for the FeatureTypes contained in the DataStore
> itself.
> Together with the MetaStore you MUST specify a Loader able to load
> that
> persisted metadata in a memory image, that is a Catalog. We have a
> few
> Loader
> implemented, one is able to load metadata from FeatureSources (aka DB
> tables)
> with a specific structure, another one is able to infer a minimum set
> of metadata from the DataStore itself, so that you can also add a
> DataStore
> for which you actually don't have a proper MetaStore.
> So we have this catalog in memory and we can use it to configure
> GeoServer
> on-the-fly using a servlet filter (actually it is a Tomcat Valve)
> that
> directly
> "talks" to the GeoServer's catalog building DTOs on the fly.
>
> But there's more to it...
>
> We're also more and more convinced that some of the things GeoServer
> is able
> to do now should be moved to GeoTools. Validation and
> Transactionality
> should be in GeoTools, probably even the GetFeature operation should
> be in
> GeoTools.
> The central point of it all is basically about be able to operate
> (querying, reading, writing, validating, etc.) against a set of
> DataStores
> instead that against each one separately, and this capability should
> be in
> GeoTools,
> not in GeoServer.
I agree with all of this.  And this is easily one of the primary goals
of the geoserver 2.0 rewrite.  Get the catalog and meta information in
geotools.

>
> If things were like that, many of the config info now used by
> GeoServer,
> the ones regarding FeatureTypes metadata, would go into the GeoTools
> catalog
> configuration.
>
> GeoServer will then have only the config info relevant to each OGC
> service
> it exposes (WFS, WMS, WCS, etc.) and it only needs to have references
> to the
> metadata configured inside the GeoTools catalog.
>
> So each service plugged into GeoServer will have it's own
> configuration
> mechanism,
> for its own configuration info. GeoServer will only need a system to
> configure the plugins,
> data will instead be configured inside the GeoTools catalog.
And that's basically the other big goal, geoserver as the plug-in
machine.

>
---------------------------------------------------------------------------

>
> ...that was a very long dissertation, I'm sorry...
> And implementing it is much more impacting then what you're
> proposing,
> so it may take a while to do (even if we already implemented a
> certain part
> of it).
> Also in this very moment we have other more urgent aspects to see
> after,
> so I'm afraid I won't be able to heavily work on this for a while.
> I'm very sorry about this, because I'd like to see others using what
> we've done so far, I'd have to find time to make it general enough
> and to publish it...
Yeah, I think we're pretty much agreed on these things as long term
goals, what I'm calling geoserver 2.0.  For the immediate I think just
focusing on DTOs and the current mechanisms we have in place will make
for a decent ingestion engine, and we can take the lessons from there
and apply them to the next config design, since with 2.0 we can even
redo the format of the config files if we'd like.  I'd always like the
XML option, and the datastore persistance idea is interesting - being
able to persist to any datastore would be nice.  We'll have to play
with it and see.

best regards,

Chris


>
> Bye
> Paolo Rizzi
>
>
>
> > -----Messaggio originale-----
> > Da: Alessio Fabiani [mailto:[hidden email]]
> > Inviato: martedì 19 luglio 2005 11.30
> > A: [hidden email];
> > [hidden email]
> > Oggetto: [Geoserver-devel] Ingestion Engine proposal
> >
> >
> > Hi all,
> > I will explain in this email our proposal for a GeoServer
> > Ingestion Engine.
> >
> > The Ingestion Engine we would like to implement for GeoServer
> should
> > be configured as a PlugIn that an Administrator can plug into
> > GeoServer and use as an alternative to the web interface to manage
> the
> > configuration files, i.e. the "catalog.xml" which is where
> NameSpaces,
> > DataStores and CoverageFormats parameters are stored and the
> different
> > "info.xml" associated to each GeoServer features and coverages
> which
> > is where all the information relative to the FeatureType or
> > GridCoverage are stored.
> >
> > In order to achieve this objective, we do not want to modify the
> > actual GeoServer configuration concept, at this moment every time
> an
> > Administrator wants to add a new FeatureType or Coverage to
> GeoServer
> > he has to follow several steps:
> >
> > Step 1: Defining the parameters and the ID for a new DataStore or
> > Format. In the new release of GeoServer-WCS experiment we have
> renamed
> > DataStore as FeatureStore and Format as CoverageStore because they
> are
> > theoretically the same thing respectively for Vectorial and Gridded
> > data. GeoServer stores all those informations in the catalog.xml.
> >
> > Step 2: Creating a new FeatureType or GridCoverage starting from
> the
> > Store created in the Step 1. GeoServer creates a new directory with
> > the same name of the Store ID and Feature/Coverage name and stores
> > inside an info.xml file containing all the metadata associated to
> the
> > latter.
> >
> > Notice that GeoServer makes the configuration files persistent only
> > after the Administrator does a Save action by clicking over the
> button
> > associated.
> >
> > The Ingestion Engine we have in mind should be able to perform Step
> 1,
> > Step 2 and Save configuration automatically.
> >
> > We have two main objectives to achieve:
> >
> > 1. Building something that is pluggable and unplaggable to
> > GeoServer
> > 2. Building something that allows GeoServer to automatically modify
> > the configuration performing the above steps without removing the
> > actual GeoServer configuration management system
> >
> > To achieve the first objective we think about building a Servlet
> with
> > his own classes that the administrator can add/remove, configure
> and
> > enable/disable by GeoServer web.xml. This Servlet will work on a
> > temporal based schedule by simply checking the file system
> structure
> > for changes.
> > To achieve the second objective the Servlet simply will automatize
> the
> > Administrator steps for each change.
> >
> > How the servlet works:
> > First of all we do not want to force users to maintain a predefined
> > file system structure. We think about a system that mainly leaves
> > unaltered the file system manually created by the Administrator
> using
> > the web interface but creates and maintains it's own structure for
> the
> > subdirectories automatically managed, compatible with the first
> one.
> >
> > Suppose that the Administrator wants to create a new subdirectory
> > automatically managed by the GeoServer Ingestion Engine for a set
> of
> > files belonging to a particular Store.
> > What he has to do is creating this subdirectory and placing inside
> it
> > a particular xml file which describes the Store type the common
> > parameters and the metadata that the Ingestion Engine will use. An
> > external tool, that we want to create too, can be used to create
> this
> > configuration file. The Ingestion Servlet will scan the directory
> and
> > every time it will encounter a new compatible file it will create a
> > new subdirectory where this file (and all related) will be moved
> and
> > the relative info.xml will be created. For FeatureStores like
> postgis
> > those files can be just xml files containing the parameters named
> like
> > the final FeatureType. If the Administrator deletes one or more of
> > those subdirectories, the Ingestion Servlet will remove the
> > Features/Coverages (and the associated Store) from the GeoServer
> > configuration. Notice that the Administrator can even manually
> remove
> > those features/coverages by using the web interface.
> >
> > Attached there are two images that show how the Ingestion Engine
> > should work on an "auto-managed" subdirectory.
> >
> > Moreover notice that by adding few more metadata informations we
> can
> > even handle WMS nested layers. We don't need to reflect the exact
> File
> > System tree structure, we can even build a virtual WMS layer tree
> > structure by handling some metadata. I will explain in detail in
> the
> > next email.
> >
>
>
> -------------------------------------------------------
> SF.Net email is sponsored by: Discover Easy Linux Migration
> Strategies
> from IBM. Find simple to follow Roadmaps, straightforward articles,
> informative Webcasts and more! Get everything you need to get up to
> speed, fast. <a href="http://ads.osdn.com/?ad_idt77&alloc_id492&op=click">http://ads.osdn.com/?ad_idt77&alloc_id492&op=click
> _______________________________________________
> Geoserver-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>




----------------------------------------------------------
This mail sent through IMP: https://webmail.limegroup.com/


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. <a href="http://ads.osdn.com/?ad_idt77&alloc_id492&op=click">http://ads.osdn.com/?ad_idt77&alloc_id492&op=click
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: R: [Geoserver-devel] Ingestion Engine proposal

Jody Garnett
In reply to this post by P.Rizzi Ag.Mobilità Ambiente
P. Rizzi Ag.Mobilità Ambiente wrote:

>Hi Alessio,
>we here have, like many others I think, the same need to automatically
>configure GeoServer, without passing through its Web Admin interface.
>We are, until now, only working with DataStores, but I believe there
>should be no much differences for the Coverages part.
>  
>
I was going to say we design GeoServe with the use of DTO objects for
exactly this kind of use .. the DTO objects are nice and serializable
and there are facilities to turn persist them to and from XML (from
another web app, or off to disk).

Everyone knows what a Data Transfer Object is right?

Jody


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. <a href="http://ads.osdn.com/?ad_idt77&alloc_id492&op=click">http://ads.osdn.com/?ad_idt77&alloc_id492&op=click
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel