Improving App-Schema connection usage

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Improving App-Schema connection usage

Stefano Costa-3
Dear developers,
picking up from this discussion:

I've been working on making App-Schema Data Stores use a single connection when constructing complex features, no matter how many levels of nested features must be chained together to do so.

I have a working implementation in this PR:

The implementation works only if joining is enabled, which in turn requires a database backend be used for all joined types, but this is not a limitation, just the context where this improvement makes any sense.

There is however an actual limitation. To achieve connection re-use among the many feature iterators required to build a feature containing several levels of joined types, I've made them share a Transaction instance, as suggested by Mauro in the discussion linked above, which is then passed to the underlying JDBCFeatureSource; however, for this to result in the same connection being used, all the feature sources must have been created by the same (as in Object.equals) JDBCDataStore.
To achieve this I had to teach AppSchemaDataConfigurator to keep track of the source data stores created while parsing the XML mapping configuration (which may be spread over multiple files) and create a brand new data store only if an existing data store with the same configuration parameters is not available:

Unfortunately, the "tracking" is done on a per-store basis, so feature types belonging to different App-Schema data stores will not be able to share the connection, even if the underlying JDBC data store is actually the same. I preferred this simpler (but still useful) solution to avoid introducing a sort of global registry for source data stores.

Does anybody have any further thoughts on this?

Many thanks,
Stefano


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
GeoTools-Devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geotools-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Improving App-Schema connection usage

Stefano Costa-3
Hi all,
It's been slightly more than a month since this work was merged on master and I would like to backport it to stable and maintenance.

Any objections?

Thanks,
Stefano


Il 04/giu/2017 18:59, "Stefano Costa" <[hidden email]> ha scritto:
Dear developers,
picking up from this discussion:

I've been working on making App-Schema Data Stores use a single connection when constructing complex features, no matter how many levels of nested features must be chained together to do so.

I have a working implementation in this PR:

The implementation works only if joining is enabled, which in turn requires a database backend be used for all joined types, but this is not a limitation, just the context where this improvement makes any sense.

There is however an actual limitation. To achieve connection re-use among the many feature iterators required to build a feature containing several levels of joined types, I've made them share a Transaction instance, as suggested by Mauro in the discussion linked above, which is then passed to the underlying JDBCFeatureSource; however, for this to result in the same connection being used, all the feature sources must have been created by the same (as in Object.equals) JDBCDataStore.
To achieve this I had to teach AppSchemaDataConfigurator to keep track of the source data stores created while parsing the XML mapping configuration (which may be spread over multiple files) and create a brand new data store only if an existing data store with the same configuration parameters is not available:

Unfortunately, the "tracking" is done on a per-store basis, so feature types belonging to different App-Schema data stores will not be able to share the connection, even if the underlying JDBC data store is actually the same. I preferred this simpler (but still useful) solution to avoid introducing a sort of global registry for source data stores.

Does anybody have any further thoughts on this?

Many thanks,
Stefano



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
GeoTools-Devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geotools-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Improving App-Schema connection usage

Ben Caradoc-Davies-2
+1 for backport to stable. +0 for maintenance, but this is arguably a
bug fix that should be backported to maintenance as well.

Kind regards,
Ben.

On 15/07/17 09:46, Stefano Costa wrote:

> Hi all,
> It's been slightly more than a month since this work was merged on master
> and I would like to backport it to stable and maintenance.
>
> Any objections?
>
> Thanks,
> Stefano
>
>
> Il 04/giu/2017 18:59, "Stefano Costa" <[hidden email]> ha scritto:
>
> Dear developers,
> picking up from this discussion:
> http://osgeo-org.1560.x6.nabble.com/PostGIS-Connections-hanging-under-App-
> Schema-Plugin-td5319085.html#a5319171
>
> I've been working on making App-Schema Data Stores use a single connection
> when constructing complex features, no matter how many levels of nested
> features must be chained together to do so.
>
> I have a working implementation in this PR:
> https://github.com/geotools/geotools/pull/1616
>
> The implementation works only if joining is enabled, which in turn requires
> a database backend be used for all joined types, but this is not a
> limitation, just the context where this improvement makes any sense.
>
> There is however an actual limitation. To achieve connection re-use among
> the many feature iterators required to build a feature containing several
> levels of joined types, I've made them share a Transaction instance, as
> suggested by Mauro in the discussion linked above, which is then passed to
> the underlying JDBCFeatureSource; however, for this to result in the same
> connection being used, all the feature sources must have been created by
> the same (as in Object.equals) JDBCDataStore.
> To achieve this I had to teach AppSchemaDataConfigurator to keep track of
> the source data stores created while parsing the XML mapping configuration
> (which may be spread over multiple files) and create a brand new data store
> only if an existing data store with the same configuration parameters is
> not available:
> https://github.com/geotools/geotools/pull/1616/files#diff-
> c8fc7a6b4988b5953aafd9f69af6da98R702
>
> Unfortunately, the "tracking" is done on a per-store basis, so feature
> types belonging to different App-Schema data stores will not be able to
> share the connection, even if the underlying JDBC data store is actually
> the same. I preferred this simpler (but still useful) solution to avoid
> introducing a sort of global registry for source data stores.
>
> Does anybody have any further thoughts on this?
>
> Many thanks,
> Stefano
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
>
> _______________________________________________
> GeoTools-Devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>

--
Ben Caradoc-Davies <[hidden email]>
Director
Transient Software Limited <http://transient.nz/>
New Zealand

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
GeoTools-Devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geotools-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Improving App-Schema connection usage

Stefano Costa-3
Thanks, Ben, for the quick response.

PRs are out for stable:


...and maintenance:


I would like to have the PRs against stable merged before the next release is cut (scheduled on July 18th); for maintenance, I can wait longer.

Best regards,
Stefano


On Sat, Jul 15, 2017 at 3:53 AM, Ben Caradoc-Davies <[hidden email]> wrote:
+1 for backport to stable. +0 for maintenance, but this is arguably a bug fix that should be backported to maintenance as well.

Kind regards,
Ben.


On 15/07/17 09:46, Stefano Costa wrote:
Hi all,
It's been slightly more than a month since this work was merged on master
and I would like to backport it to stable and maintenance.

Any objections?

Thanks,
Stefano


Il 04/giu/2017 18:59, "Stefano Costa" <[hidden email]> ha scritto:

Dear developers,
picking up from this discussion:
http://osgeo-org.1560.x6.nabble.com/PostGIS-Connections-hanging-under-App-
Schema-Plugin-td5319085.html#a5319171

I've been working on making App-Schema Data Stores use a single connection
when constructing complex features, no matter how many levels of nested
features must be chained together to do so.

I have a working implementation in this PR:
https://github.com/geotools/geotools/pull/1616

The implementation works only if joining is enabled, which in turn requires
a database backend be used for all joined types, but this is not a
limitation, just the context where this improvement makes any sense.

There is however an actual limitation. To achieve connection re-use among
the many feature iterators required to build a feature containing several
levels of joined types, I've made them share a Transaction instance, as
suggested by Mauro in the discussion linked above, which is then passed to
the underlying JDBCFeatureSource; however, for this to result in the same
connection being used, all the feature sources must have been created by
the same (as in Object.equals) JDBCDataStore.
To achieve this I had to teach AppSchemaDataConfigurator to keep track of
the source data stores created while parsing the XML mapping configuration
(which may be spread over multiple files) and create a brand new data store
only if an existing data store with the same configuration parameters is
not available:
https://github.com/geotools/geotools/pull/1616/files#diff-
c8fc7a6b4988b5953aafd9f69af6da98R702

Unfortunately, the "tracking" is done on a per-store basis, so feature
types belonging to different App-Schema data stores will not be able to
share the connection, even if the underlying JDBC data store is actually
the same. I preferred this simpler (but still useful) solution to avoid
introducing a sort of global registry for source data stores.

Does anybody have any further thoughts on this?

Many thanks,
Stefano



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot



_______________________________________________
GeoTools-Devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geotools-devel


--
Ben Caradoc-Davies <[hidden email]>
Director
Transient Software Limited <http://transient.nz/>
New Zealand


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
GeoTools-Devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geotools-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Improving App-Schema connection usage

Ben Caradoc-Davies-2
All merged. Thanks very much, Stefano.

Kind regards,
Ben.


On 16/07/17 01:40, Stefano Costa wrote:

> Thanks, Ben, for the quick response.
>
> PRs are out for stable:
>
> https://github.com/geotools/geotools/pull/1654
> https://github.com/geoserver/geoserver/pull/2463
>
> ...and maintenance:
>
> https://github.com/geotools/geotools/pull/1655
> https://github.com/geoserver/geoserver/pull/2464
>
> I would like to have the PRs against stable merged before the next release
> is cut (scheduled on July 18th); for maintenance, I can wait longer.
>
> Best regards,
> Stefano
>
>
> On Sat, Jul 15, 2017 at 3:53 AM, Ben Caradoc-Davies <[hidden email]>
> wrote:
>
>> +1 for backport to stable. +0 for maintenance, but this is arguably a bug
>> fix that should be backported to maintenance as well.
>>
>> Kind regards,
>> Ben.
>>
>>
>> On 15/07/17 09:46, Stefano Costa wrote:
>>
>>> Hi all,
>>> It's been slightly more than a month since this work was merged on master
>>> and I would like to backport it to stable and maintenance.
>>>
>>> Any objections?
>>>
>>> Thanks,
>>> Stefano
>>>
>>>
>>> Il 04/giu/2017 18:59, "Stefano Costa" <[hidden email]> ha
>>> scritto:
>>>
>>> Dear developers,
>>> picking up from this discussion:
>>> http://osgeo-org.1560.x6.nabble.com/PostGIS-Connections-
>>> hanging-under-App-
>>> Schema-Plugin-td5319085.html#a5319171
>>>
>>> I've been working on making App-Schema Data Stores use a single connection
>>> when constructing complex features, no matter how many levels of nested
>>> features must be chained together to do so.
>>>
>>> I have a working implementation in this PR:
>>> https://github.com/geotools/geotools/pull/1616
>>>
>>> The implementation works only if joining is enabled, which in turn
>>> requires
>>> a database backend be used for all joined types, but this is not a
>>> limitation, just the context where this improvement makes any sense.
>>>
>>> There is however an actual limitation. To achieve connection re-use among
>>> the many feature iterators required to build a feature containing several
>>> levels of joined types, I've made them share a Transaction instance, as
>>> suggested by Mauro in the discussion linked above, which is then passed to
>>> the underlying JDBCFeatureSource; however, for this to result in the same
>>> connection being used, all the feature sources must have been created by
>>> the same (as in Object.equals) JDBCDataStore.
>>> To achieve this I had to teach AppSchemaDataConfigurator to keep track of
>>> the source data stores created while parsing the XML mapping configuration
>>> (which may be spread over multiple files) and create a brand new data
>>> store
>>> only if an existing data store with the same configuration parameters is
>>> not available:
>>> https://github.com/geotools/geotools/pull/1616/files#diff-
>>> c8fc7a6b4988b5953aafd9f69af6da98R702
>>>
>>> Unfortunately, the "tracking" is done on a per-store basis, so feature
>>> types belonging to different App-Schema data stores will not be able to
>>> share the connection, even if the underlying JDBC data store is actually
>>> the same. I preferred this simpler (but still useful) solution to avoid
>>> introducing a sort of global registry for source data stores.
>>>
>>> Does anybody have any further thoughts on this?
>>>
>>> Many thanks,
>>> Stefano
>>>
>>>
>>>
>>> ------------------------------------------------------------
>>> ------------------
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>
>>>
>>>
>>> _______________________________________________
>>> GeoTools-Devel mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>>>
>>>
>> --
>> Ben Caradoc-Davies <[hidden email]>
>> Director
>> Transient Software Limited <http://transient.nz/>
>> New Zealand
>>
>

--
Ben Caradoc-Davies <[hidden email]>
Director
Transient Software Limited <http://transient.nz/>
New Zealand

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
GeoTools-Devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geotools-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Improving App-Schema connection usage

Stefano Costa-3
Thank you, Ben, for taking the time to watch over failing Travis builds and persevere until merge! :-)

Best,
Stefano


On Mon, Jul 17, 2017 at 6:07 AM, Ben Caradoc-Davies <[hidden email]> wrote:
All merged. Thanks very much, Stefano.

Kind regards,
Ben.



On 16/07/17 01:40, Stefano Costa wrote:
Thanks, Ben, for the quick response.

PRs are out for stable:

https://github.com/geotools/geotools/pull/1654
https://github.com/geoserver/geoserver/pull/2463

...and maintenance:

https://github.com/geotools/geotools/pull/1655
https://github.com/geoserver/geoserver/pull/2464

I would like to have the PRs against stable merged before the next release
is cut (scheduled on July 18th); for maintenance, I can wait longer.

Best regards,
Stefano


On Sat, Jul 15, 2017 at 3:53 AM, Ben Caradoc-Davies <[hidden email]>
wrote:

+1 for backport to stable. +0 for maintenance, but this is arguably a bug
fix that should be backported to maintenance as well.

Kind regards,
Ben.


On 15/07/17 09:46, Stefano Costa wrote:

Hi all,
It's been slightly more than a month since this work was merged on master
and I would like to backport it to stable and maintenance.

Any objections?

Thanks,
Stefano


Il 04/giu/2017 18:59, "Stefano Costa" <[hidden email]> ha
scritto:

Dear developers,
picking up from this discussion:
http://osgeo-org.1560.x6.nabble.com/PostGIS-Connections-
hanging-under-App-
Schema-Plugin-td5319085.html#a5319171

I've been working on making App-Schema Data Stores use a single connection
when constructing complex features, no matter how many levels of nested
features must be chained together to do so.

I have a working implementation in this PR:
https://github.com/geotools/geotools/pull/1616

The implementation works only if joining is enabled, which in turn
requires
a database backend be used for all joined types, but this is not a
limitation, just the context where this improvement makes any sense.

There is however an actual limitation. To achieve connection re-use among
the many feature iterators required to build a feature containing several
levels of joined types, I've made them share a Transaction instance, as
suggested by Mauro in the discussion linked above, which is then passed to
the underlying JDBCFeatureSource; however, for this to result in the same
connection being used, all the feature sources must have been created by
the same (as in Object.equals) JDBCDataStore.
To achieve this I had to teach AppSchemaDataConfigurator to keep track of
the source data stores created while parsing the XML mapping configuration
(which may be spread over multiple files) and create a brand new data
store
only if an existing data store with the same configuration parameters is
not available:
https://github.com/geotools/geotools/pull/1616/files#diff-
c8fc7a6b4988b5953aafd9f69af6da98R702

Unfortunately, the "tracking" is done on a per-store basis, so feature
types belonging to different App-Schema data stores will not be able to
share the connection, even if the underlying JDBC data store is actually
the same. I preferred this simpler (but still useful) solution to avoid
introducing a sort of global registry for source data stores.

Does anybody have any further thoughts on this?

Many thanks,
Stefano



------------------------------------------------------------
------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot



_______________________________________________
GeoTools-Devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geotools-devel


--
Ben Caradoc-Davies <[hidden email]>
Director
Transient Software Limited <http://transient.nz/>
New Zealand



--
Ben Caradoc-Davies <[hidden email]>
Director
Transient Software Limited <http://transient.nz/>
New Zealand


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
GeoTools-Devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geotools-devel
Loading...