Problems with the FeatureSizeFeatureCollection

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Problems with the FeatureSizeFeatureCollection

Jim Hughes-2
Hi all,

While testing out GeoMesa with GeoServer 2.13.x, we ran into some issues with the FeatureSizeFeatureCollection.  I think there are two issues:

First, for a layers backed by a large volume of data, getCount returning -1 is partially a way of saying that you don't want to count all the features;).  Given this pattern, I'd suggest that the FeatureSizeFeatureCollection might be taken off the usual path completely.  It looks like it might be an optimization when working with a bunch of little shapefiles.

Second, WPSes can be implemented with a visitor pattern.  For instance, the gs:Unique process leverages the GeoTools UniqueVisitor(1) which is recognized by the JDBC DataStores.  (In GeoMesa, we do a bunch of similar things.)

The fact that the current FeatureSizeFeatureCollection doesn't delegate prevents the UniqueProcess (and the GeoMesa ones) from working properly...

In the short term, any objection to a PR which enables delegation for that FeatureCollection?

Cheers,

Jim

1. UniqueVisitor: https://github.com/geotools/geotools/blob/master/modules/library/main/src/main/java/org/geotools/feature/visitor/UniqueVisitor.java
https://github.com/geotools/geotools/blob/master/modules/library/jdbc/src/main/java/org/geotools/jdbc/SQLDialect.java#L375

p.s.  I debugged this using PostGIS and the gs:Unique process. When the FeatureSizeFC was 'on', the queries did not have the 'distinct' keyword.  When the FC was off, GeoServer did make a 'distinct' query.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Problems with the FeatureSizeFeatureCollection

geowolf
Ah, 
that change was controversial and the pull request sat there for ages before being merged:

(shows me as the author, but it's not really me, I'm just unwillingly involved because nobody else would take action)

Please discuss.

Cheers
Andrea


On Fri, Jun 8, 2018 at 5:57 PM, Jim Hughes <[hidden email]> wrote:
Hi all,

While testing out GeoMesa with GeoServer 2.13.x, we ran into some issues with the FeatureSizeFeatureCollection.  I think there are two issues:

First, for a layers backed by a large volume of data, getCount returning -1 is partially a way of saying that you don't want to count all the features;).  Given this pattern, I'd suggest that the FeatureSizeFeatureCollection might be taken off the usual path completely.  It looks like it might be an optimization when working with a bunch of little shapefiles.

Second, WPSes can be implemented with a visitor pattern.  For instance, the gs:Unique process leverages the GeoTools UniqueVisitor(1) which is recognized by the JDBC DataStores.  (In GeoMesa, we do a bunch of similar things.)

The fact that the current FeatureSizeFeatureCollection doesn't delegate prevents the UniqueProcess (and the GeoMesa ones) from working properly...

In the short term, any objection to a PR which enables delegation for that FeatureCollection?

Cheers,

Jim

1. UniqueVisitor: https://github.com/geotools/geotools/blob/master/modules/library/main/src/main/java/org/geotools/feature/visitor/UniqueVisitor.java
https://github.com/geotools/geotools/blob/master/modules/library/jdbc/src/main/java/org/geotools/jdbc/SQLDialect.java#L375

p.s.  I debugged this using PostGIS and the gs:Unique process. When the FeatureSizeFC was 'on', the queries did not have the 'distinct' keyword.  When the FC was off, GeoServer did make a 'distinct' query.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel




--

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Problems with the FeatureSizeFeatureCollection

Jim Hughes-2
Andrea,

Thanks for helping the PR through. 

I haven't tested out the first case completely.  I've definitely run into the issue that Alvaro is trying to address with WFS 2.0. 

In the short term, is allowing the FC to delegate reasonable?

Cheers,

Jim

On 06/08/2018 12:07 PM, Andrea Aime wrote:
Ah, 
that change was controversial and the pull request sat there for ages before being merged:

(shows me as the author, but it's not really me, I'm just unwillingly involved because nobody else would take action)

Please discuss.

Cheers
Andrea


On Fri, Jun 8, 2018 at 5:57 PM, Jim Hughes <[hidden email]> wrote:
Hi all,

While testing out GeoMesa with GeoServer 2.13.x, we ran into some issues with the FeatureSizeFeatureCollection.  I think there are two issues:

First, for a layers backed by a large volume of data, getCount returning -1 is partially a way of saying that you don't want to count all the features;).  Given this pattern, I'd suggest that the FeatureSizeFeatureCollection might be taken off the usual path completely.  It looks like it might be an optimization when working with a bunch of little shapefiles.

Second, WPSes can be implemented with a visitor pattern.  For instance, the gs:Unique process leverages the GeoTools UniqueVisitor(1) which is recognized by the JDBC DataStores.  (In GeoMesa, we do a bunch of similar things.)

The fact that the current FeatureSizeFeatureCollection doesn't delegate prevents the UniqueProcess (and the GeoMesa ones) from working properly...

In the short term, any objection to a PR which enables delegation for that FeatureCollection?

Cheers,

Jim

1. UniqueVisitor: https://github.com/geotools/geotools/blob/master/modules/library/main/src/main/java/org/geotools/feature/visitor/UniqueVisitor.java
https://github.com/geotools/geotools/blob/master/modules/library/jdbc/src/main/java/org/geotools/jdbc/SQLDialect.java#L375

p.s.  I debugged this using PostGIS and the gs:Unique process. When the FeatureSizeFC was 'on', the queries did not have the 'distinct' keyword.  When the FC was off, GeoServer did make a 'distinct' query.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel




--

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Problems with the FeatureSizeFeatureCollection

geowolf
Hi Jim,
yes, properly implemented collection wrappers should allow delegation 

Cheers
Andrea

On Fri, Jun 8, 2018 at 6:47 PM, Jim Hughes <[hidden email]> wrote:
Andrea,

Thanks for helping the PR through. 

I haven't tested out the first case completely.  I've definitely run into the issue that Alvaro is trying to address with WFS 2.0. 

In the short term, is allowing the FC to delegate reasonable?

Cheers,

Jim


On 06/08/2018 12:07 PM, Andrea Aime wrote:
Ah, 
that change was controversial and the pull request sat there for ages before being merged:

(shows me as the author, but it's not really me, I'm just unwillingly involved because nobody else would take action)

Please discuss.

Cheers
Andrea


On Fri, Jun 8, 2018 at 5:57 PM, Jim Hughes <[hidden email]> wrote:
Hi all,

While testing out GeoMesa with GeoServer 2.13.x, we ran into some issues with the FeatureSizeFeatureCollection.  I think there are two issues:

First, for a layers backed by a large volume of data, getCount returning -1 is partially a way of saying that you don't want to count all the features;).  Given this pattern, I'd suggest that the FeatureSizeFeatureCollection might be taken off the usual path completely.  It looks like it might be an optimization when working with a bunch of little shapefiles.

Second, WPSes can be implemented with a visitor pattern.  For instance, the gs:Unique process leverages the GeoTools UniqueVisitor(1) which is recognized by the JDBC DataStores.  (In GeoMesa, we do a bunch of similar things.)

The fact that the current FeatureSizeFeatureCollection doesn't delegate prevents the UniqueProcess (and the GeoMesa ones) from working properly...

In the short term, any objection to a PR which enables delegation for that FeatureCollection?

Cheers,

Jim

1. UniqueVisitor: https://github.com/geotools/geotools/blob/master/modules/library/main/src/main/java/org/geotools/feature/visitor/UniqueVisitor.java
https://github.com/geotools/geotools/blob/master/modules/library/jdbc/src/main/java/org/geotools/jdbc/SQLDialect.java#L375

p.s.  I debugged this using PostGIS and the gs:Unique process. When the FeatureSizeFC was 'on', the queries did not have the 'distinct' keyword.  When the FC was off, GeoServer did make a 'distinct' query.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel




--

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.





--

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Problems with the FeatureSizeFeatureCollection

Jim Hughes-2
Hi Andrea,

I tossed up a PR here: https://github.com/geoserver/geoserver/pull/2905 with a ticket https://osgeo-org.atlassian.net/browse/GEOS-8779.

I am not seeing a quick way to provide a unit test.  If you (or anyone) has a suggestion on that front, I'm happy to include one.

Cheers,

Jim

On 06/08/2018 12:49 PM, Andrea Aime wrote:
Hi Jim,
yes, properly implemented collection wrappers should allow delegation 

Cheers
Andrea

On Fri, Jun 8, 2018 at 6:47 PM, Jim Hughes <[hidden email]> wrote:
Andrea,

Thanks for helping the PR through. 

I haven't tested out the first case completely.  I've definitely run into the issue that Alvaro is trying to address with WFS 2.0. 

In the short term, is allowing the FC to delegate reasonable?

Cheers,

Jim


On 06/08/2018 12:07 PM, Andrea Aime wrote:
Ah, 
that change was controversial and the pull request sat there for ages before being merged:

(shows me as the author, but it's not really me, I'm just unwillingly involved because nobody else would take action)

Please discuss.

Cheers
Andrea


On Fri, Jun 8, 2018 at 5:57 PM, Jim Hughes <[hidden email]> wrote:
Hi all,

While testing out GeoMesa with GeoServer 2.13.x, we ran into some issues with the FeatureSizeFeatureCollection.  I think there are two issues:

First, for a layers backed by a large volume of data, getCount returning -1 is partially a way of saying that you don't want to count all the features;).  Given this pattern, I'd suggest that the FeatureSizeFeatureCollection might be taken off the usual path completely.  It looks like it might be an optimization when working with a bunch of little shapefiles.

Second, WPSes can be implemented with a visitor pattern.  For instance, the gs:Unique process leverages the GeoTools UniqueVisitor(1) which is recognized by the JDBC DataStores.  (In GeoMesa, we do a bunch of similar things.)

The fact that the current FeatureSizeFeatureCollection doesn't delegate prevents the UniqueProcess (and the GeoMesa ones) from working properly...

In the short term, any objection to a PR which enables delegation for that FeatureCollection?

Cheers,

Jim

1. UniqueVisitor: https://github.com/geotools/geotools/blob/master/modules/library/main/src/main/java/org/geotools/feature/visitor/UniqueVisitor.java
https://github.com/geotools/geotools/blob/master/modules/library/jdbc/src/main/java/org/geotools/jdbc/SQLDialect.java#L375

p.s.  I debugged this using PostGIS and the gs:Unique process. When the FeatureSizeFC was 'on', the queries did not have the 'distinct' keyword.  When the FC was off, GeoServer did make a 'distinct' query.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel




--

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.





--

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Problems with the FeatureSizeFeatureCollection

geowolf

On Fri, Jun 8, 2018 at 7:02 PM, Jim Hughes <[hidden email]> wrote:
Hi Andrea,

I tossed up a PR here: https://github.com/geoserver/geoserver/pull/2905 with a ticket https://osgeo-org.atlassian.net/browse/GEOS-8779.

I am not seeing a quick way to provide a unit test.  If you (or anyone) has a suggestion on that front, I'm happy to include one.

Cheers,

Jim


On 06/08/2018 12:49 PM, Andrea Aime wrote:
Hi Jim,
yes, properly implemented collection wrappers should allow delegation 

Cheers
Andrea

On Fri, Jun 8, 2018 at 6:47 PM, Jim Hughes <[hidden email]> wrote:
Andrea,

Thanks for helping the PR through. 

I haven't tested out the first case completely.  I've definitely run into the issue that Alvaro is trying to address with WFS 2.0. 

In the short term, is allowing the FC to delegate reasonable?

Cheers,

Jim


On 06/08/2018 12:07 PM, Andrea Aime wrote:
Ah, 
that change was controversial and the pull request sat there for ages before being merged:

(shows me as the author, but it's not really me, I'm just unwillingly involved because nobody else would take action)

Please discuss.

Cheers
Andrea


On Fri, Jun 8, 2018 at 5:57 PM, Jim Hughes <[hidden email]> wrote:
Hi all,

While testing out GeoMesa with GeoServer 2.13.x, we ran into some issues with the FeatureSizeFeatureCollection.  I think there are two issues:

First, for a layers backed by a large volume of data, getCount returning -1 is partially a way of saying that you don't want to count all the features;).  Given this pattern, I'd suggest that the FeatureSizeFeatureCollection might be taken off the usual path completely.  It looks like it might be an optimization when working with a bunch of little shapefiles.

Second, WPSes can be implemented with a visitor pattern.  For instance, the gs:Unique process leverages the GeoTools UniqueVisitor(1) which is recognized by the JDBC DataStores.  (In GeoMesa, we do a bunch of similar things.)

The fact that the current FeatureSizeFeatureCollection doesn't delegate prevents the UniqueProcess (and the GeoMesa ones) from working properly...

In the short term, any objection to a PR which enables delegation for that FeatureCollection?

Cheers,

Jim

1. UniqueVisitor: https://github.com/geotools/geotools/blob/master/modules/library/main/src/main/java/org/geotools/feature/visitor/UniqueVisitor.java
https://github.com/geotools/geotools/blob/master/modules/library/jdbc/src/main/java/org/geotools/jdbc/SQLDialect.java#L375

p.s.  I debugged this using PostGIS and the gs:Unique process. When the FeatureSizeFC was 'on', the queries did not have the 'distinct' keyword.  When the FC was off, GeoServer did make a 'distinct' query.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel




--

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.





--

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.





--

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Problems with the FeatureSizeFeatureCollection

Emilio Lahr-Vivaz
Thanks Jim, that addresses the main problem. However, I'm worried that the override of size() is going to cause performance issues. I was under the impression that returning -1 is a valid response to size(), and that seemed to be confirmed by Jody in the original PR (https://github.com/geoserver/geoserver/pull/1355#issuecomment-263688535). In GeoMesa, by default return we -1, as counts are expensive for us, but we allow overriding that behavior with a system property or a query hint. We have an optimized method to determine counts when needed, which is much faster than iterating over the entire feature collection, but that would be bypassed here.

Would it make sense to just set the default value for DEFAULT_CACHE_SIZE to 0? That way, it would be disabled by default, but could easily be enabled by setting the system property.

It seems long-term it would be nice to have a standardized way to request a quick count vs an accurate count.

Thanks,

Emilio

On 06/10/2018 02:23 AM, Andrea Aime wrote:

On Fri, Jun 8, 2018 at 7:02 PM, Jim Hughes <[hidden email]> wrote:
Hi Andrea,

I tossed up a PR here: https://github.com/geoserver/geoserver/pull/2905 with a ticket https://osgeo-org.atlassian.net/browse/GEOS-8779.

I am not seeing a quick way to provide a unit test.  If you (or anyone) has a suggestion on that front, I'm happy to include one.

Cheers,

Jim


On 06/08/2018 12:49 PM, Andrea Aime wrote:
Hi Jim,
yes, properly implemented collection wrappers should allow delegation 

Cheers
Andrea

On Fri, Jun 8, 2018 at 6:47 PM, Jim Hughes <[hidden email]> wrote:
Andrea,

Thanks for helping the PR through. 

I haven't tested out the first case completely.  I've definitely run into the issue that Alvaro is trying to address with WFS 2.0. 

In the short term, is allowing the FC to delegate reasonable?

Cheers,

Jim


On 06/08/2018 12:07 PM, Andrea Aime wrote:
Ah, 
that change was controversial and the pull request sat there for ages before being merged:

(shows me as the author, but it's not really me, I'm just unwillingly involved because nobody else would take action)

Please discuss.

Cheers
Andrea


On Fri, Jun 8, 2018 at 5:57 PM, Jim Hughes <[hidden email]> wrote:
Hi all,

While testing out GeoMesa with GeoServer 2.13.x, we ran into some issues with the FeatureSizeFeatureCollection.  I think there are two issues:

First, for a layers backed by a large volume of data, getCount returning -1 is partially a way of saying that you don't want to count all the features;).  Given this pattern, I'd suggest that the FeatureSizeFeatureCollection might be taken off the usual path completely.  It looks like it might be an optimization when working with a bunch of little shapefiles.

Second, WPSes can be implemented with a visitor pattern.  For instance, the gs:Unique process leverages the GeoTools UniqueVisitor(1) which is recognized by the JDBC DataStores.  (In GeoMesa, we do a bunch of similar things.)

The fact that the current FeatureSizeFeatureCollection doesn't delegate prevents the UniqueProcess (and the GeoMesa ones) from working properly...

In the short term, any objection to a PR which enables delegation for that FeatureCollection?

Cheers,

Jim

1. UniqueVisitor: https://github.com/geotools/geotools/blob/master/modules/library/main/src/main/java/org/geotools/feature/visitor/UniqueVisitor.java
https://github.com/geotools/geotools/blob/master/modules/library/jdbc/src/main/java/org/geotools/jdbc/SQLDialect.java#L375

p.s.  I debugged this using PostGIS and the gs:Unique process. When the FeatureSizeFC was 'on', the queries did not have the 'distinct' keyword.  When the FC was off, GeoServer did make a 'distinct' query.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel




--

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.





--

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.





--

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Problems with the FeatureSizeFeatureCollection

Jim Hughes-2
In reply to this post by geowolf
Hi Andrea,

Thanks for the pointer.  I've updated my PR with a unit test.  I worked to make it a bit more general so that additional FeatureCollection + Visitor combinations could be tested in one place.

Cheers,

Jim


On 06/10/2018 02:23 AM, Andrea Aime wrote:

On Fri, Jun 8, 2018 at 7:02 PM, Jim Hughes <[hidden email]> wrote:
Hi Andrea,

I tossed up a PR here: https://github.com/geoserver/geoserver/pull/2905 with a ticket https://osgeo-org.atlassian.net/browse/GEOS-8779.

I am not seeing a quick way to provide a unit test.  If you (or anyone) has a suggestion on that front, I'm happy to include one.

Cheers,

Jim


On 06/08/2018 12:49 PM, Andrea Aime wrote:
Hi Jim,
yes, properly implemented collection wrappers should allow delegation 

Cheers
Andrea

On Fri, Jun 8, 2018 at 6:47 PM, Jim Hughes <[hidden email]> wrote:
Andrea,

Thanks for helping the PR through. 

I haven't tested out the first case completely.  I've definitely run into the issue that Alvaro is trying to address with WFS 2.0. 

In the short term, is allowing the FC to delegate reasonable?

Cheers,

Jim


On 06/08/2018 12:07 PM, Andrea Aime wrote:
Ah, 
that change was controversial and the pull request sat there for ages before being merged:

(shows me as the author, but it's not really me, I'm just unwillingly involved because nobody else would take action)

Please discuss.

Cheers
Andrea


On Fri, Jun 8, 2018 at 5:57 PM, Jim Hughes <[hidden email]> wrote:
Hi all,

While testing out GeoMesa with GeoServer 2.13.x, we ran into some issues with the FeatureSizeFeatureCollection.  I think there are two issues:

First, for a layers backed by a large volume of data, getCount returning -1 is partially a way of saying that you don't want to count all the features;).  Given this pattern, I'd suggest that the FeatureSizeFeatureCollection might be taken off the usual path completely.  It looks like it might be an optimization when working with a bunch of little shapefiles.

Second, WPSes can be implemented with a visitor pattern.  For instance, the gs:Unique process leverages the GeoTools UniqueVisitor(1) which is recognized by the JDBC DataStores.  (In GeoMesa, we do a bunch of similar things.)

The fact that the current FeatureSizeFeatureCollection doesn't delegate prevents the UniqueProcess (and the GeoMesa ones) from working properly...

In the short term, any objection to a PR which enables delegation for that FeatureCollection?

Cheers,

Jim

1. UniqueVisitor: https://github.com/geotools/geotools/blob/master/modules/library/main/src/main/java/org/geotools/feature/visitor/UniqueVisitor.java
https://github.com/geotools/geotools/blob/master/modules/library/jdbc/src/main/java/org/geotools/jdbc/SQLDialect.java#L375

p.s.  I debugged this using PostGIS and the gs:Unique process. When the FeatureSizeFC was 'on', the queries did not have the 'distinct' keyword.  When the FC was off, GeoServer did make a 'distinct' query.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel




--

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.





--

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.





--

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel