[gdal-dev] Interesting debugging log from FileGBD driver

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Interesting debugging log from FileGBD driver

geowolf
Hi,
I'm looking at a performance issue rendering a very large ESRI File geodatabase, and in the logs I see:

CPLQuadTree: Estimated spatial index tree depth: 20
CPLQuadTree: Falling back to max number of allowed index tree levels (12).

Wondering if this might be a reason for slow data extraction, and if there are settings to make it us the full
quadtree depth? The environment is also a multi-threaded (multiple threads reading from the file at the same time), this could also be a cause.

The file in question is massive, so OpenFileGDB does not seem like an option

Cheers
Andrea

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Interesting debugging log from FileGBD driver

Even Rouault-2
Hi Andrea,

> I'm looking at a performance issue rendering a very large ESRI File
> geodatabase, and in the logs I see:
>
> CPLQuadTree: Estimated spatial index tree depth: 20
> CPLQuadTree: Falling back to max number of allowed index tree levels (12).
>
> Wondering if this might be a reason for slow data extraction,

I wouldn't think so. The maximum depth of 12 should be large enough.

> and if there
> are settings to make it us the full
> quadtree depth? The environment is also a multi-threaded (multiple threads
> reading from the file at the same time), this could also be a cause.

What is slow exactly ? The first time you do a spatial filtering (which is
expected, since it requires to do a full scan of the file, because the driver
doesn't know how to use the on-disk spatial index), or later spatial filtering
operations with the connection still opened (which should be fast normally) ?

> The file in question is massive, so OpenFileGDB does not seem like an option

Yep, still hoping that someone will manage to reverse engineer those last
missing bits of the format.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Interesting debugging log from FileGBD driver

geowolf
On Tue, Apr 30, 2019 at 3:22 PM Even Rouault <[hidden email]> wrote:
What is slow exactly ? The first time you do a spatial filtering (which is
expected, since it requires to do a full scan of the file, because the driver
doesn't know how to use the on-disk spatial index), or later spatial filtering
operations with the connection still opened (which should be fast normally) ?

Oh wait! You mean those messages pop up only if OpenFileGDB is used?
Dang, thought I had forced usage of FileGBD but I did not, very sorry for the noise.

The log messages are gone, performance is still really bad. Can I blame multithreading for that?
All the datasource objects are opened in read only mode

Cheers
Andrea

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Interesting debugging log from FileGBD driver

Even Rouault-2
On mardi 30 avril 2019 15:33:22 CEST Andrea Aime wrote:

> On Tue, Apr 30, 2019 at 3:22 PM Even Rouault <[hidden email]>
>
> wrote:
> > What is slow exactly ? The first time you do a spatial filtering (which is
> > expected, since it requires to do a full scan of the file, because the
> > driver
> > doesn't know how to use the on-disk spatial index), or later spatial
> > filtering
> > operations with the connection still opened (which should be fast
> > normally) ?
>
> Oh wait! You mean those messages pop up only if OpenFileGDB is used?
> Dang, thought I had forced usage of FileGBD but I did not, very sorry for
> the noise.
>
> The log messages are gone, performance is still really bad. Can I blame
> multithreading for that?
> All the datasource objects are opened in read only mode

Ah, the FileGDB SDK used by the FileGDB driver was/is not thread-safe, so the
FileGDB driver holds a global lock...

Even


--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Interesting debugging log from FileGBD driver

geowolf
On Tue, Apr 30, 2019 at 3:41 PM Even Rouault <[hidden email]> wrote:
Ah, the FileGDB SDK used by the FileGDB driver was/is not thread-safe, so the
FileGDB driver holds a global lock...

Yes, what I expected.... thanks for the swift confirmation!

Cheers
Andrea

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev