[gdal-dev] Understanding VSI caching

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Understanding VSI caching

geowolf
Hi,
is there any document explaning how VSI caching works. E.g., memory vs disk caching, whether the disk cache
is long lived or exists only for the duration of the process (and if so, what about long lived processes
like mapserver fastcgi?) and so on?

Cheers
Andrea

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility  for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.



_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Understanding VSI caching

Even Rouault-2
On lundi 9 avril 2018 18:33:32 CEST Andrea Aime wrote:
> Hi,
> is there any document explaning how VSI caching works. E.g., memory vs disk
> caching, whether the disk cache
> is long lived or exists only for the duration of the process (and if so,
> what about long lived processes
> like mapserver fastcgi?) and so on?

Hi Andrea,

Are you speaking about the cache machanism enabled with the VSI_CACHE=YES env variable ?
If so that's described at
http://gdal.org/gdal_virtual_file_systems.html#gdal_virtual_file_systems_vsicache
I see it lacks a bit high level details, like specifying this is RAM only, and not disk cached.
I'm going to add some words about that

Otherwise, the full "documentation" :-) is at:
https://github.com/OSGeo/gdal/blob/master/gdal/port/cpl_vsil_cache.cpp

The /vsicurl/ and related network file systems have also their own RAM cache mechanism
described at
http://gdal.org/gdal_virtual_file_systems.html#gdal_virtual_file_systems_vsicurl

Even

>
> Cheers
> Andrea
>
> ==
>
> GeoServer Professional Services from the experts! Visit http://goo.gl/it488V
> for more information.
> ==
>
> Ing. Andrea Aime
> @geowolf
> Technical Lead
>
> GeoSolutions S.A.S.
> Via di Montramito 3/A
> 55054  Massarosa (LU)
> phone: +39 0584 962313
> fax: +39 0584 1660272
> mob: +39  339 8844549
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
>
> Le informazioni contenute in questo messaggio di posta elettronica e/o
> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
> darcene notizia via e-mail e di procedere alla distruzione del messaggio
> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
> principi dettati dal D.Lgs. 196/2003.
>
> The information in this message and/or attachments, is intended solely for
> the attention and use of the named addressee(s) and may be confidential or
> proprietary in nature or covered by the provisions of privacy act
> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
> Code).Any use not in accord with its purpose, any disclosure, reproduction,
> copying, distribution, or either dissemination, either whole or partial, is
> strictly forbidden except previous formal approval of the named
> addressee(s). If you are not the intended recipient, please contact
> immediately the sender by telephone, fax or e-mail and delete the
> information in this message that has been received in error. The sender
> does not give any warranty or accept liability as the content, accuracy or
> completeness of sent messages and accepts no responsibility  for changes
> made after they were sent or for other risks which arise as a result of
> e-mail transmission, viruses, etc.


--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Understanding VSI caching

geowolf
On Mon, Apr 9, 2018 at 6:43 PM, Even Rouault <[hidden email]> wrote:
Hi Andrea,

Are you speaking about the cache machanism enabled with the VSI_CACHE=YES env variable ?

I read there was some caching, but was not that specific. So, memory caching is there, it's per file, 
but not enabled by default, while disk caching is there, global, and enabled by default?
 
If so that's described at
http://gdal.org/gdal_virtual_file_systems.html#gdal_virtual_file_systems_vsicache
I see it lacks a bit high level details, like specifying this is RAM only, and not disk cached.
I'm going to add some words about that

Otherwise, the full "documentation" :-) is at:
https://github.com/OSGeo/gdal/blob/master/gdal/port/cpl_vsil_cache.cpp

Hehehe :-)
 
The /vsicurl/ and related network file systems have also their own RAM cache mechanism
described at
http://gdal.org/gdal_virtual_file_systems.html#gdal_virtual_file_systems_vsicurl

The "detect sequential reading" part is interesting, if I'm not mistaken the logic is here,
right?


Thanks a lot for the useful information, as usual :-)

Cheers
Andrea
 

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility  for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.



_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Understanding VSI caching

Even Rouault-2
> I read there was some caching, but was not that specific. So, memory
> caching is there, it's per file,
> but not enabled by default,

Yes

> while disk caching is there, global, and
> enabled by default?

Which disk caching are you refering too ? There's some disk caching for the
WMS driver, but that's all

(well there's some provisional code for disk caching in /vsicurl/ but it is
not compiled by default, and probably broken. There was some discussion
recently on the ML about this if I remember well)

>
> The "detect sequential reading" part is interesting, if I'm not mistaken
> the logic is here,
> right?
>
> https://github.com/OSGeo/gdal/blob/master/gdal/port/cpl_vsil_curl.cpp#L1845

Yes

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Understanding VSI caching

geowolf
On Mon, Apr 9, 2018 at 8:58 PM, Even Rouault <[hidden email]> wrote:
Which disk caching are you refering too ? There's some disk caching for the
WMS driver, but that's all


"In addition, a global least-recently-used cache of 16 MB shared among all downloaded content is enabled by default, and content in it may be reused after a file handle has been closed and reopen. Starting with GDAL 2.3, the size of this global LRU cache can be modified by setting the configuration option CPL_VSIL_CURL_CACHE_SIZE (in bytes)."

and I see in the code:


(btw, there is no path, does it mean the file is created in the current directory?)

 
(well there's some provisional code for disk caching in /vsicurl/ but it is
not compiled by default, and probably broken. There was some discussion
recently on the ML about this if I remember well)

I'll check the archives.

Cheers
Andrea

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility  for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.



_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Understanding VSI caching

Even Rouault-2
On lundi 9 avril 2018 21:03:12 CEST Andrea Aime wrote:

> On Mon, Apr 9, 2018 at 8:58 PM, Even Rouault <[hidden email]>
>
> wrote:
> > Which disk caching are you refering too ? There's some disk caching for
> > the
> > WMS driver, but that's all
>
> "In addition, a global least-recently-used cache of 16 MB shared among all
> downloaded content is enabled by default, and content in it may be reused
> after a file handle has been closed and reopen. Starting with GDAL 2.3, the
> size of this global LRU cache can be modified by setting the configuration
> option CPL_VSIL_CURL_CACHE_SIZE (in bytes)."

Ah, it probably means a precision "during the life-time of the process or
until VSICurlClearCache() is called" (just added). It's just a RAM cache.

>
> and I see in the code:
>
> https://github.com/OSGeo/gdal/blob/master/gdal/port/cpl_vsil_curl.cpp#L203
>
> (btw, there is no path, does it mean the file is created in the current
> directory?)

Yes, that was the more or less dead code I was mentionning too. Actually it is
compiled in, and can be enabled with CPL_VSIL_CURL_USE_CACHE=YES, but it is
very likely broken, unfit for any reasonable purpose etc. No warranty. Consult
your lawyer, be sure you have a fallout shelter ready, etc ;-)

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev