[gdal-dev] Check of sibling files when opening an image file

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Check of sibling files when opening an image file

Armin Burger
Hi all

When gdal opens a file, it checks for various sibling files, like *.MTL,
*.aux.xml, *.RPC, etc (this is the case when the full directory scanning
is deactivated). This can be ~30 stat or open requests when opening a
single image file. Is there a way to define the file types that gdal
checks for their existence, thus allowing to reduce this to the pure
minimum, like *.ovr files only? Is this check driver dependent?

Thanks in advance for any hints
Armin

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Check of sibling files when opening an image file

Even Rouault-2
On jeudi 21 mars 2019 18:34:11 CET Armin Burger wrote:
> Hi all
>
> When gdal opens a file, it checks for various sibling files, like *.MTL,
> *.aux.xml, *.RPC, etc (this is the case when the full directory scanning
> is deactivated). This can be ~30 stat or open requests when opening a
> single image file. Is there a way to define the file types that gdal
> checks for their existence, thus allowing to reduce this to the pure
> minimum, like *.ovr files only? Is this check driver dependent?

If using /vsicurl/ and derived file systems, you can use
CPL_VSIL_CURL_ALLOWED_EXTENSIONS=".tif,.ovr" for example

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Check of sibling files when opening an image file

Armin Burger


On 21/03/2019 19:09, Even Rouault wrote:

> On jeudi 21 mars 2019 18:34:11 CET Armin Burger wrote:
>> Hi all
>>
>> When gdal opens a file, it checks for various sibling files, like *.MTL,
>> *.aux.xml, *.RPC, etc (this is the case when the full directory scanning
>> is deactivated). This can be ~30 stat or open requests when opening a
>> single image file. Is there a way to define the file types that gdal
>> checks for their existence, thus allowing to reduce this to the pure
>> minimum, like *.ovr files only? Is this check driver dependent?
>
> If using /vsicurl/ and derived file systems, you can use
> CPL_VSIL_CURL_ALLOWED_EXTENSIONS=".tif,.ovr" for example
>
> Even
>

Hi Even

Thanks, but we're using a POSIX file system (mounted via FUSE client),
so I assume this will not work then.

It would also be fine for us to reduce the number of extensions at
compile time by modifying configuration or source code files. But so far
I haven't found any appearances in the source code files for file
extensions that are obviously by default checked for their existence,
like e.g. "RPC.TXT". So any hints if/how this is possible would be great.

Cheers
Armin
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Check of sibling files when opening an image file

Armin Burger
Just in case someone is interested in a solution for this issue:

The most efficient setting was to set the gdal option
   GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR

This can reduce the file stat/open calls noticeably, depending on the
raster format. The worst format I found was JP2, which is now
unfortunately more often used like e.g. in Sentinel-2 data. For a JP2
image, a gdal Open() request for a single JP2 file makes 46 file
stat/open calls... With the above setting this is reduced to 6. In order
to set this as default, one needs to change this in the source code in
file gcore/gdalopeninfo.cpp.

If in addition the option GDAL_PAM_ENABLED is set to NO, then another 5
stat/open calls are eliminated. Here this setting can be put as default
at compile time by adding to the "configure" script the option
   --without-pam

Cheers
Armin




On 21/03/2019 19:54, Armin Burger wrote:

>
>
> On 21/03/2019 19:09, Even Rouault wrote:
>> On jeudi 21 mars 2019 18:34:11 CET Armin Burger wrote:
>>> Hi all
>>>
>>> When gdal opens a file, it checks for various sibling files, like *.MTL,
>>> *.aux.xml, *.RPC, etc (this is the case when the full directory scanning
>>> is deactivated). This can be ~30 stat or open requests when opening a
>>> single image file. Is there a way to define the file types that gdal
>>> checks for their existence, thus allowing to reduce this to the pure
>>> minimum, like *.ovr files only? Is this check driver dependent?
>>
>> If using /vsicurl/ and derived file systems, you can use
>> CPL_VSIL_CURL_ALLOWED_EXTENSIONS=".tif,.ovr" for example
>>
>> Even
>>
>
> Hi Even
>
> Thanks, but we're using a POSIX file system (mounted via FUSE client),
> so I assume this will not work then.
>
> It would also be fine for us to reduce the number of extensions at
> compile time by modifying configuration or source code files. But so far
> I haven't found any appearances in the source code files for file
> extensions that are obviously by default checked for their existence,
> like e.g. "RPC.TXT". So any hints if/how this is possible would be great.
>
> Cheers
> Armin
> _______________________________________________
> gdal-dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev