[gdal-dev] vrt and too many open files error

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] vrt and too many open files error

Julien Michel-2
Dear all,

I have a VRT file, which I generated by a script I wrote, that simply
mosaics 2750 tiles (images of the same size). I can not read back this
vrt file with gdal, it says:

$ gdalinfo cloud_plain.vrt

ERROR 4: Too many open files
gdalinfo failed - unable to open 'cloud_plain.vrt'.

Reading this tracker [1], I think it should not complain ? Note that
this happens when parsing the vrt file, not actually reading data. If I
gdal_translate only a small portion spanning a few tiles, it will still
complain. Shall I file a bug report or is this behavior expected ?

Regards,

Julien


[1] https://trac.osgeo.org/gdal/ticket/6604



--
Julien MICHEL
CNES - DSO/SI/2A

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: vrt and too many open files error

Even Rouault-2

On vendredi 6 octobre 2017 17:29:40 CEST Julien Michel wrote:

> Dear all,

>

> I have a VRT file, which I generated by a script I wrote, that simply

> mosaics 2750 tiles (images of the same size). I can not read back this

> vrt file with gdal, it says:

>

> $ gdalinfo cloud_plain.vrt

>

> ERROR 4: Too many open files

> gdalinfo failed - unable to open 'cloud_plain.vrt'.

>

> Reading this tracker [1], I think it should not complain ? Note that

> this happens when parsing the vrt file, not actually reading data. If I

> gdal_translate only a small portion spanning a few tiles, it will still

> complain. Shall I file a bug report or is this behavior expected ?

 

Julien,

 

it is likely that if you built it at hand that your VRT sources lack a SourceProperties element

 

<SourceProperties RasterXSize="xxx" RasterYSize="xxx" DataType="xxx" BlockXSize="xxx" BlockYSize="xxx" />

 

Without that element, GDAL must open each file, and you hit the 1024 file descriptor limit

 

~~~

 

That said, in theory the VRT driver could be improved to avoid having that element at all (and it would be desirable not having it). It is just needed so we can instanciate a ProxyDataset object since it needs to know those characteristics, but the VRT logic itself shouldn't need them until it identifies that it must read from the source. At the time where I improved that point in GDAL, I was still a bit unfamiliar with the code base and didn't dare touching that aspect ;-)

 

 

Even

 

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: vrt and too many open files error

Julien Michel-2
Thanks Even, it worked.

Regards,

Julien

Le 06/10/2017 à 17:55, Even Rouault a écrit :

>
> On vendredi 6 octobre 2017 17:29:40 CEST Julien Michel wrote:
>
> > Dear all,
>
> >
>
> > I have a VRT file, which I generated by a script I wrote, that simply
>
> > mosaics 2750 tiles (images of the same size). I can not read back this
>
> > vrt file with gdal, it says:
>
> >
>
> > $ gdalinfo cloud_plain.vrt
>
> >
>
> > ERROR 4: Too many open files
>
> > gdalinfo failed - unable to open 'cloud_plain.vrt'.
>
> >
>
> > Reading this tracker [1], I think it should not complain ? Note that
>
> > this happens when parsing the vrt file, not actually reading data. If I
>
> > gdal_translate only a small portion spanning a few tiles, it will still
>
> > complain. Shall I file a bug report or is this behavior expected ?
>
> Julien,
>
> it is likely that if you built it at hand that your VRT sources lack a
> SourceProperties element
>
> <SourceProperties RasterXSize="xxx" RasterYSize="xxx" DataType="xxx"
> BlockXSize="xxx" BlockYSize="xxx" />
>
> Without that element, GDAL must open each file, and you hit the 1024
> file descriptor limit
>
> ~~~
>
> That said, in theory the VRT driver could be improved to avoid having
> that element at all (and it would be desirable not having it). It is
> just needed so we can instanciate a ProxyDataset object since it needs
> to know those characteristics, but the VRT logic itself shouldn't need
> them until it identifies that it must read from the source. At the
> time where I improved that point in GDAL, I was still a bit unfamiliar
> with the code base and didn't dare touching that aspect ;-)
>
> Even
>
> --
>
> Spatialys - Geospatial professional services
>
> http://www.spatialys.com
>

--
Julien MICHEL
CNES - DSO/SI/2A - BPI 1219
18, avenue Edouard Belin
31401 Toulouse Cedex 09 - France
Tel: +33 561 282 894 - Fax: +33 561 283 109

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev