Quantcast

[gdal-dev] Help requested: Concurrent read of a GeoTiff

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[gdal-dev] Help requested: Concurrent read of a GeoTiff

Grégory Bataille
Hello all,

Reaching out to the community caused I have failed for the past few days.

short version, I'm trying to multithread the gdal2tiles utility, and I'm getting this

Generating Base Tiles:
ERROR 1: LZWDecode:Wrong length of decoded string: data probably corrupted at scanline 256
ERROR 1: TIFFReadEncodedTile() failed.
ERROR 1: /Users/gbataille/Downloads/Project_58704_transparent_mosaic_group1.tif, band 1: IReadBlock failed at X offset 1, Y offset 0
ERROR 1: GetBlockRef failed at X block offset 1, Y block offset 0
ERROR 1: gba.vrt, band 1: IReadBlock failed at X offset 0, Y offset 0

Any idea?

Long version
So I'm trying to add multithreading to gdal2tiles. I know it's possible, others like https://gitlab.com/GitLabRGI/geopackage-python/blob/master/Packaging/tiles2gpkg_parallel.py have done it.
I wanted a slightly different approach where I compute all the tile coordinates to generate, then I create the tiles one by one in multiple processes

- I can't see anything that I'm doing very differently from the above project
- I have confirmed that if I run my code with a pool of one process (still a sub process, but a single one) then it works as expected
- It fails only when I use more than 1 sub-process, i.e. I guess it's some concurrent file access that fails.

The trouble is that others have done it, so there is a way :)

Have you seen this error? Does it ring any bells that could point me in the right direction?

Thanks

---
Gregory Bataille

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help requested: Concurrent read of a GeoTiff

Even Rouault-2

On jeudi 16 mars 2017 17:16:20 CET Grégory Bataille wrote:

> Hello all,

>

> Reaching out to the community caused I have failed for the past few days.

>

> *short version*, I'm trying to multithread the gdal2tiles utility, and I'm

> getting this

>

> Generating Base Tiles:

> ERROR 1: LZWDecode:Wrong length of decoded string: data probably corrupted

> at scanline 256

> ERROR 1: TIFFReadEncodedTile() failed.

> ERROR 1:

> /Users/gbataille/Downloads/Project_58704_transparent_mosaic_group1.tif,

> band 1: IReadBlock failed at X offset 1, Y offset 0

> ERROR 1: GetBlockRef failed at X block offset 1, Y block offset 0

> ERROR 1: gba.vrt, band 1: IReadBlock failed at X offset 0, Y offset 0

>

> Any idea?

 

Yes, you need one dataset object per thread . Dataset objects cannot be used simultaneously from several threads.

 

Even

 

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help requested: Concurrent read of a GeoTiff

Grégory Bataille
Does that mean a different vrt file or simply reopen the file to create a gdal object in each process. Because I'm reopening the vrt file in each thread


On Thu, 16 Mar 2017 at 17:23, Even Rouault <[hidden email]> wrote:

On jeudi 16 mars 2017 17:16:20 CET Grégory Bataille wrote:

> Hello all,

>

> Reaching out to the community caused I have failed for the past few days.

>

> *short version*, I'm trying to multithread the gdal2tiles utility, and I'm

> getting this

>

> Generating Base Tiles:

> ERROR 1: LZWDecode:Wrong length of decoded string: data probably corrupted

> at scanline 256

> ERROR 1: TIFFReadEncodedTile() failed.

> ERROR 1:

> /Users/gbataille/Downloads/Project_58704_transparent_mosaic_group1.tif,

> band 1: IReadBlock failed at X offset 1, Y offset 0

> ERROR 1: GetBlockRef failed at X block offset 1, Y block offset 0

> ERROR 1: gba.vrt, band 1: IReadBlock failed at X offset 0, Y offset 0

>

> Any idea?

 

Yes, you need one dataset object per thread . Dataset objects cannot be used simultaneously from several threads.

 

Even

 

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com

--
Greg

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help requested: Concurrent read of a GeoTiff

Even Rouault-2

On jeudi 16 mars 2017 18:45:03 CET Grégory Bataille wrote:

> Does that mean a different vrt file or simply reopen the file to create a

> gdal object in each process. Because I'm reopening the vrt file in each

> thread

 

Ah, that must be issue of

http://gdal.org/gdal_vrttut.html#gdal_vrttut_mt

 

Try defining

gdal.SetConfigOption('VRT_SHARED_SOURCE', '0')

before opening the VRTs

 

>

>

> On Thu, 16 Mar 2017 at 17:23, Even Rouault <[hidden email]>

>

> wrote:

> > On jeudi 16 mars 2017 17:16:20 CET Grégory Bataille wrote:

> > > Hello all,

> > >

> > >

> > >

> > > Reaching out to the community caused I have failed for the past few

> > > days.

> > >

> > >

> > >

> > > *short version*, I'm trying to multithread the gdal2tiles utility, and

> >

> > I'm

> >

> > > getting this

> > >

> > >

> > >

> > > Generating Base Tiles:

> > >

> > > ERROR 1: LZWDecode:Wrong length of decoded string: data probably

> >

> > corrupted

> >

> > > at scanline 256

> > >

> > > ERROR 1: TIFFReadEncodedTile() failed.

> > >

> > > ERROR 1:

> > >

> > > /Users/gbataille/Downloads/Project_58704_transparent_mosaic_group1.tif,

> > >

> > > band 1: IReadBlock failed at X offset 1, Y offset 0

> > >

> > > ERROR 1: GetBlockRef failed at X block offset 1, Y block offset 0

> > >

> > > ERROR 1: gba.vrt, band 1: IReadBlock failed at X offset 0, Y offset 0

> > >

> > >

> > >

> > > Any idea?

> >

> > Yes, you need one dataset object per thread . Dataset objects cannot be

> > used simultaneously from several threads.

> >

> >

> >

> > Even

> >

> >

> >

> >

> >

> > --

> >

> > Spatialys - Geospatial professional services

> >

> > http://www.spatialys.com

 

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help requested: Concurrent read of a GeoTiff

Grégory Bataille
no luck.
I tried this config.
I also tried (with this config) to do a copy of the vrt file (with shell utility, not gdal) in each thread before opening it (still goes to the same TIFF file though), but no luck.

But at least now that you confirm that it's related to concurrent reads, I have something precise to search for. I'll try to investigate how the gdal2tiles_parallel does it because it did not look different than what I was doing, but likely I missed something.

Thanks


---
Gregory Bataille

On Thu, Mar 16, 2017 at 7:58 PM, Even Rouault <[hidden email]> wrote:

On jeudi 16 mars 2017 18:45:03 CET Grégory Bataille wrote:

> Does that mean a different vrt file or simply reopen the file to create a

> gdal object in each process. Because I'm reopening the vrt file in each

> thread

 

Ah, that must be issue of

http://gdal.org/gdal_vrttut.html#gdal_vrttut_mt

 

Try defining

gdal.SetConfigOption('VRT_SHARED_SOURCE', '0')

before opening the VRTs

 

>

>

> On Thu, 16 Mar 2017 at 17:23, Even Rouault <[hidden email]>

>

> wrote:

> > On jeudi 16 mars 2017 17:16:20 CET Grégory Bataille wrote:

> > > Hello all,

> > >

> > >

> > >

> > > Reaching out to the community caused I have failed for the past few

> > > days.

> > >

> > >

> > >

> > > *short version*, I'm trying to multithread the gdal2tiles utility, and

> >

> > I'm

> >

> > > getting this

> > >

> > >

> > >

> > > Generating Base Tiles:

> > >

> > > ERROR 1: LZWDecode:Wrong length of decoded string: data probably

> >

> > corrupted

> >

> > > at scanline 256

> > >

> > > ERROR 1: TIFFReadEncodedTile() failed.

> > >

> > > ERROR 1:

> > >

> > > /Users/gbataille/Downloads/Project_58704_transparent_mosaic_group1.tif,

> > >

> > > band 1: IReadBlock failed at X offset 1, Y offset 0

> > >

> > > ERROR 1: GetBlockRef failed at X block offset 1, Y block offset 0

> > >

> > > ERROR 1: gba.vrt, band 1: IReadBlock failed at X offset 0, Y offset 0

> > >

> > >

> > >

> > > Any idea?

> >

> > Yes, you need one dataset object per thread . Dataset objects cannot be

> > used simultaneously from several threads.

> >

> >

> >

> > Even

> >

> >

> >

> >

> >

> > --

> >

> > Spatialys - Geospatial professional services

> >

> > http://www.spatialys.com

 

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com



_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help requested: Concurrent read of a GeoTiff

Grégory Bataille
Hey Even,

reaching out again, because I just don't know...

so here is gdal2tiles_parallel strategy (as far as I understand it)
- in each process, open the input file, compute the autowarped vrt (in the case of the dataset I use at least) loop over tiles to generate and work only depending on the process number (so that this work is made in parallel). We therefore end up with, in each process
    - the input file opened as a dataset (but not touched after the autowarped vrt is generated)
    - a different autowarped vrt in memory (for each process) but pointing to the same source (as far as I can see)


My strategy:
- open the input file.
- generate the autowarped vrt and save it to disk
- compute all the tile details and store them in a data structure.
Then in each process:
- take one tile detail
- open the vrt (same file for each process, pointing to the same source TIFF)
- read the vrt and write the tile
And that's the read that is failing as mentioned in my first email.

I can't see much difference. To try and be complete, I tried to (or think I tried to):
- open the input file in each process
- generate a different vrt file for each thread

I even copy/pasted the code that gdal2tiles_parallel uses in their processes, but with no success, always the same error. The VRT_SHARED_SOURCE option does not seem to change anything (but looking at the wiki, it's not clear whether it applies to autowarped vrt too)

If you have any idea, I'll take it, I have been stuck for a long time without making any progress :(

Cheers



---
Gregory Bataille

On Fri, Mar 17, 2017 at 6:15 AM, Grégory Bataille <[hidden email]> wrote:
no luck.
I tried this config.
I also tried (with this config) to do a copy of the vrt file (with shell utility, not gdal) in each thread before opening it (still goes to the same TIFF file though), but no luck.

But at least now that you confirm that it's related to concurrent reads, I have something precise to search for. I'll try to investigate how the gdal2tiles_parallel does it because it did not look different than what I was doing, but likely I missed something.

Thanks


---
Gregory Bataille

On Thu, Mar 16, 2017 at 7:58 PM, Even Rouault <[hidden email]> wrote:

On jeudi 16 mars 2017 18:45:03 CET Grégory Bataille wrote:

> Does that mean a different vrt file or simply reopen the file to create a

> gdal object in each process. Because I'm reopening the vrt file in each

> thread

 

Ah, that must be issue of

http://gdal.org/gdal_vrttut.html#gdal_vrttut_mt

 

Try defining

gdal.SetConfigOption('VRT_SHARED_SOURCE', '0')

before opening the VRTs

 

>

>

> On Thu, 16 Mar 2017 at 17:23, Even Rouault <[hidden email]>

>

> wrote:

> > On jeudi 16 mars 2017 17:16:20 CET Grégory Bataille wrote:

> > > Hello all,

> > >

> > >

> > >

> > > Reaching out to the community caused I have failed for the past few

> > > days.

> > >

> > >

> > >

> > > *short version*, I'm trying to multithread the gdal2tiles utility, and

> >

> > I'm

> >

> > > getting this

> > >

> > >

> > >

> > > Generating Base Tiles:

> > >

> > > ERROR 1: LZWDecode:Wrong length of decoded string: data probably

> >

> > corrupted

> >

> > > at scanline 256

> > >

> > > ERROR 1: TIFFReadEncodedTile() failed.

> > >

> > > ERROR 1:

> > >

> > > /Users/gbataille/Downloads/Project_58704_transparent_mosaic_group1.tif,

> > >

> > > band 1: IReadBlock failed at X offset 1, Y offset 0

> > >

> > > ERROR 1: GetBlockRef failed at X block offset 1, Y block offset 0

> > >

> > > ERROR 1: gba.vrt, band 1: IReadBlock failed at X offset 0, Y offset 0

> > >

> > >

> > >

> > > Any idea?

> >

> > Yes, you need one dataset object per thread . Dataset objects cannot be

> > used simultaneously from several threads.

> >

> >

> >

> > Even

> >

> >

> >

> >

> >

> > --

> >

> > Spatialys - Geospatial professional services

> >

> > http://www.spatialys.com

 

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com




_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help requested: Concurrent read of a GeoTiff

Even Rouault-2

On mardi 21 mars 2017 05:51:18 CET Grégory Bataille wrote:

> Hey Even,

>

> reaching out again, because I just don't know...

>

> so here is gdal2tiles_parallel strategy (as far as I understand it)

> - in each process, open the input file, compute the autowarped vrt (in the

> case of the dataset I use at least) loop over tiles to generate and work

> only depending on the process number (so that this work is made in

> parallel). We therefore end up with, in each process

> - the input file opened as a dataset (but not touched after the

> autowarped vrt is generated)

> - a different autowarped vrt in memory (for each process) but pointing

> to the same source (as far as I can see)

>

>

> My strategy:

> - open the input file.

> - generate the autowarped vrt and save it to disk

> - compute all the tile details and store them in a data structure.

> Then in each process:

 

I'm confused. You're talking about process, but is it a real operating system process or a thread ? I guess the later.

 

> - take one tile detail

> - open the vrt (same file for each process, pointing to the same source

> TIFF)

> - read the vrt and write the tile

> And that's the read that is failing as mentioned in my first email.

>

> I can't see much difference. To try and be complete, I tried to (or think I

> tried to):

> - open the input file in each process

> - generate a different vrt file for each thread

>

> I even copy/pasted the code that gdal2tiles_parallel uses in their

> processes, but with no success, always the same error. The

> VRT_SHARED_SOURCE option

> does not seem to change anything (but looking at the wiki, it's not clear

> whether it applies to autowarped vrt too)

 

VRT_SHARED_SOURCE indeed only works for "regular" mosaicing VRTs. So not to warped VRT.

 

Warped VRT opens the source dataset as a shared dataset in

https://github.com/OSGeo/gdal/blob/trunk/gdal/alg/gdalwarper.cpp#L1675

So you could have an issue if you open the VRT twice in the same thread, and then use each handle in a different thread, since both VRT handles would point to the same shared dataset. But it doesn't look that's what you are doing.

 

As a workaround to investigate, you could try to remove the GDAL_OF_SHARED | flag in the line I mentionned and see if that makes a difference (but we could potentially - not sure - have issues at closing time since different call sites might make an assumption on the dataset being opened with shared semantics)

 

What you could do too is run the script through your prefered debugger and set a breakpoint in GDALOpenEx and look at the calling sites. If you are using 2 threads, then if we want things to work correctly, GDALOpenEx() should be called twice on the TIFF dataset and return a different handle.

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help requested: Concurrent read of a GeoTiff

Grégory Bataille
hum... debugging in the C layers, that'll be interesting...
Ok, I'll continue to dig in, thanks


---
Gregory Bataille

On Tue, Mar 21, 2017 at 8:44 AM, Even Rouault <[hidden email]> wrote:

On mardi 21 mars 2017 05:51:18 CET Grégory Bataille wrote:

> Hey Even,

>

> reaching out again, because I just don't know...

>

> so here is gdal2tiles_parallel strategy (as far as I understand it)

> - in each process, open the input file, compute the autowarped vrt (in the

> case of the dataset I use at least) loop over tiles to generate and work

> only depending on the process number (so that this work is made in

> parallel). We therefore end up with, in each process

> - the input file opened as a dataset (but not touched after the

> autowarped vrt is generated)

> - a different autowarped vrt in memory (for each process) but pointing

> to the same source (as far as I can see)

>

>

> My strategy:

> - open the input file.

> - generate the autowarped vrt and save it to disk

> - compute all the tile details and store them in a data structure.

> Then in each process:

 

I'm confused. You're talking about process, but is it a real operating system process or a thread ? I guess the later.

 

> - take one tile detail

> - open the vrt (same file for each process, pointing to the same source

> TIFF)

> - read the vrt and write the tile

> And that's the read that is failing as mentioned in my first email.

>

> I can't see much difference. To try and be complete, I tried to (or think I

> tried to):

> - open the input file in each process

> - generate a different vrt file for each thread

>

> I even copy/pasted the code that gdal2tiles_parallel uses in their

> processes, but with no success, always the same error. The

> VRT_SHARED_SOURCE option

> does not seem to change anything (but looking at the wiki, it's not clear

> whether it applies to autowarped vrt too)

 

VRT_SHARED_SOURCE indeed only works for "regular" mosaicing VRTs. So not to warped VRT.

 

Warped VRT opens the source dataset as a shared dataset in

https://github.com/OSGeo/gdal/blob/trunk/gdal/alg/gdalwarper.cpp#L1675

So you could have an issue if you open the VRT twice in the same thread, and then use each handle in a different thread, since both VRT handles would point to the same shared dataset. But it doesn't look that's what you are doing.

 

As a workaround to investigate, you could try to remove the GDAL_OF_SHARED | flag in the line I mentionned and see if that makes a difference (but we could potentially - not sure - have issues at closing time since different call sites might make an assumption on the dataset being opened with shared semantics)

 

What you could do too is run the script through your prefered debugger and set a breakpoint in GDALOpenEx and look at the calling sites. If you are using 2 threads, then if we want things to work correctly, GDALOpenEx() should be called twice on the TIFF dataset and return a different handle.

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com



_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Loading...